Handling Massive JSON Payloads Without Crashing Your Workflow Runner

Workflow runners crash on large JSON payloads for one of two reasons: they load the entire document into memory before doing anything with it, or they pass a massive item set through multiple nodes and accumulate execution data at every step. The fix for the first is streaming. The fix for the second is batching combined with disabling execution data persistence for the workflow. Neither requires changing your infrastructure or moving to a larger server.

The incident that prompted this post: a product catalog sync workflow that fetched 22,000 products as a single JSON array from a PIM system’s export endpoint. The response was 84MB. n8n loaded the entire payload, tried to serialize all 22,000 items into execution data at the first node boundary, and exhausted available memory at around 2.1GB on a 4GB VPS. The workflow had been working fine for months when the product catalog was 4,000 items. Nobody noticed it was going to fail until the catalog grew past the point where it did.

After restructuring the workflow — streaming parse, batched processing, execution data disabled — the same sync runs in 4 minutes using 180MB peak memory. This post covers what changed and how to apply the same pattern to your own large-payload problem.

What’s Actually Consuming the Memory

The obvious culprit is the JSON parse itself — loading an 84MB file into memory produces a JavaScript or Python object that’s typically 3–8x the size of the source JSON, because in-memory objects carry overhead that the flat text representation doesn’t. An 84MB JSON file can easily become 400–600MB as a parsed object.

The less obvious culprit — and the one that hit me — is how n8n handles execution data. At every node boundary, n8n serializes the complete output of that node into its execution database so you can inspect what each node produced. For a workflow that receives 22,000 items and passes them through 8 nodes, that means n8n writes 176,000 rows of serialized item data to its SQLite or Postgres database during the execution. This happens regardless of whether you ever look at that execution history. The database writes slow the execution, the serialization keeps large objects alive in memory longer than they need to be, and the database grows proportionally to how many large-dataset workflows you’ve run.

Both problems need addressing. The streaming approach handles the initial parse. The execution data setting handles the node-by-node accumulation. Solving only one of them gets you partway there.

Streaming JSON: Processing Without Loading

A streaming JSON parser reads the document token by token — it processes the opening brace, the first key, the first value, the comma, the second key, and so on — without constructing the full object in memory. You handle each element as it arrives and can discard it before reading the next one. Peak memory becomes proportional to one record at a time, not the full document.

pip install ijson requests

# ❌ The approach that crashes on large payloads
import requests, json

response = requests.get("https://pim.example.com/export/products")
products = json.loads(response.text)  # Entire 84MB string into memory as one object
for product in products:
    process(product)

# ✅ Streaming parse — constant memory regardless of payload size
import ijson
import requests

def stream_products(url: str, auth_header: str):
    """
    Streams a JSON array from a URL, yielding one object at a time.
    Peak memory: size of one product record, not the full file.
    """
    with requests.get(
        url,
        headers={"Authorization": auth_header},
        stream=True,    # Critical: don't buffer the response body
        timeout=120
    ) as response:
        response.raise_for_status()

        # ijson.items() yields each element of the top-level array
        # "item" is ijson's path prefix for root-level array elements
        parser = ijson.items(response.raw, "item")

        for product in parser:
            yield product  # One product at a time — previous one is GC'd

# Usage: process in batches of 500
BATCH_SIZE = 500
batch = []

for product in stream_products(EXPORT_URL, auth_header=f"Bearer {TOKEN}"):
    batch.append(product)

    if len(batch) >= BATCH_SIZE:
        process_batch(batch)
        batch = []           # Release the batch — memory drops back to near-zero
        batch = []

if batch:                    # Process the final partial batch
    process_batch(batch)

The stream=True parameter on the requests call is as important as the streaming parser. Without it, requests buffers the entire response body in memory before returning — you’d have the full 84MB in RAM before ijson even starts reading. With stream=True, the response body is read in chunks as the parser requests them.

npm install stream-json node-fetch

import { parser } from "stream-json";
import { streamArray } from "stream-json/streamers/StreamArray.js";
import { pipeline } from "stream/promises";
import fetch from "node-fetch";

async function streamProducts(url, token) {
  const response = await fetch(url, {
    headers: { Authorization: `Bearer ${token}` },
  });

  if (!response.ok) throw new Error(`HTTP ${response.status}`);

  const results = [];
  let batch = [];
  const BATCH_SIZE = 500;

  // Build a processing pipeline:
  // HTTP stream → JSON parser → array streamer → your handler
  await new Promise((resolve, reject) => {
    const jsonStream = response.body
      .pipe(parser())           // Tokenize the JSON stream
      .pipe(streamArray());     // Emit {key, value} for each array element

    jsonStream.on("data", ({ value: product }) => {
      batch.push(product);

      if (batch.length >= BATCH_SIZE) {
        processBatch(batch);
        batch = [];             // Release — GC can reclaim this memory
      }
    });

    jsonStream.on("end", () => {
      if (batch.length > 0) processBatch(batch);  // Final partial batch
      resolve();
    });

    jsonStream.on("error", reject);
  });
}

function processBatch(products) {
  console.log(`Processing batch of ${products.length} products`);
  // Your batch logic here
}

The Node.js streaming approach uses the same principle: pipe the HTTP response through a JSON tokenizer, which feeds a streaming array emitter, which fires an event per element. Each element is handled and released before the next one arrives. The stream pipeline manages backpressure automatically — if your processing falls behind the download speed, the parser pauses the HTTP stream until you catch up.

Batching in n8n: SplitInBatches and Execution Data

If the workflow is in n8n rather than a standalone script, you have two things to address: how the initial data enters the workflow, and how n8n handles it between nodes.

For data that comes from an HTTP Request node — where the entire response loads before n8n can do anything with it — the immediate fix is restructuring the fetch. If the API supports pagination, use it (the cursor-based pagination pattern covered separately for GraphQL applies to REST with offset pagination equally). If the API only provides a bulk export endpoint, use a Code node with streaming to fetch and parse the data rather than the HTTP Request node, so you control the memory behavior.

// Code node: Run Once for All Items
// Fetches large JSON export and returns items without loading all into memory at once

const EXPORT_URL = "https://pim.example.com/export/products";
const TOKEN = $env.PIM_API_TOKEN;
const BATCH_SIZE = 250;  // Items per batch returned to n8n

// Note: n8n's this.helpers.httpRequest() buffers the full response.
// For truly streaming behavior, use the fetch API directly in Code node.

const response = await fetch(EXPORT_URL, {
  headers: {
    "Authorization": `Bearer ${TOKEN}`,
    "Accept": "application/json",
  }
});

if (!response.ok) {
  throw new Error(`Export fetch failed: ${response.status} ${response.statusText}`);
}

// Read the body as text — unavoidable for the JSON.parse approach.
// For payloads over ~50MB, see the external storage pattern below.
const body = await response.text();
const products = JSON.parse(body);

console.log(`Fetched ${products.length} products (${Math.round(body.length / 1024 / 1024)}MB)`);

// Return all items to n8n — the SplitInBatches node downstream handles chunking
return products.map(p => ({ json: p }));

Once the items are in n8n, the SplitInBatches node (Loop Over Items in newer versions) controls how many items each downstream node processes at once. Connect it after the data source node and set the batch size to 200–500. Every node after SplitInBatches receives that many items at a time, loops, and accumulates results — without all 22,000 items being live in memory simultaneously.

The Hidden Cost Nobody Mentions: n8n’s Execution Data

This is the thing that took me longest to understand and the reason I’d call it out explicitly rather than burying it in a list of tips.

The fix is two settings in n8n’s workflow configuration:

# In the n8n workflow editor:
# Click the workflow name → "Settings" tab → Execution section

# Option 1: Don't save execution data at all for this workflow
# "Save Execution Progress": OFF
# "Save Successful Execution Data": OFF
# "Save Failed Execution Data": ON  ← keep failures for debugging

# Option 2: Save only the final result, not intermediate node outputs
# In n8n's global settings (Settings → n8n → Execution):
# executionDataSaveOnSuccess: all | none | lastNode
# Setting to "lastNode" keeps only the final output, not all intermediate states

# Option 3: Prune execution data for large workflows using n8n's pruning settings
# EXECUTIONS_DATA_MAX_AGE=168          (keep 7 days of execution data)
# EXECUTIONS_DATA_SAVE_ON_SUCCESS=none (for specific workflow types)

# Via n8n API — disable execution data for a specific workflow:
curl -X PATCH "http://localhost:5678/api/v1/workflows/{workflowId}" \
  -H "X-N8N-API-KEY: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "saveExecutionProgress": false,
      "saveDataSuccessExecution": "none",
      "saveDataErrorExecution": "all"
    }
  }'

The tradeoff is real: with execution data disabled, you can’t click on a past successful run and inspect what each node produced. For debugging, you’re relying on your workflow’s own logging (console.log in Code nodes) and error notifications. For a well-tested large-dataset workflow that runs daily on a schedule, this is an acceptable tradeoff. For a workflow you’re still iterating on, keep execution data enabled until it’s stable, then disable it for production runs.

External Storage: The Escape Hatch for Very Large Payloads

For payloads genuinely too large to handle in memory even with streaming — exports over 100MB, files with millions of records — the right approach is to treat the workflow runner as an orchestrator rather than a data processor. The workflow triggers the export, writes it to external storage, and then processes it from there in chunks that the runner can handle comfortably.

import boto3
import ijson
import requests

s3 = boto3.client("s3")
BUCKET = "your-data-bucket"

# Step 1: Fetch the export and stream directly to S3
# No in-memory accumulation — HTTP stream → S3 multipart upload
def fetch_to_s3(export_url: str, s3_key: str, token: str):
    with requests.get(export_url, headers={"Authorization": f"Bearer {token}"},
                      stream=True, timeout=300) as response:
        response.raise_for_status()

        # S3 multipart upload streams the HTTP body directly to S3
        # without loading it into memory
        s3.upload_fileobj(
            response.raw,
            BUCKET,
            s3_key,
            ExtraArgs={"ContentType": "application/json"}
        )
        print(f"Uploaded export to s3://{BUCKET}/{s3_key}")

# Step 2: Process from S3 in a streaming loop
def process_from_s3(s3_key: str, batch_size: int = 500):
    s3_obj = s3.get_object(Bucket=BUCKET, Key=s3_key)
    stream = s3_obj["Body"]

    batch = []
    total = 0

    for product in ijson.items(stream, "item"):
        batch.append(product)
        total += 1

        if len(batch) >= batch_size:
            process_batch(batch)
            batch = []

    if batch:
        process_batch(batch)

    print(f"Processed {total} records")

    # Clean up the temp file
    s3.delete_object(Bucket=BUCKET, Key=s3_key)

# Full flow:
S3_KEY = f"exports/products-{datetime.now().strftime('%Y%m%d-%H%M%S')}.json"
fetch_to_s3(EXPORT_URL, S3_KEY, TOKEN)
process_from_s3(S3_KEY)

The S3 multipart upload streams the HTTP response body directly to S3 without buffering it locally. The subsequent read from S3 via ijson streaming processes the file one record at a time. Total peak memory: the size of one batch (500 product records at maybe 200KB) rather than the 84MB file. This pattern scales to arbitrarily large exports — a 2GB file works the same way as an 84MB file because neither is ever fully in memory at once.

Make.com: Working Within Hard Payload Limits

Make.com enforces a 10MB data limit per module execution on most plans. This isn’t a soft limit you can work around with clever configuration — it’s enforced at the platform level. An HTTP module that receives a response larger than 10MB will error regardless of what you do with headers or settings.

The practical approaches within Make.com’s constraints:

Use pagination at the API level. Don’t request a bulk export — use the API’s pagination to fetch data in pages of 100–500 records. Multiple modules, each with a small response, rather than one module with a huge one. This works for any API that supports pagination. For APIs that only offer bulk exports, Make.com may genuinely not be the right tool for that specific task.

Use an HTTP module to trigger an external process. A Make.com scenario can call a lightweight endpoint (Cloudflare Worker, Vercel function) that triggers the actual data processing pipeline. Make.com passes the parameters, the external process handles the large data, Make.com polls for or receives completion via a webhook. The large payload never touches Make.com’s execution environment.

Use Make.com’s Data Store for intermediate state. If you’re aggregating data across multiple small API responses, the Data Store module can accumulate records across multiple scenario executions. Each individual execution stays well under the 10MB limit; the Data Store holds the accumulated result. Query it in a final aggregation step once all pages have been fetched. This is Make.com’s native answer to the pagination accumulation problem.

Know when to use a different tool. Make.com is well-suited for workflows where each operation handles a modest amount of data and the value comes from connecting services intelligently. It’s not well-suited for bulk data processing. For a workflow that genuinely needs to handle 22,000 product records as a batch operation, n8n on a self-hosted VPS or a standalone Python script is the right tool — not because Make.com is worse, but because it’s designed for a different use case.

Approach Comparison by Payload Size

Practical Checklist for Large Payload Workflows

FAQ

How do I know what’s causing the OOM if the process just dies?

Check the system logs rather than the application logs — the OOM killer log entry tells you which process was killed and what its memory usage was at the time. On Linux: dmesg | grep -i oom or journalctl -k | grep -i oom. The entry shows the process name (typically node for n8n), its RSS at the time of kill, and which other processes were considered. If you see n8n’s Node process in there, memory exhaustion during data processing is confirmed. For more granular visibility before the crash, run watch -n 2 "ps aux --sort=-%mem | head -5" on the server during an execution — it refreshes every 2 seconds and shows which process is consuming memory as it happens.

Can I increase Node.js heap size to avoid the crash without restructuring the workflow?

You can, and it buys time, but it doesn’t fix the underlying problem. Setting NODE_OPTIONS=--max-old-space-size=4096 gives Node.js 4GB of heap instead of the default ~1.5GB. For n8n, add this to your environment before starting the n8n process. This works until the dataset grows past the new limit, which it will if the workflow is a daily sync and the product catalog keeps growing. The streaming approach is a permanent fix; the heap size increase is a temporary reprieve. That said, if you need a workflow to work today while you implement the proper fix, the heap size increase is a valid immediate measure.

Does Zapier have the same execution data problem as n8n?

Zapier doesn’t expose execution data storage the same way n8n does — you can’t inspect the full item-level output of every step in a Zap the way you can in n8n. The tradeoff is that Zapier’s execution data problem is hidden rather than avoidable: you can’t turn off the storage, but you also can’t see it accumulating. Zapier has its own data limits (typically 10MB per step, similar to Make.com) and will error on large payloads regardless of memory. For large-dataset processing, Zapier’s limits are harder to work around than n8n’s because you have less control over the execution environment.

What’s the difference between JSONL and standard JSON for this purpose?

Standard JSON arrays require a streaming parser (like ijson) to process record-by-record, because the parser has to understand the array structure before it can emit individual elements. JSONL files (one JSON object per line, no wrapping array) can be processed with a simple line-by-line file read — each line is a complete valid JSON object you can parse independently. No streaming parser library needed: for line in file: record = json.loads(line). This makes JSONL substantially easier to process in a memory-efficient way and is why bulk data export APIs increasingly prefer it. Shopify’s bulk operation API, for example, returns JSONL rather than a JSON array for exactly this reason.

I fixed the memory issue but the workflow is still slow. What else could be causing it?

Three common culprits after memory is resolved. First, database write throughput: if your processing writes each record to a database individually rather than in bulk inserts, the database round-trip time per record multiplies by record count. Replace individual inserts with batch inserts of 100–500 records per query. Second, n8n’s execution data: even with batching, if execution data saving is on, n8n is writing to its own database at every node boundary — this adds latency that scales with record count. Disable execution data as described above. Third, downstream API rate limits: if each batch triggers API calls (HubSpot updates, Shopify writes), you may be hitting rate limits that add wait time. The rate limiting pattern from the GraphQL pagination post applies here too — check for throttle headers in your API responses and build adaptive delays into your batch processing loop.

Is there a simple rule of thumb for when to stream vs when JSON.parse() is fine?

The number I use: if the JSON payload is under 5MB in a script with more than 500MB available memory, JSON.parse() is fine and the simplicity is worth it. If the payload is over 5MB, or if you’re in a constrained environment (a Cloudflare Worker with 128MB, a Lambda function with 256MB, an n8n instance on a 2GB VPS that runs other things simultaneously), stream it. The overhead of setting up streaming is about 10 extra lines of code and one library installation — it’s not meaningful complexity, and it means the workflow’s memory use doesn’t grow as the dataset grows. For anything that runs on a schedule and processes an ever-growing dataset, streaming is the right default from the start.