Why a LangGraph run can vanish silently before its first checkpoint

I run browser and tool-calling agents out of Singapore, and the failure that annoyed me most this year was not a crash with a stack trace. it was a run that was accepted, acknowledged, and then simply not there. no exception in the logs I was watching, no failed row, no partial state. the user’s request went in and nothing came out, and when I asked the checkpointer about the thread it told me, politely, that it had never heard of it.

This is the durability gap before the first checkpoint. everything LangGraph promises about resuming after a crash is a promise about state that has already been written. before the first write, there is nothing to resume from, and nothing to report either. this article is my answer to the question raised in langgraph issue 8764: how can a run disappear without an error, and what do you do about it. I have not tied this to one specific bug report. I am describing the mechanism as I understand it from the documentation and from running these systems, and I flag where you should check your own version.

The stakes are practical. if your agent does paid work, sends messages, or holds a user’s task, “silently never started” is the worst failure class because nothing alerts you. a loud crash gets fixed. a vanished run gets noticed by the user three days later. the window is small, often a few seconds, but at fleet scale a small window gets hit every day.

background and prior art

LangGraph’s persistence model is documented in the LangGraph persistence docs. when you compile a graph with a checkpointer and invoke it with a thread_id, state is saved as checkpoints at the boundaries of super-steps. a super-step is one round of node execution, where all nodes scheduled in that round run, and their writes are applied together. the checkpoint is a snapshot of state plus metadata about what runs next. resume, time travel, human-in-the-loop interrupts and fault tolerance all read from those snapshots.

The companion page on durable execution describes the durability modes. as of October 2026 the documented options are "exit", "async" and "sync". in "exit" mode state is persisted only when the graph finishes or errors out in a way the process can still handle. in "async" mode the checkpoint write for step N is handed to a background task while step N+1 starts. in "sync" mode the write for step N completes before step N+1 begins. I covered the tradeoffs between them in langgraph durability sync vs async and what actually survives a crash. this article is about the case that piece only touches: the crash that happens before any of those modes has had anything to write.

Prior art outside LangGraph is old. every job system has a “accepted but not yet recorded” window. message queues solve it by making the enqueue the durable act. databases solve it with write-ahead logs. workflow engines like Temporal record workflow start as the first event before any worker touches it. the lesson from all of them is the same: the durable record of intent has to exist before the work starts, and it has to be written by something other than the worker that might die. LangGraph, as a library running inside your process, leaves that responsibility with you.

the core mechanism

Think of a run as a timeline with four points.

• t0: your API handler receives the request and decides to start a run.
• t1: the process calls graph.invoke or graph.astream with a config containing a thread_id.
• t2: the first checkpoint write lands in your store (Postgres, SQLite, Redis, whatever you configured).
• t3: nodes execute, more checkpoints land, the run completes.

Between t0 and t2 there is no durable record anywhere that this run exists, unless you created one. the checkpointer is the only thing LangGraph writes to, and it has written nothing. so any failure in that window has these properties:

• no thread row exists, so get_state returns an empty snapshot and get_state_history yields nothing
• no pending writes exist, so there is nothing for a retry to pick up
• the process that died took its in-memory state and its logs buffer with it
• the caller may already have received a 202 and a run id, so from the outside the run looks accepted

What kills a run in that window? the list is boring, which is why it bites. an OOM kill from the container runtime. a deploy that sends SIGTERM and then SIGKILL after the grace period while the handler is mid-flight. a Kubernetes pod eviction. a worker process in a pool that segfaults inside a native dependency. an unhandled exception raised inside a background task whose result nobody awaits, so Python logs “Task exception was never retrieved” at garbage collection time, if at all. a database connection failure on the very first checkpointer call.

That last one deserves attention because of how async durability interacts with it. in "async" mode, the first write is submitted without the graph waiting on it. if the write fails, the failure surfaces from a background future. depending on the version and how you consume the stream, it can arrive late, arrive as a different exception than you expected, or arrive after the process has already moved on. I would not assume a failed first write aborts the run before node code executes. test it on your version rather than trusting me or the docs on this point.

why “exit” mode widens the window

In "exit" mode the first checkpoint is not written until the run ends. so the window from t1 to t2 is not seconds, it is the entire run. a run that takes 11 minutes and dies at minute 10 leaves exactly the same trace as a run that died instantly: nothing. exit mode is a legitimate choice for short, cheap, restartable graphs. for anything with long tool calls it turns every crash into a total loss, and I would not use it for agents that do external work.

why the first node is special

Even in "sync" mode, the checkpoint that matters for resume is the one taken after a node’s writes are applied. if your first node is the expensive one, say a browser launch plus a login plus a 40 page scrape, then the first state-bearing checkpoint cannot exist until that node returns. the graph input may be saved earlier as an input checkpoint (the persistence docs describe checkpoints with metadata about their source), but a checkpoint holding only your input is of limited use if what you need is the expensive node’s output. and you should verify on your own version whether and when the input checkpoint is written, instead of assuming. get_state_history on a thread you deliberately killed is the honest test.

the missing signal

There is a second half to the mechanism, which is the absence of a signal. LangGraph does not know about runs that never reached it. a monitoring setup built on checkpoint rows or on LangSmith traces sees only runs that got far enough to produce data. if you alert on “runs that failed”, a run that never wrote anything cannot fail. the failure is a negative: something that should exist and does not. you can only detect negatives by comparing against an independent list of what should exist. that list is the run ledger, and it is the core fix.

the fix in three parts

The pattern I settled on has three parts, none of them clever.

• a run ledger: a row written by the API handler, before invoking the graph, in a table you own. columns: run_id, thread_id, input hash, status, created_at, last_heartbeat_at, attempt count.
• an early, cheap first checkpoint: a trivial first node, with durability set to sync for that step, so the “accepted” state is in the checkpointer before anything expensive runs.
• a reaper: a periodic job that finds ledger rows in pending or running with no recent heartbeat and no checkpoint, and either re-enqueues them or pages you.

The ledger is the part that catches the vanish. the early checkpoint shrinks the window. the reaper turns detection into action.

import hashlib
import json
import time
import uuid

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.postgres import PostgresSaver
from typing_extensions import TypedDict


class State(TypedDict, total=False):
    task: str
    accepted: bool
    result: str


def intake(state: State) -> State:
    # trivial on purpose. its only job is to put a checkpoint in the store
    # before the expensive node runs.
    return {"accepted": True}


def expensive_work(state: State) -> State:
    # browser launch, login, scrape, long tool call
    return {"result": do_the_thing(state["task"])}


builder = StateGraph(State)
builder.add_node("intake", intake)
builder.add_node("expensive_work", expensive_work)
builder.add_edge(START, "intake")
builder.add_edge("intake", "expensive_work")
builder.add_edge("expensive_work", END)


def start_run(ledger, saver, task: str) -> str:
    run_id = str(uuid.uuid4())
    input_hash = hashlib.sha256(task.encode()).hexdigest()

    # 1. durable intent first, outside the graph, outside the worker
    ledger.insert(
        run_id=run_id,
        thread_id=run_id,
        input_hash=input_hash,
        payload=json.dumps({"task": task}),
        status="pending",
        created_at=time.time(),
    )

    # 2. only now touch the graph
    graph = builder.compile(checkpointer=saver)
    config = {"configurable": {"thread_id": run_id}}
    graph.invoke({"task": task}, config, durability="sync")

    ledger.update(run_id, status="done")
    return run_id

Two details in that sketch matter. first, thread_id equals run_id, so the ledger and the checkpointer share a key and the reaper can join them with one query. second, the ledger stores the payload, because if the run vanished the checkpointer does not have your input and you need something to re-submit from. check the durability argument against your installed version, since this parameter replaced older checkpointing flags and its exact name and placement have moved between releases.

worked examples

These are constructed scenarios with round numbers, to show the arithmetic. they are not measurements from a benchmark, and the timings are illustrative of the shape of the problem.

example 1: the deploy that eats a run

Say an agent service runs 6 pods behind a load balancer, handling about 900 runs a day, which is roughly 37 an hour. a deploy rolls pods one at a time, and each pod gets a 30 second termination grace period. suppose each run spends its first 8 seconds before the first checkpoint lands, because the first node cold-starts a browser context. a pod that receives SIGTERM while 2 runs are in that 8 second window will, if the shutdown handler does not drain them, drop both.

Across a 6 pod rollout, the chance that any individual pod has at least one run in the window at the moment of termination is not small. with 37 runs an hour spread across 6 pods, each pod sees about 6 starts an hour, and an 8 second window is 0.22 percent of an hour, so any given pod has a run in the window around 1 time in 75 terminations on average. across a few deploys a day, you lose a run every couple of weeks. that rate is low enough that nobody notices and high enough that it keeps happening. the numbers are an illustration of how to size your own exposure, so plug in your own starts per hour and your own pre-checkpoint seconds.

The counter-strategy is twofold. the ledger row means the reaper finds the orphan within one sweep interval. and a SIGTERM handler that stops accepting new runs and waits for in-flight ones to pass their first checkpoint turns the loss into a delay. Python’s signal module documentation covers installing handlers, with the standing caveat that handlers run in the main thread and you need to hand off to your event loop deliberately.

example 2: exit durability on a 12 minute run

Consider a graph set to durability="exit" that calls a slow vendor API and a headless browser, averaging 12 minutes. the container has a 2 GiB memory limit and the browser tool occasionally spikes past it, getting OOM killed. in exit mode, the checkpoint table holds nothing for that thread, because nothing was persisted on the way. the run is not “failed at step 3”. it is not recorded at all.

Switch the same graph to "sync" and add the intake node. now the thread exists after the first second, each completed node leaves a checkpoint, and the OOM kill leaves a thread whose latest checkpoint says exactly which node was next. the reaper sees a heartbeat that stopped, finds the thread, and resumes by invoking the graph with None input and the same thread_id, which is the documented way to continue from the last checkpoint. the cost is one extra write per step. for a run with 5 nodes that is 5 writes against a 12 minute run, which is noise next to the browser time.

If you run long tool calls on self-hosted infrastructure, I compared where checkpoint storage actually lives and what it costs in self-hosted LangGraph checkpointing vs LangGraph Cloud for long tool calls. the pre-checkpoint window exists in both setups. what differs is who owns the ledger.

example 3: the browser agent that times out on startup

In an earlier piece I wrote about a browser agent that works locally but times out 30 seconds into startup on a server. that failure mode has a nasty interaction with this one. suppose the first node launches the browser, and on the server the launch hangs for 30 seconds before a supervisor kills the worker. the run produced no checkpoint, because the node never returned, and the supervisor’s restart policy brought the worker back healthy. every health check is green. the run is gone.

With a ledger, the same event looks like this: row created at 10:00:00, status pending, no heartbeat, no thread in the checkpointer at 10:01:00. the reaper’s query returns it on the next sweep. you see it, you see that it has been retried twice, and on the third attempt it gets flagged for a human. without a ledger you would have a user complaint and no trail. the browser startup problem is separate and has its own fixes, but the vanish is what made it expensive.

edge cases and failure modes

ledger and checkpoint disagree

The ledger says running, the checkpointer has a thread with a recent checkpoint, but your heartbeat column is stale. the worker is alive, you just forgot to update the heartbeat. a naive reaper re-enqueues and you get two workers on one thread. counter: make the reaper check the checkpointer’s latest checkpoint timestamp as well as the ledger heartbeat, and treat either being recent as alive. better, take a lease: the worker holds a row-level lock or an expiring lease token, and a resumed run must acquire it. two runs writing to one thread is a worse failure than a vanished run.

the retry that repeats side effects

Re-submitting a vanished run is only safe if the pre-checkpoint work was idempotent. in the first node of many agents that is not true: it may have sent an email or created a record before the crash. the crash left no checkpoint, but it did leave side effects in the outside world. counter: keep the first node free of side effects, use an idempotency key derived from the run_id for anything external, and record in the ledger which external calls were attempted. the intake node pattern helps here precisely because it is a point you can declare side-effect free.

the poison input

A run that deterministically crashes the process, for example a pathological page that OOMs the browser, will vanish, get re-enqueued, vanish again, and loop forever, burning compute each time. counter: an attempt counter in the ledger with a hard cap, say 3, after which the status becomes quarantined and a human looks at it. this is the same shape of problem as an agent looping until the recursion limit, which I wrote up in how to stop a LangGraph agent from looping until it hits the recursion limit. different layer, same discipline: every retry path needs a budget.

first write fails but the run continues

In async durability, a failed first checkpoint write may not stop the run. nodes execute, tools fire, and the checkpointer never recorded any of it. you get a run that completed with no persistence at all, which is a quieter cousin of the vanish. counter: after the intake node, have the worker read back the thread with get_state and confirm the checkpoint exists before proceeding to the expensive node. one extra read per run is cheap insurance. if the read comes back empty, fail loudly and let the reaper handle it. also confirm setup() has been run on your Postgres saver so the tables exist, since a missing table is the dullest possible cause of this whole class of problem.

clock and sweep interval mistakes

A reaper that sweeps every 5 minutes with a 10 minute staleness threshold means a vanished run is detected between 10 and 15 minutes late. if your users expect results in 2 minutes, that is useless. counter: set the staleness threshold from the pre-checkpoint duration you actually see (the intake node should land within a second or two), not from total run time. a run with no thread after 30 seconds is already suspicious even if the whole run is allowed 12 minutes. use two thresholds: a short one for “never got a first checkpoint” and a long one for “stopped making progress”.

what we learned in production

The biggest shift for me was treating the ledger as the source of truth for “what did we promise”, and the checkpointer as the source of truth for “how far did we get”. those are different questions and I had been using one store to answer both. once they were separate, the vanished runs became visible as a simple anti-join: ledger rows with no matching thread. that query is the single most useful alert I have for agent infrastructure, and it costs almost nothing to run.

The other lesson is that graceful shutdown is a feature you have to build. frameworks tend to assume the process lives long enough to finish what it started. containers do not give you that. I now treat “pass the first checkpoint” as the minimum an in-flight run must reach before a worker is allowed to exit, and I size the termination grace period against that number, not against total run length. this is not a LangGraph defect so much as a property of running a library inside a process you do not fully control. the docs are clear about what a checkpoint does. the gap is in the time before one exists, and nobody else is going to cover it for you.

If you are also running agents against real sites from Singapore, the same discipline applies to the network side, but that is a different article. for this one, the short version is: write down that a run exists before you start it, make your first node cheap and boring, and go looking for the runs that should exist and do not.

references and further reading

• LangGraph persistence documentation: checkpoints, threads, state snapshots and the checkpointer interface.
• LangGraph durable execution documentation: the durability modes and what each one guarantees on failure. verify against your installed version, as of October 2026.
• Python signal module documentation: installing SIGTERM handlers for graceful drain.
• langgraph issue 8764: the reader question this article answers. read the thread for the specifics of your version.
• on this site: the blog index has the rest of the agent operations series, including the durability and checkpointing pieces linked above.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-10.