Idempotency

Idempotency is a plane. Every layer that makes an effect passes through it, and it protects the effect, not the orchestration.

Repeating the same command should have the same effect as making it once.

Use one durable PostgreSQL table and service. Do not use Redis or distributed locks for the correctness boundary.

Record

agent.idempotency_keys
  id
  organization_id
  scope
  key
  request_hash
  status: processing | completed
  lease_token          the fence; a new value on every claim and reclaim
  resource_type
  resource_id
  response
  claimed_at
  lease_expires_at     the worker lease; short
  retain_until         how long the answer is remembered; long

unique(organization_id, scope, key)

request_hash prevents the same key from being reused for different content.

Two clocks, not one

The lease and the retention answer different questions, and one field cannot hold both.

A tool claim must be remembered for at least the longest max_run_duration any definition may set, because an approval can hold a Run for days and the claim is the replay journal. That is a per definition value and retention is a per scope constant, so the constant takes the ceiling: 30 days, which is also the cap validation puts on max_run_duration. A tool claim must also be reclaimable within seconds, because a worker that died mid call blocks every retry until its lease ends.

One expires_at cannot be both. Set it long and a segment that restarts five seconds after a crash meets a live lease it cannot take, so the Run stalls until the lease ends. Set it short and the journal forgets an effect the Run already made, so a restarted segment repeats it.

tool.email.send     lease 60 s          retain 30 days
webhook.nylas       lease 30 s          retain 24 hours
surface.saved_search.start  lease 30 s  retain 24 hours

The tool retention is 30 days for every tool scope, not a per tool judgement. The paragraph above takes the ceiling on purpose, so a shorter number here would be the same bug it exists to prevent.

A claim is reclaimable when lease_expires_at has passed and the status is still processing. A claim is readable while retain_until holds, whatever the lease says.

Repository and service

class IdempotencyRepository(Protocol):
    async def try_claim(
        self,
        organization_id: UUID,
        scope: str,
        key: str,
        request_hash: str,
        *,
        lease_token: UUID,            # the caller mints it; it must hold it
        lease_expires_at: datetime,
        retain_until: datetime,
    ) -> IdempotencyRecord | None: ...   # None when the key was already claimed

    async def reclaim(
        self,
        organization_id: UUID,
        scope: str,
        key: str,
        *,
        lease_token: UUID,
        lease_expires_at: datetime,
        retain_until: datetime,       # the floor; a reclaim never shortens it
    ) -> IdempotencyRecord | None: ...   # None when another worker took it

    async def get(
        self, organization_id: UUID, scope: str, key: str
    ) -> IdempotencyRecord | None: ...   # the service reads after every conflict

    async def complete(
        self,
        claim_id: UUID,
        lease_token: UUID,
        *,
        resource_type: str | None,
        resource_id: str | None,
        response: dict | None,
    ) -> IdempotencyRecord | None: ...   # None when the lease moved on

    async def release(self, claim_id: UUID, lease_token: UUID) -> bool:
        """Delete an unfinished claim we still hold. False when the lease moved on."""

class IdempotencyService:
    async def read(
        self, organization_id: UUID, scope: str, key: str, request_hash: str
    ) -> ClaimOutcome: ...       # never writes; answers 'absent' when nothing is stored
    async def claim(
        self, organization_id: UUID, scope: str, key: str, request_hash: str,
        *, lease: timedelta
    ) -> ClaimOutcome: ...
    async def complete(self, claim, *, resource_type, resource_id, response): ...
    async def release(self, claim): ...

read() is a separate method, and it takes no claim. The tool path reads the journal before the policy checkpoint and claims after it, so a read that claimed would take the key for a call a person has not yet allowed. That claim then sits processing for a whole lease and blocks the resume. See tools and integrations.

The repository owns atomic SQL behavior; the service owns duplicate semantics.

complete() on the repository and on the service both name the resource. The record holds resource_type and resource_id. A method that writes the id alone leaves the type null on every row. A reader then cannot tell a CRM company from a Nylas message.

claim() takes organization_id, because the unique key starts with it. This platform has no ambient tenant, and every other repository here takes the tenant as an argument. complete() and release() filter on claim_id. The claim itself answered that id, so neither needs the tenant again.

The lease is an argument, and the retention is a constant. The table above reads as though both come from the scope, and only the retention does. The lease is the operation timeout plus a margin, which the caller knows and the scope does not: ToolSpec.timeout_s runs to the 120 second STEP_BUDGET_S, so a fixed 60 second tool lease would let a second worker reclaim a key while the first is still inside its own timeout. The service raises the retention floor to the lease when a caller asks for a longer one, because the row carries a retain_until >= lease_expires_at check.

The caller mints the lease token, and try_claim() answers None on a conflict. A worker must hold the token it will carry to its ending, and a conflict is the ordinary outcome the caller reads the stored row for.

The service answers an outcome, never a record. The four endings of a claim are different things to the caller, and a record makes each caller re-derive which one it holds.

@dataclass(frozen=True)
class ClaimOutcome:
    kind: Literal['absent', 'claimed', 'completed', 'processing', 'conflict']
    claim: IdempotencyRecord | None   # 'claimed' alone; complete()/release() take it
    response: dict | None             # 'completed' alone; the stored answer
    resource_type: str | None         # 'completed' alone
    resource_id: str | None           # 'completed' alone

absent         the journal has no answer; read() alone answers it
claimed        we own the key; execute, then complete or release
completed      the stored result; return it, and mark the span replayed
processing     another worker holds a LIVE lease on this key
conflict       the same key, with a different request hash

⚠️ absent is "no answer", not "no row". A processing row whose lease has passed belongs to a worker that is gone, and the next caller reclaims it. So read() answers absent for that row. The caller runs its checkpoints, and claim() performs the atomic reclaim. A read that reported processing there would raise on every attempt, the reclaim would never run, and the run would fail on a worker that died seconds earlier.

The request hash is compared first, whatever the lease says. A passed lease on a key claimed for different content is still a conflict.

processing is ordinary concurrency, and a caller must not answer it with a value. Three cases reach it: two orchestrator attempts, one duplicate delivery, and one fan-out that proposes a call twice. In each one the first worker is healthy and still running. A crashed worker is the rarer case, and its lease expires into the reclaim branch below.

Whatever the cause, the effect may be in flight. A result that reported a failure would state something the platform never observed.

A caller that a replaying orchestrator drives raises. A tool call runs inside an Inngest step, so the raise ends the step and Inngest runs it again. The lease must expire before that orchestrator makes its last attempt, or every attempt meets the lease and the run fails. The lease is the operation timeout plus a margin, which satisfies that for the retry policies this platform sets. See tools and integrations.

A caller at an HTTP boundary does not raise. A webhook handler is the outermost frame. A 5xx tells the vendor that a delivery which succeeded failed. Slack disables an Event Subscription that answers 5xx often enough. So the handler drops the delivery, records the reason, and answers its ACK.

completed   ->  drop reason `duplicate`, then ACK
processing  ->  drop reason `duplicate`, then ACK
conflict    ->  drop reason `hash_mismatch`, then ACK

completed is the common case. The vendor re-sent a delivery this platform already handled.

conflict at a webhook means the vendor re-sent one delivery id with different content. The delivery is not a duplicate, and it is not safe to process either: the key already answers for a different body. Both endings count a drop reason, so neither is a silent loss. See channel gateway.

The lease token is a fence, and a reclaim is unsafe without one

A worker that stalls past its lease has not stopped. It wakes up later and finishes.

worker A claims                       lease 60 s
worker A stalls
                                      the lease passes
worker B reclaims, and executes
worker A wakes, calls release()    ->  it deletes the claim B holds
worker A wakes, calls complete()   ->  it stores A's stale answer over B's

Either line is worse than having no claim at all. The second hands the next caller a result for work that was really done differently.

So every claim and every reclaim mints a new lease_token, and both endings carry the token they were given.

UPDATE agent.idempotency_keys SET ... WHERE id = :claim_id AND lease_token = :lease_token

Zero rows means the lease moved on. complete() returns None and release() returns False, and the caller does nothing: it must not retry and it must not raise, because another worker owns that key now.

The reclaim rule and this rule are two halves of one guarantee. The first stops two workers reclaiming at once. The second stops the worker they replaced from writing afterwards.

Complete, or release

A claim has exactly two endings. Leaving it in processing is a defect, not a third ending.

A retryable failure releases the claim. A tool that returns ToolResult(ok=False, retryable=True) after a vendor 429 made no effect. If that answer were completed, the stored result would be a failure the caller can never get past: the agent proposes the same call, the claim returns the cached 429, and the Run fails on a vendor that recovered minutes ago.

A business failure completes the claim. not_found and invalid_input are answers. Repeating the call returns the same answer, and caching it is correct.

The distinction is exactly ToolError.retryable. See tools and integrations.

Atomic claim

INSERT ... ON CONFLICT DO NOTHING
          │
      ┌───┴────┐
   inserted  conflict
      │          │
   new claim   load row
                 │
          request_hash same?
             ┌───┴───┐
            no      yes
            │         │
         conflict  completed -> return saved result
                   processing + live lease   -> processing
                   processing + lease passed -> atomic reclaim

Stale claim takeover must itself be conditional/atomic so two workers cannot reclaim the same lease.

A reclaim moves the lease to the new worker. It never shortens retain_until.

Boundaries

⚠️ A provider callback resolves its tenant before it claims. The key is scoped to one organization, and a vendor callback names no organization. The handler reads agent.provider_jobs first, and claims after it.

Two of its endings never reach a claim, so they carry their own drop reasons beside duplicate and hash_mismatch:

unknown_job    the path names a job this platform does not hold
job_mismatch   the body names a different job than the path

Both answer an ACK, for the reason every drop here does. See asynchronous provider jobs.

Run admission does not use this table. Its effect is a row we own, so agent.runs carries the start key and its insert is the Run claim. One narrow surface exception happens before admission. A saved-search start must freeze a brief and baseline from mutable product tables before it can build the Run request. surface.saved_search.start claims that input preparation and stores the immutable input snapshot as its response. It never records admission, dispatch or Run status. The completed claim supplies a stable derived key to the normal agent.runs insert. See saved searches and runtime execution.

Inngest checkpoints orchestration. Idempotency protects the business effect.

A completed Tool claim returns its stored response, so this table is also the agent replay journal. A restarted segment reads back what the earlier attempt did, and the runtime needs no second table for it.

The Tool key never uses the model tool-call ID. A restarted Agent segment makes the model mint a new one, so that key would miss the earlier claim and repeat the effect. See tools and integrations.

External side effects

Postgres cannot atomically commit with a vendor API.

Postgres claim       prevents concurrent local execution
Vendor idempotency   protects crash-after-vendor-success retry

If a provider has no idempotency support, the handler needs a provider-specific reconciliation strategy before the operation can be considered retry-safe.

What it does not replace

Retention

retain_until is set per scope, and a scheduled job deletes past it.

That job is idempotency.sweeper, an Inngest cron beside run.reaper. It runs on the hour, which is 24 times finer than the shortest retention it enforces. Nothing else deletes a claim: release() removes an unfinished one, and a completed one is the journal, so it must outlive every replay that could read it.

A row the sweep deletes is answerable by nobody. The Run that claimed it ended at least a retention ago, so no replay can reach it.

⚠️ The pass is bounded, and the READ count is what says whether it keeps up. An unbounded delete would hold row locks over every expired claim of every tenant, on the table the tool call path writes. A pass whose read fills its bound leaves rows for the next hour and logs a warning, so a backlog that never drains is visible without reading every quiet pass. The bound stays under PGRST_DB_MAX_ROWS, because PostgREST truncates a read there and reports the truncation in Content-Range alone, and a larger bound is refused rather than clamped. The delete count answers a different question: a writer that removes a claim between the read and the delete lowers it, and says nothing about the backlog.

Do not retain a large saved response indefinitely. A response above the store limit keeps a ResourceRef in place of the body.

Rules

• PostgreSQL is source of truth.
• Claim and stale reclaim are atomic repository operations.
• Every claim and reclaim mints a new lease_token.
• complete() and release() are conditional on the token. A lost fence is a silent no-op.
• Same key + different request hash is a conflict.
• Live processing claim is never executed a second time.
• A live processing claim is not a value. A caller an orchestrator replays raises ClaimInFlight; an HTTP boundary drops the delivery.
• A lease expires before the orchestrator's last retry attempt, or every attempt meets it.
• read() answers absent for a claim whose lease passed, so claim() reclaims it.
• Completed claim returns saved resource/result.
• The lease and the retention are two clocks. Never derive one from the other.
• A claim ends in complete() or release(), and both endings are terminal.
• A completion carries a response or a resource. One that records neither replays an empty success.
• A retryable failure releases. A business failure completes.
• Pass key to downstream vendor when supported.
• Idempotency logic is shared; Tool/Trigger/Run layers do not reimplement it.
• A Tool claim outlives the Run it belongs to.
• Run admission is guarded by a unique index on agent.runs, never by this table. The saved-search surface claim freezes input before admission and does not claim the Run.

Minimum contract tests

• 20 concurrent claims for same key produce one owner.
• A direct capability or generic definition start writes no row in this table.
• A saved-search start atomically freezes one input before Run admission; 20 concurrent preparations produce one snapshot.
• A live saved-search preparation returns retryable in-progress. One caller reclaims it after the 30-second lease.
• A saved-search preparation fence lost before completion starts no Run.
• A saved-search snapshot remains for 24 hours. Its stable Run key replays an admitted Run through the 396-day Run retention window.
• Same key/different hash fails.
• Completed duplicate returns stored resource/result.
• read() writes no row, and it answers absent for a claim whose lease passed.
• A crashed worker's claim is reclaimed by the next caller, and the run continues.
• Live lease cannot be stolen.
• A live lease on the caller's own key raises, and no result reports it.
• A passed lease is reclaimed by exactly one worker.
• A worker whose lease passed cannot complete the claim afterwards.
• A worker whose lease passed cannot release the claim another worker holds.
• A claim whose lease passed is still readable while retain_until holds.
• A reclaim moves the lease and never shortens retain_until.
• A retryable tool failure releases the claim, and the next attempt executes.
• A business failure completes the claim, and the next attempt returns the stored answer.
• Vendor idempotency prevents duplicate send after crash/retry.
• A Tool claim survives a Run that waited on an approval for longer than a day.