fix(jobs): classify transient database failures once for background retries by waleedlatif1 · Pull Request #8194 · simstudioai/sim

waleedlatif1 · 2026-09-23T05:59:09Z

Summary

One classifier for database failures in background jobs: classifyDatabaseFailure in @sim/utils/errors returns capacity (57014 statement timeout, 55P03, 25P03, 25P04, 53300), conflict (40P01, 40001), connection (08xxx, 57P01–57P03, postgres.js CONNECTION_*/CONNECT_TIMEOUT, and socket codes such as ECONNRESET, ECONNREFUSED or ENOTFOUND only when Drizzle's query error or a postgres.js query error is in the cause chain, so a client error carrying its own query stays a source failure), or permanent. It walks the cause chain, and a 57014 from an explicit cancellation stays permanent
Shared Trigger catchError helper (lib/core/errors/background-retry.ts): transient database failures back off 2 → 4 → 8 → 16 → 30 min with jitter up to their own attempt ceiling; everything else keeps the task's normal retries and attempt count
Workspace file search uses the helper. Its behavior is unchanged except that deadlocks, serialization failures, and dropped connections now get the same minute-scale retries as timeouts
Document processing (knowledge-process-document) gets a catchError. On a transient database failure the service writes the document as pending with processingDeferredUntil set to the scheduled retry, instead of failed, and the task retries at that same time. failed is written only when database retries run out or the error is permanent. So an uploaded document, which no recovery sweep covers, is no longer left failed forever by a slow database window. Connector documents keep their recovery sweep, which still sees a pending + deferred-until row after its grace period
An uploaded document deferred for a database retry gets a one-shot outbox check committed with the deferral, due once the retry is past the queue grace. If the same deferral is still pending and no run is live, the check marks the document failed ("its scheduled retry did not run") so the user's Retry re-dispatches it; no actor, no re-dispatch, no scan. A live or unknown run is rechecked hourly without spending an attempt, bounded by the recovery window. The deferral also stamps processingQueuedAt when unset, so a dispatch cannot claim the row as never-queued during the retry window
The worker's redacted database error now keeps the original in cause. Trigger records only name, message, and stack, so SQL and bound parameters still never reach it
Connector sync (content and members mode): a transient database failure records the error and never advances the failure counter, so a slow database week cannot auto-disable connectors (same exemption as capacity and rate-limit failures). A run that wrote documents (or completed a member) before failing retries in about 3 minutes; otherwise the retry climbs the failure ladder by the streak of zero-progress failed runs read from the run log, and ten in a row log an alertable error
0021 projection backfill drops its copy of the timeout matching and uses the shared classifier. 57014 and 55P03 page retries behave the same; deadlocks and dropped connections now retry the page too, since a page only fills rows that are still unset
Outbox left unchanged: marking database failures as deferred without consuming an attempt would retry a deterministic statement timeout (a query too heavy for its budget) forever, and the outbox has no separate transient budget to cap it

Note: #8195 depends on this PR.

Type of Change

Bug fix

Testing

Unit tests for every classifier code, cause chains, the user-cancel exclusion, and the socket-code query gate; retry helper delays, ceilings, and the permanent path; each adoption site (processing catchError + pending/failed status, sync auto-disable exemption through executeSync, 0021 page retries, file search)
Mutation-checked each branch (classifier codes, query gate, helper ceilings/pacing, the processing callback/throw/catchError wiring, the service pending write, the sync branch and builder, the 0021 retry gate): every mutation turned its named tests red
bun run lint, bun run check:audits (47/47), check:api-validation, type-check (apps/sim, packages/utils, packages/db)

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…etries - classifyDatabaseFailure in @sim/utils/errors: capacity / conflict / connection / permanent - shared Trigger catchError decision helper with minute-scale database backoff - workspace file search, document processing, connector sync, and the 0021 backfill adopt it - a scheduled document retry leaves the document pending instead of failed - a transient database failure no longer counts toward connector auto-disable

vercel · 2026-09-23T05:59:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 23, 2026 7:21am UTC

greptile-apps · 2026-09-23T06:02:09Z

Confidence Score: 5/5

The PR appears safe to merge; the previous findings are resolved and the latest changes introduce no new actionable issue.

Summary

This PR centralizes transient database-failure classification and applies bounded, database-aware retry behavior across background document processing, workspace search, connector synchronization, and projection backfills.

Distinguishes capacity, conflict, connection, cancellation, and permanent database failures.
Adds bounded minute-scale retry policies for background tasks.
Preserves uploaded documents as pending during database deferrals and adds an outbox watchdog for lost scheduled retries.
Prevents transient database outages from advancing connector auto-disable counters.
Adds focused unit, integration, and PostgreSQL coverage.
The changes since the previous review replace the connector retry tests’ newly added dynamic imports with static imports, resolving the prior repository-rule finding.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Background operation fails] --> B[classifyDatabaseFailure]
  B -->|Capacity / conflict / connection| C[Transient database failure]
  B -->|Cancellation / other permanent error| D[Normal failure handling]
  C --> E{Subsystem}
  E -->|Document processing| F[Record pending and deferred-until]
  F --> G[Schedule Trigger retry]
  F --> H[Commit uploaded-document watchdog]
  H --> I{Retry observed after grace?}
  I -->|Live or unknown| J[Recheck hourly within recovery window]
  I -->|No live run| K[Mark failed so user Retry is available]
  E -->|Connector sync| L[Keep failure counter unchanged]
  L --> M[Retry based on progress or zero-progress streak]
  E -->|Search / backfill| N[Apply bounded background or page retry]
  D --> O[Use ordinary retry ceiling or permanent failure path]

_{Reviews (7) · Last reviewed commit: "test(knowledge): import the connector re..."}

cubic-dev-ai

All reported issues were addressed across 15 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…ver members mode - connector and members-mode syncs climb the failure ladder by the failed-run streak read from their run logs; database failures never advance the auto-disable counter - classify 25P03 as capacity and ECONNREFUSED/EHOSTUNREACH/ENOTFOUND/EAI_AGAIN (with a query) as connection - in-process document processing keeps recording database failures as failed, documented - offer Retry for a pending document whose dispatch or deferred retry is past the retry API's grace - correct the processing task's retry-ceiling comment

waleedlatif1 · 2026-09-23T06:18:27Z

@greptile

waleedlatif1 · 2026-09-23T06:18:28Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T06:18:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

…fields Restores the document list, contract, serializer, and Retry condition to staging; Retry stays failed-only.

cubic-dev-ai

All reported issues were addressed across 27 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…try never ran - the deferral write and a deferred-retry-check outbox event commit together for uploaded documents - the check fails the document only if the same deferral is still pending, overdue past the queue grace, and no run is live; otherwise it rechecks without spending an attempt, bounded by the recovery window - a database deferral stamps processingQueuedAt when unset, so a dispatch cannot claim the row as never-queued during the retry window

waleedlatif1 · 2026-09-23T06:40:31Z

@greptile

waleedlatif1 · 2026-09-23T06:40:32Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T06:40:38Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 26 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…se failures A transient database failure while checking postpones the check without spending an attempt, paced by how overdue it is and capped at the recheck interval, until a terminal bound past the recovery window; other errors, and any error past that bound, spend attempts as before.

…al, strict query shapes - the database-failure streak counts only failed runs that made no progress; a run that wrote documents (or completed a member) retries in minutes - ten zero-progress database failures in a row log an alertable error; the connector is never disabled for them - the classifier's query check matches only Drizzle's query error and the postgres.js query error shapes, so a client error carrying its own query stays a source failure

waleedlatif1 · 2026-09-23T06:56:12Z

@greptile

waleedlatif1 · 2026-09-23T06:56:13Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T06:56:18Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 26 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…river connection errors - the connector run-history read runs under short statement and lock timeouts and falls back to this run alone - members-mode progress counts lifecycle purges (docs_purged in the run log) - postgres.js connection codes count only on an error the driver built, or under a query error; the driver's query signature is its four own properties, which a refused connection carries with no SQL yet - real-error PostgreSQL test for terminated transactions, refused connections, and an ending pool, wired into CI - knowledge-processing tests use the static task import

waleedlatif1 · 2026-09-23T07:11:05Z

@greptile

waleedlatif1 · 2026-09-23T07:11:07Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T07:11:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 28 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

… query as connection failures

waleedlatif1 · 2026-09-23T07:15:37Z

@greptile

waleedlatif1 · 2026-09-23T07:15:38Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T07:15:49Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-23T07:21:27Z

@greptile

waleedlatif1 · 2026-09-23T07:21:28Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T07:21:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

vercel Bot deployed to Preview September 23, 2026 06:00 View deployment