fix(knowledge): bound every connector-lease ACL transaction to one short page by waleedlatif1 · Pull Request #8195 · simstudioai/sim

waleedlatif1 · 2026-09-23T06:22:46Z

Summary

Every connector-lease transaction that assigns document.acl is one short page: bounded by its documents' chunk rows (PROJECTION_ROW_BATCH_SIZE, the constant detachment already uses, moved to sync-limits; a document larger than the cap gets a page alone), under LEASE_PAGE_LOCK_TIMEOUT_MS (15 s) / LEASE_PAGE_STATEMENT_TIMEOUT_MS (30 s). The ACL trigger rewrites every filled search projection row of a document whose ACL is assigned, so a transaction spanning many batches held the connector row (and the table lock a migration queues behind) across unbounded fan-out, and a timeout rolled back every batch that had landed
Lease proved last: leaseTransaction, withMemberLease, the content pass's lease transaction and the stale-member sweep write first and take the connector row lock as their final statement (a lost lease still rolls the page back). No page waits on document rows — which a processing commit may hold for its whole write — while holding a knowledge_connector lock. Only ACL pages carry the page bounds; other lease bodies keep the role's timeouts
persistDocumentAcls: the evidence refresh returns what it matched; only the rest is read and written as changes, in row-bounded pages. The unchanged-refresh vs changed-write split and the unresolved-evidence guard are unchanged
restoreWorkspaceDocumentAcls: out of the completion transaction, paged, mode-guarded, and run only while accessRewritePending is set (the mode switch and ACL-resetting edits are the only sources of that drift; every other writer of a workspace-mode document writes the workspace ACL)
Every ACL page locks its documents and rereads their chunk counts before writing, so a reprocess that lands between planning and writing cannot overfill a page (a processing commit holds its document row while it replaces chunks and sets chunk_count)
rewriteConnectorAcls and the restore share one single-pass keyset walker ((external_id, id) windows of 500); evidence-only rows are cleared without assigning acl
Member engine: listing observations and the read watermark land per 500 documents, then only documents whose ACL differs from their observers are rematerialised (rematerializeDocumentAcls, row-bounded pages). Observation changes that must commit with their ACLs (unseen removal, membership rewrite, stale sweep) take at most 25 documents and one page of projection rows; the invariant they keep is per document. The content completion's connector-wide count runs before the lease transaction
disableMemberSync: suspends members (which revokes reads at once: a reader needs an active member's observation as well as ACL overlap), revokes ACLs in guarded row-bounded pages, and flips the connector to disabled only after the last page, in one statement that also proves the lease. It was one unguarded SET acl = '{}' under FOR UPDATE that could time out and retry forever. An interrupted or out-of-budget run resumes on the next run; a disable that fails part-way goes through the normal failure path (log failed, lease released) instead of leaving the connector running
Stale-member sweep: row-bounded pages with a 60 s tick budget checked before every page; a page that fails transiently (the shared getTransientDatabaseFailure classifier) defers that connector to the next tick and the others are still swept; the scheduler logs a failed sweep and still dispatches

Round trips per 100k unchanged documents

Admin permission write: before this PR 1 transaction, ~4,200 statements under the connector share lock; first revision of this PR 4,200 transactions; now 200 transactions of 3 statements, no change reads
Member listing observe + materialise: first revision 4,000 transactions of 5 statements; now 200 observation transactions + 200 lock-free reads, no materialise transactions
Post-listing rematerialisation: before 1 transaction over every seen document; now 200 lock-free reads, no transactions
Workspace restore on a healthy connector: first revision 200 window reads per sync; now none

Type of Change

Bug fix

Testing

connector-lease-pages.integration.ts (real migrated PostgreSQL 17, wired into both PostgreSQL CI jobs), 18 tests: every ACL assignment is recorded with its transaction and timeouts; each transaction stays within one page under the bounds; a page and a member page waiting on a locked document row hold no knowledge_connector lock (pg_locks); a document above the row cap still lands alone; a lease lost between pages stops further writes while committed pages stay; the pending restore runs before completion and a healthy workspace sync never walks; listing, change-feed and membership rematerialisation page by rows; an interrupted disable resumes to disabled with every ACL empty; a failing disable closes as an ordinary failure; a stale sweep defers a connector a member run holds and still sweeps the others
Unit tests for paging, fence-last ordering, lease loss, the row pager, the refresh-returning skip, rematerialisation, sweep deferral/budget, the route still dispatching after a sweep failure, and the content completion count
Mutation-checked on file copies: every change (fence order in each lease path, bounds, lock timeout value, row cap and single-document minimum, refresh skip, lock-free pre-select, restore gate, count placement, sweep isolation/budget/busy-connector skip, disable failure path, route isolation, plus the earlier paging mutations) turns at least one test red
lib/knowledge + app/api/knowledge unit suites, the knowledge *.postgres.test.ts files, 0021 migration test, related knowledge integration suites on a freshly migrated database, lint, check:audits, type-check

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…ort page

vercel · 2026-09-23T06:22:48Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 23, 2026 8:36am UTC

greptile-apps · 2026-09-23T06:32:45Z

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issues remain, and all previous findings are resolved.

Summary

This PR bounds connector-lease ACL rewrites to short, projection-row-sized transactions while preserving lease fencing and resumability.

Pages ACL updates according to document chunk counts and applies lock and statement timeouts only to ACL-page transactions.
Proves connector leases after page writes so transactions do not hold connector locks while waiting for document rows.
Makes workspace restoration, member rematerialization, connector disabling, and stale-observation cleanup resumable across bounded pages.
Keeps scheduler dispatch running when stale-observation cleanup fails transiently.
Adds extensive unit and PostgreSQL integration coverage for page limits, lock ordering, lease loss, deadlines, and interrupted operations.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Select keyset window] --> B[Plan pages from chunk counts]
  B --> C[Heartbeat and check deadline]
  C --> D[Begin bounded transaction]
  D --> E[Lock document rows]
  E --> F[Re-read chunk counts]
  F --> G[Write one ACL page]
  G --> H[Prove connector lease last]
  H --> I[Commit page]
  I --> J{More documents and budget remains?}
  J -->|Yes| A
  J -->|No| K[Finish or retain resumable pending state]

_{Reviews (6) · Last reviewed commit: "test(knowledge): expire the restore budg..."}

…les, update ACL test callers

…e-transactions-bounded

…ction rows, and skip unchanged work

waleedlatif1 · 2026-09-23T07:19:54Z

@greptile

waleedlatif1 · 2026-09-23T07:19:56Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T07:20:02Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…e-transactions-bounded # Conflicts: # apps/sim/lib/knowledge/connectors/member-sync-engine.ts

… locked chunk counts, and check the sweep budget per page

waleedlatif1 · 2026-09-23T07:45:27Z

@greptile

waleedlatif1 · 2026-09-23T07:45:28Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T07:45:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 21 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

… pending rewrites until they finish

…s on any provisioned database

waleedlatif1 · 2026-09-23T08:25:17Z

@greptile

waleedlatif1 · 2026-09-23T08:25:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T08:25:24Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 21 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-23T08:29:41Z

@greptile

waleedlatif1 · 2026-09-23T08:29:42Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T08:29:48Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 21 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…e next window

waleedlatif1 · 2026-09-23T08:36:00Z

@greptile

waleedlatif1 · 2026-09-23T08:36:02Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-23T08:36:10Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 21 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

fix(knowledge): bound every connector-lease ACL transaction to one sh…

bb4623a

…ort page

vercel Bot deployed to Preview September 23, 2026 06:24 View deployment

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts Outdated

waleedlatif1 added 2 commits September 22, 2026 23:41

fix(knowledge): isolate stale sweep lock failures, close failed disab…

0a7dd0c

…les, update ACL test callers

Merge remote-tracking branch 'origin/staging' into fix/connector-leas…

55986a3

…e-transactions-bounded

vercel Bot deployed to Preview September 23, 2026 06:43 View deployment

waleedlatif1 mentioned this pull request Sep 23, 2026

fix(jobs): classify transient database failures once for background retries #8194

Merged

6 tasks

fix(knowledge): prove connector leases last, page ACL writes by proje…

2d3f1f3

…ction rows, and skip unchanged work

vercel Bot temporarily deployed to Preview September 23, 2026 07:13 Inactive

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-observations.ts

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-sync-engine.ts

Comment thread apps/sim/lib/knowledge/connectors/member-observations.ts

Comment thread apps/sim/lib/knowledge/connectors/sync-persistence.ts

waleedlatif1 added 2 commits September 23, 2026 00:39

Merge remote-tracking branch 'origin/staging' into fix/connector-leas…

c09f9a3

…e-transactions-bounded # Conflicts: # apps/sim/lib/knowledge/connectors/member-sync-engine.ts

fix(knowledge): row-bound every observation page, size ACL pages from…

d0d2bcd

… locked chunk counts, and check the sweep budget per page

vercel Bot deployed to Preview September 23, 2026 07:41 View deployment

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/sync-engine.ts Outdated

Comment thread apps/sim/lib/knowledge/__integration__/connector-lease-pages.integration.ts

Comment thread apps/sim/lib/knowledge/connectors/member-observations.ts

fix(knowledge): stop every ACL page walker at the run budget and keep…

c157ef1

… pending rewrites until they finish

vercel Bot temporarily deployed to Preview September 23, 2026 08:04 Inactive

test(knowledge): seed filled projection rows so the fan-out test hold…

76db356

…s on any provisioned database

vercel Bot temporarily deployed to Preview September 23, 2026 08:18 Inactive

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-observations.ts Outdated

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

fix(knowledge): check the page deadline after the lease heartbeat

2837d6a

vercel Bot temporarily deployed to Preview September 23, 2026 08:29 Inactive

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

test(knowledge): expire the restore budget at the heartbeat before th…

ac7c2bc

…e next window

vercel Bot temporarily deployed to Preview September 23, 2026 08:36 Inactive

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

waleedlatif1 merged commit 624793f into staging Sep 23, 2026
34 checks passed

waleedlatif1 mentioned this pull request Sep 23, 2026

v0.8.58: desktop app catchas, insights dashboard, kb fixes #8200

Merged

waleedlatif1 deleted the fix/connector-lease-transactions-bounded branch September 23, 2026 15:28

Conversation

waleedlatif1 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Round trips per 100k unchanged documents

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 23, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 23, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

waleedlatif1 commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

waleedlatif1 commented Sep 23, 2026 •

edited

Loading

vercel Bot commented Sep 23, 2026 •

edited

Loading

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading