feat(knowledge): project document ACL and chunk changes asynchronously

[waleedlatif1]

Summary

• Document ACL/source changes and chunk writes no longer have to rewrite search projection rows inside the writer's transaction. Every such write now marks its document in a new knowledge_projection_dirty table (one upsert, generation bump), and a new knowledge projector converges embedding_search, the GIN keyword projection and the Tin projection per document, in short transactions of at most 250 chunk rows, several documents at once, removing a mark only on the generation it read
• Writers only skip the synchronous triggers when their transaction declares sim.projection_mode = 'async', which the app does only while the new knowledge-async-projection flag is on. Two places pick the mode, each reading the flag once before its transaction opens and requesting a pass once it commits: processDocumentAsync and aclPageTransaction (every connector-lease ACL page runs through it). Everything else, and every release older than this one, keeps writing projection rows itself. Flag off is the rollout default
• Marks are written in both modes on purpose: during the cutover old Trigger workers, the flag flip window and a rollback can all have a synchronous writer and a projector pass on the same document at once, and the generation bump is what stops a pass that read the document earlier from settling over the newer write
• Search decides a marked document's rows on the document itself (the same path rows the source/ACL fill had not reached already take), so a pending projection never admits a revoked grant, and never hides a new grant from a statement that reaches the row. One gap is accepted: per-source vector walks and the sliced scope reach rows by the source on the row (that is what keeps them on their per-source partial HNSW indexes), so a document that moved to another source joins that source's ranking only after the pass rewrites its rows. Until then it can be missing there; it is never shown where it is not readable. Deletes still cascade synchronously, and hydration re-checks content, enabled state and the full ACL, so disables, soft deletes and connector removal stay immediate
• The projector runs as a Trigger.dev task: requested debounced after each write, plus a one-minute cron sweep (Helm and Docker Compose) as the backstop. The sweep first checks whether there is anything to do (any mark, or, with the fill on, any unfilled row: each an index probe that is empty once caught up) and starts no pass on an idle deployment. Without Trigger.dev the pass runs in-process, fire-and-forget and one at a time, as document processing does there. Each pass has its own budget and stops between pages, leaving unfinished marks for the next pass
• The projection source/ACL backfill task and script are removed; the fill is folded into the projector behind its own knowledge-projection-fill flag, keeping at most 100 documents marked at once so fresh writes never queue behind much of it. Its cursor only moves past rows whose documents it marked, so a window with more unfilled documents than room is picked up where it stopped
• Migrations are expand-only: one new table (drizzle 0378) and script migration 0024. 0016, 0019 and 0021 are unchanged, and 0022/0023 change only in comments, so every migration up to 0023 is fully synchronous and every write between them succeeds on its own. 0024 alone adds the mode guards and the marking, re-creating the projection triggers with the guard in the same short, lock-timeout-retried transaction that installs the marks, so a guard never exists without the marks. db:push runs 0024 last

Throughput

• One pass runs at a time across the fleet (queue concurrency 1) and projects up to KB_CONFIG_PROJECTION_CONCURRENCY documents at once (default 8), each worker on its own connection named sim-knowledge-projector. A round opens only as many workers as there are marks (a bounded count), so a pass over a few documents holds a few connections
• A single queue keeps the projector's database footprint to one known number of connections; parallel passes would split the same oldest marks at the advisory locks anyway, just with an unbounded number of connections. Measured locally, throughput rose to 938 rows/s at 8 workers without deadlocks
• The ceiling is concurrency times per-document speed. To raise it, raise KB_CONFIG_PROJECTION_CONCURRENCY (each step is one more database connection for the length of a pass). The backlog shows as rows in knowledge_projection_dirty and their oldest marked_at

Rollout

1. Deploy with both flags off. Every writer behaves as today, plus one mark per changed document; the projector only verifies (measured 0 rows rewritten, about 6ms per 200-chunk document, 0.3% of writer time)
2. Once Trigger.dev is promoted and runs on the previous version have drained, turn on knowledge-async-projection
3. Turn on knowledge-projection-fill when there is I/O headroom to fill projection rows written before they carried a source and ACL

Rollback

• Turning knowledge-async-projection off is immediate: writers go back to writing projection rows themselves, and the projector keeps converging what is already marked
• Rolling the app back to a release before this one needs more than the flag. Old releases do not read marks: their search trusts the source and ACL on the projection rows and has no path for a chunk without rows, so every write that deferred its rows must be projected first, ACL changes as much as chunk commits (a deferred revocation still on the rows would be admitted by an old reader). The immediate rollback applies to the flag only
  1. Turn knowledge-async-projection off and note the time T
  2. Wait until 15 minutes past T (the flag cache is about 30s; a processing attempt that read the flag just before T can commit up to its 10-minute limit later)
  3. Wait until SELECT count(*) FROM knowledge_projection_dirty WHERE marked_at < T + interval '15 minutes' returns 0. A mark keeps the time it was first created while writes bump it, so every mark a deferred write touched is covered; marks created after that point come from synchronous writers whose rows are already written
  4. Roll the app back

Cleanup follow-up (after the flag has been on everywhere)

• One script migration that drops the synchronous branches: the WHEN guards and sync_embedding_search / sync_embedding_keyword_search / sync_embedding_keyword_tin / set_projection_source_acl triggers, and the fan-out in sync_projection_source_acl (keeping only the mark)
• Remove sim.projection_mode, DEFER_KNOWLEDGE_PROJECTION, the knowledge-async-projection flag and KNOWLEDGE_ASYNC_PROJECTION
• Collapse the row-bounded ACL paging (pagesByProjectionRows, lockProjectionPage, writeProjectionPages) to plain pages of ACL_CHANGE_BATCH_SIZE documents; the lease-last fence and deadlines stay

Type of Change

• New feature

Testing

• Postgres integration (both modes): revoke refused and grant served before the projector runs on the vector and keyword rankings; moved documents; disabled and deleted chunks invisible at once; re-mark during a pass survives; document deleted mid-pass; bounded pages; passes started together split the work; an async connector ACL page writes no projection rows
• processDocumentAsync with the flag on through the real service: the commit writes the chunks, a content mark and no projection row, requests a pass, and the pass writes every projection row with the document's source and ACL and settles the mark
• Fill convergence with more unfilled documents than the fill may mark at once: every document is filled
• Migration ordering: through 0023 no trigger carries the mode guard, no mark function exists, and even a writer that asks for the asynchronous mode fans out synchronously; after 0024 every guarded trigger carries the guard and changes mark their document
• Stale page under a concurrent synchronous ACL change: the projector page reads the document, a synchronous writer changes the ACL and commits while the page waits on its row locks, the page writes the older values; the writer's generation bump keeps the mark, search decides on the live ACL, and the next pass converges
• Mutation checks, each failing its test: the reader's dirty branch off, the generation compare dropped, the mode not declared in connector ACL pages or in the processing commit, the pass deadline ignored, the fill cursor moved past unmarked documents, 0021's body calling the mark function, and a failed worker cutting the others off
• Local measurements: a 1,000-chunk commit took 9,956ms with the synchronous triggers and 125ms deferred; an ACL change on that document took 2,023ms vs 2ms; projector throughput 254 / 448 / 804 / 938 rows/s at 1 / 2 / 4 / 8 workers with no deadlocks; the insert mark trigger added no measurable time and no temporary files to a 1,000-chunk insert; the filled search fast path keeps the same plan and timing with no marks
• lint, type-check, check:audits, check:migrations, check:api-validation, docs-manifest:check, unit suites, and the CI Postgres steps

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 23, 2026 10:15am UTC

[cubic-dev-ai]

All reported issues were addressed across 57 files

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding blocking findings or new changes since the previous review.

Summary

This PR introduces asynchronous convergence of knowledge search projections while preserving synchronous behavior as the default rollout mode.

• Adds a generation-based dirty-document queue and a bounded, concurrent projector for vector, keyword, and Tin projection rows.
• Makes search consult live document ACL/source state while marked projections are pending.
• Integrates projection requests with document processing, connector ACL transactions, Trigger.dev, cron, Helm, and Docker Compose.
• Folds the previous source/ACL backfill into the projector behind a dedicated feature flag.
• Adds expand-only database and script migrations plus extensive unit and PostgreSQL integration coverage.

Diagram

sequenceDiagram
  participant W as Knowledge writer
  participant DB as PostgreSQL
  participant Q as Projection scheduler
  participant P as Knowledge projector
  participant S as Search

  W->>DB: Commit document/chunk/ACL changes
  W->>DB: Upsert dirty mark and bump generation
  alt Asynchronous projection enabled
    W-->>DB: Skip synchronous projection fan-out
  else Default synchronous mode
    W->>DB: Update projection rows synchronously
  end
  W->>Q: Request debounced projection pass
  Q->>P: Start bounded pass
  P->>DB: Read dirty document generation
  P->>DB: Converge projection rows in bounded pages
  P->>DB: Remove mark only if generation is unchanged
  S->>DB: Query projection candidates
  DB-->>S: Use live document ACL/source for marked documents

Loading

_{Reviews (4) · Last reviewed commit: "test(knowledge): count projection reques..."}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 54 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 54 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 54 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}