Skip to content

[FLINK-40740][pipeline-connector/fluss] Prevent duplicate split initialization during discovery - #4555

Open
fxbing wants to merge 1 commit into
apache:masterfrom
fxbing:fix/20260922-fluss-discovery-duplicate-splits
Open

fxbing wants to merge 1 commit into
apache:masterfrom
fxbing:fix/20260922-fluss-discovery-duplicate-splits

Conversation

@fxbing

@fxbing fxbing commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Prevent periodic Fluss source discovery from initializing and assigning the same bucket splits more than once when a previous initialization callback has not completed. Reassigning an earliest-offset split can rewind an active reader and produce duplicate records. See FLINK-40740.

Brief change log

  • Track initializing physical table paths before scheduling asynchronous split creation, and release only the current batch in its coordinator callback.
  • Preserve initialization failures and the existing checkpoint format; unfinished initialization remains discoverable after recovery.
  • Add deterministic regressions for overlapping discovery, complete bucket assignment, independent partitions, pending readers, initialization failures, and recovery during initialization.

Verifying this change

  • The overlapping-discovery regression fails against the original implementation: three buckets produce nine split assignments instead of three.
  • FlussSourceEnumeratorTest (16 tests) and FlussSourceEnumStateSerializerTest (1 test) pass on both Java 11 / Flink 1.20.3 and Java 17 / Flink 2.2.0, with no failures, errors, or skipped tests.
  • Spotless and Checkstyle pass. Full pipeline/CLI end-to-end tests were not run.

Documentation

  • Does this pull request introduce a new feature? No.
  • If yes, how is the feature documented? Not applicable.
Was generative AI tooling used to co-author this PR?
  • Yes — Codex CLI; Claude and GLM assisted with analysis, implementation, and review.

Generated-by: Codex CLI 0.155.1

…alization during discovery

Track initializing physical table paths before scheduling split creation and
release each request in its coordinator callback without changing checkpoints.
Add regressions for overlapping discovery, pending readers, partitions and recovery.

Generated-by: Codex CLI 0.155.1
@loserwang1024
loserwang1024 self-requested a review September 22, 2026 09:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant