Conversation
github-actions
Bot
added
sqllogictest
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25602 +/- ##
==========================================
- Coverage 82.45% 82.45% -0.01%
==========================================
Files 1140 1142 +2
Lines 436589 436973 +384
Branches 436589 436973 +384
==========================================
+ Hits 359996 360301 +305
- Misses 54838 54892 +54
- Partials 21755 21780 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sunchao
reviewed
Sep 22, 2026
sunchao
left a comment
sunchao
left a comment
Member
There was a problem hiding this comment.
Thanks for working on this. I compared adbb3ce with base 714956b and found two regressions, detailed inline. Both have executable base/head reproductions. The 109 pruning unit tests and 64 ordinary SQL cases per revision passed; the separate nullable-CASE and memory-limit reproducers below fail only on head.
Comment on lines
+1875
to
+1880
| // A missing `ELSE` yields NULL, which never matches, so it adds nothing. | ||
| return case | ||
| .when_then_expr() | ||
| .iter() | ||
| .map(|(_, then)| then) | ||
| .chain(case.else_expr()) |
Member
There was a problem hiding this comment.
[P1] Preserve the implicit NULL branch during full-match inference
Omitting the missing ELSE is conservative for ordinary filtering, but this rewriter is also used to build the inverse predicate that proves an entire Parquet row group matches. That makes this change return incorrect rows when datafusion.execution.parquet.pushdown_filters=true.
I reproduced this with required Int64 columns (a,b), two one-row groups containing (2,1) and (1,2), and:
SELECT * FROM t
WHERE NOT (CASE WHEN a = 1 THEN false END) AND b > 0
ORDER BY a;Base 714956b3 returns only (1,2); head adbb3ce4 also returns (2,1), although its predicate result is NULL. With LIMIT 1 instead of ORDER BY, head returns the invalid (2,1). Disabling filter pushdown restores the correct result.
The forward rewrite falls back for NOT CASE, so it still permits full-match inversion. The inverse exposes CASE ... OR b <= 0; dropping the implicit NULL branch lets it prune both groups, which are then marked fully matched and skip row filtering. Head reports two fully matched groups and row_filter_skipped_fully_matched=1.
Could we keep CASE expressions without an explicit ELSE conservatively unhandled here, and add an end-to-end regression test? The join-generated CASE already has an explicit ELSE. Merely setting has_filter_semantics_only inside this branch would miss the forward NOT CASE path, which never descends into it.
Contributor
Author
There was a problem hiding this comment.
Good point, added a bail-out for that case as well as an SLT.
Comment on lines
+3113
to
+3114
| let pruning_bitmap = match (left_values.as_slice(), bounds.as_ref()) { | ||
| ([keys], Some(bounds)) if !keys.is_empty() => bounds |
Member
There was a problem hiding this comment.
[P2] Skip bitmap construction when dynamic filtering is inactive
bounds can exist solely for perfect-hash-join candidacy even when should_compute_dynamic_filters is false. This branch still constructs and reserves a pruning bitmap before those bounds are cleared below, so a join with dynamic filtering disabled can now fail for memory that provides no pruning benefit.
Using a native CollectLeft join with 151 Int64 build keys i * 10_000 (i = 0..151), probe keys [0, 500_000, 1_500_000], and enable_join_dynamic_filter_pushdown=false (otherwise default configuration), base 714956b3 succeeds in a 100,000-byte memory pool with 6,220 bytes reserved. Head adbb3ce4 fails with ResourcesExhausted requesting another 131,072 bytes. At a 1,000,000-byte limit both return the exact expected rows, but head reserves 137,292 bytes. I also reproduced the same failure with a Full join.
Could we gate this bitmap construction on should_compute_dynamic_filters and cover the disabled-filter case with a memory-limit regression test?
Contributor
Author
There was a problem hiding this comment.
Nice catch. I also noticed that the same problem happens for the InList path too.
So instead of guarding just the (bit)map path with should_compute_dynamic_filters, i made it guard both by extending the PushdownStrategy::Empty arm with || !should_compute_dynamic_filters; let me know if you see issues with this.
Also a unit test added.
Map the build side's keys onto a fixed-size bitmap over their range and test container min/max against it, so a scan can skip containers whose values fall in the gaps between keys. The bitmap is bounded at 128 KiB regardless of build side size, and one whose buckets are all set is discarded, so a contiguous key set costs nothing. The key range and distinct count already exist for the bounds predicate, so a dense key set is ruled out before allocating. Integer keys only; other types go unpruned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 23, 2026 07:20
6c48fab to
a6d5391
Compare
jayzhan211
reviewed
Sep 23, 2026
Comment on lines
+3129
to
+3133
| if let Some(bitmap) = pruning_bitmap.as_ref() { | ||
| // Held for the join's lifetime, so charge it like the maps. | ||
| reservation.try_grow(bitmap.size())?; | ||
| metrics.build_mem_used.add(bitmap.size()); | ||
| } |
Contributor
There was a problem hiding this comment.
The bitmap is only a pruning aid, but try_grow(...)? turns a failed reservation into a query error. A join with less than 128 KiB of headroom per build partition used to succeed and now fails with ResourcesExhausted. Drop the bitmap instead:
Suggested change
| if let Some(bitmap) = pruning_bitmap.as_ref() { | |
| // Held for the join's lifetime, so charge it like the maps. | |
| reservation.try_grow(bitmap.size())?; | |
| metrics.build_mem_used.add(bitmap.size()); | |
| } | |
| // Held for the join's lifetime, so charge it like the maps; it is | |
| // optional, so skip it rather than fail when the pool is full. | |
| let pruning_bitmap = pruning_bitmap.filter(|bitmap| { | |
| let ok = reservation.try_grow(bitmap.size()).is_ok(); | |
| if ok { | |
| metrics.build_mem_used.add(bitmap.size()); | |
| } | |
| ok | |
| }); |
Contributor
|
run benchmarks |
Contributor
|
I think we need benchmark number |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Avoid scanning redundant files/row-groups/pages in the probe side of hash joins, based on the values dictated by the build side.
What changes are included in this PR?
KeyRangeBitmapimplementation, which maps the build-side values array into a finite-sized bitmap bucket, and can answer probing questions for certain rangesbuild_predicate_expressionto build the associated pruning expression fromHashTableLookupExprusing the newKeyRangeBitmapPruningExpr, which implementsPhysicalExpron top ofKeyRangeBitmapbuild_predicate_expressionso that it now pushes down pruning forCaseExprs, since that also unlocks the partitioned hash-join scenario this pr targetsWhat is the testing strategy for this PR?
A number of unit tests added, and one SLT added.
Also verified on the query shape that motivated the original issue
Are there any user-facing changes?