Skip to content

Fix EliminateLimit not eliminating limits under LogicalPlan::Extension nodes. - #25577

Merged
alamb merged 6 commits into
apache:mainfrom
coralogix:fix-limit-elim
Sep 23, 2026
Merged

alamb merged 6 commits into
apache:mainfrom
coralogix:fix-limit-elim

Conversation

@masonh22

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Rationale for this change

#22298 updated the logical plan optimizer to apply optimization rules in place to avoid Arc unwrap/rewraps with a new map_children_mut() function. This function unconditionally updates the children of all logical plan nodes except for LogicalPlan::Extension nodes. For Extension nodes, it only updates the children if the mapping function returns true for any of the children, signifying that a child was updated.

When the EliminateLimit rule prunes a limit that has 0 skip and fetch, it modifies the node but does not return that it modified anything. This is fine for every case except for if the limit is underneath a LogicalPlan::Extension node, since it that case we ignore the output from the rule.

What changes are included in this PR?

This change adds a failing test, a fix to EliminateLimit, and changes map_children_mut() to unconditionally update the children of LogicalPlan::Extension nodes. The change to map_children_mut() could be left out, but I wouldn't be surprised if there are other optimization rules that modify the node without returning transformed=true, so this is done to avoid finding and fixing all of those cases.

What is the testing strategy for this PR?

I added a test that replicates the issue and fails without my fixes.

Are there any user-facing changes?

No

@github-actions github-actions Bot added the optimizer Optimizer rules label Sep 21, 2026
@codecov-commenter

codecov-commenter commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.45%. Comparing base (0576a0b) to head (9fcc605).
⚠️ Report is 24 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/optimizer/src/eliminate_limit.rs 78.57% 0 Missing and 3 partials ⚠️
datafusion/optimizer/src/optimizer.rs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25577      +/-   ##
==========================================
+ Coverage   82.42%   82.45%   +0.03%     
==========================================
  Files        1139     1140       +1     
  Lines      435372   436600    +1228     
  Branches   435372   436600    +1228     
==========================================
+ Hits       358845   360002    +1157     
- Misses      54826    54836      +10     
- Partials    21701    21762      +61     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@masonh22
masonh22 marked this pull request as ready for review September 21, 2026 20:58
Comment thread datafusion/optimizer/src/eliminate_limit.rs

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me -- thank you @masonh22

Comment thread datafusion/optimizer/src/eliminate_limit.rs Outdated

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧑‍🍳 👌

@alamb
alamb enabled auto-merge September 23, 2026 14:39
@alamb
alamb added this pull request to the merge queue Sep 23, 2026
Merged via the queue into apache:main with commit 700ad9c Sep 23, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants