fix: drop partition columns already present in the file schema of a ListingTable

[ShayanGho]

Which issue does this PR close?

• Closes Schema error after loading parquet stored with datafusion.execution.keep_partition_by_columns = TRUE #17420.

Rationale for this change

Parquet files written with datafusion.execution.keep_partition_by_columns = true physically contain the partition column. Reading them back as a hive-partitioned table then fails:

SET datafusion.execution.keep_partition_by_columns = TRUE;
COPY (SELECT 1 AS id, 'a' AS grp) TO '/tmp/out' STORED AS PARQUET PARTITIONED BY (grp);
CREATE EXTERNAL TABLE t2 STORED AS PARQUET PARTITIONED BY (grp) LOCATION '/tmp/out';
SELECT * FROM t2;
-- Schema error: Schema contains duplicate qualified field name t2.grp

DESCRIBE t2 shows grp twice: once with the type inferred from the files and once with the partition column type. The same error is hit without PARTITIONED BY (the factory infers hive partitions from the directory names by default) and when querying the directory path directly (SELECT * FROM '/tmp/out/').

ListingTable::try_new unconditionally appends every configured partition column to the inferred file schema. When the file schema already has a field with that name, the table schema ends up with a duplicate. CREATE EXTERNAL TABLE with an explicit column list already works because the factory projects the partition columns out of the provided schema; the inferred-schema path had no equivalent.

What changes are included in this PR?

ListingTable::try_new now drops any file-schema field whose name matches a configured partition column before appending the partition columns. The partition value continues to be read from the path and keeps the declared (or inferred) partition column type, matching the existing behaviour of the explicit-schema path. The trimmed schema is also what create_file_source and the file-schema fingerprint use, so the physical scan and the table schema agree.

What is the testing strategy for this PR?

• New unit test test_partition_column_present_in_file_schema_is_not_duplicated in datafusion/core/src/datasource/listing/table.rs builds a ListingTable whose file schema already contains the partition column and asserts the table schema lists it once, with the partition type. It failed before the fix with ["a", "pid", "pid"].
• Two new cases in datafusion/sqllogictest/test_files/copy.slt reuse the existing keep_partition_by_columns output directory and create a table over the whole directory, once with an explicit PARTITIONED BY and once relying on partition inference. Both failed before the fix with the duplicate field error.
• The existing parquet_overlapping_columns test in datafusion/core/tests/sql/path_partition.rs asserted that a partition column sharing a name with a file column must raise an error. This PR intentionally changes that behaviour to match how CREATE EXTERNAL TABLE with an explicit column list already handles the overlap. The test now asserts the new behaviour: the column appears once, with the partition type, and its values come from the path.

Are there any user-facing changes?

Reading hive-partitioned Parquet data whose files also contain the partition column now works instead of raising a schema error.

Conflict policy, made explicit: when a file column and a partition column share a name, the partition column's path-derived value and declared type take precedence. The file column is ignored, and its values are not validated against the path value.

No API changes.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.04478% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.48%. Comparing base (9082d6b) to head (8040c42).
⚠️ Report is 174 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/core/src/datasource/listing/table.rs	89.79%	0 Missing and 5 partials ⚠️
datafusion/catalog-listing/src/table.rs	94.44%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25212      +/-   ##
==========================================
+ Coverage   81.88%   82.48%   +0.60%     
==========================================
  Files        1133     1140       +7     
  Lines      424522   437681   +13159     
  Branches   424522   437681   +13159     
==========================================
+ Hits       347623   361032   +13409     
+ Misses      56285    54845    -1440     
- Partials    20614    21804    +1190     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ShayanGho]

Hello @alamb, when you get a chance can you please review this PR?
it closes #17420 (schema error reading partitioned Parquet written with keep_partition_by_columns). One behavioral change worth checking, when a file column and a partition column share a name, I chose path-derived partition column's path-derived value

[kosiew]

@ShayanGho,

Thanks for working on this. The fix looks good to me, and the regression coverage does a nice job covering both explicit and inferred Hive partition discovery. I just have one small suggestion to strengthen the unit coverage.

[kosiew]

Could we add a small unit case with two overlapping partition columns, with one of them appearing in the middle of the physical schema? It would be useful to assert that the final order has all remaining file fields first, followed by the partition fields in their configured order. The current single trailing-column case covers deduplication well, but does not exercise the projection and ordering behavior introduced by this filter.

[ShayanGho]

Thank you, added a test for two overlapping partition columns with one mid-schema.