LuciferYang commented on PR #58340: URL: https://github.com/apache/spark/pull/58340#issuecomment-5461954729
Thanks @peter-toth. Finding 8 is taken, and it took Avro with it. **8.** `capabilities` is now `supportsScanMerging && hasStrictFileReads`, with the predicate as you wrote it. I kept `ignoreMissingFiles` in it to match `FileScanRDD.hasStrictFileReads` on the physical side, and the gate is evaluated per call rather than cached, so a table built before the configuration was set still answers for the read that is running. The new test builds the table and the temp view outside the configuration scope for exactly that reason, covers both spellings, and asserts `[45, 0]` alongside two distinct scans. **9.** The comment claimed the opposite of what the reader does, and it is rewritten around the property Avro does have: no record-level parse verdict. I also withheld the capability under `positionalFieldMatching`, so the contract holds for every source that declares it rather than only where a merge cannot make the answer worse. The option is read off the options map rather than through `AvroOptions`, whose constructor resolves `avroSchemaUrl` and would do I/O inside `capabilities`. The `AvroV2Suite` test now asserts the capability, the withholding under the option, and a projection-only merge. The underlying bug is SPARK-59108, with a runnable repro on the ticket, and the gate can go once it is fixed. ORC turned out to need no equivalent: `OrcUtils.requestedColumnIds` maps both positional cases through `dataSchema.fieldIndex(name)` and reports `canPruneCols = false`. On "No query result changes": with both gates in place it holds, so I dropped the migration-guide entry instead of qualifying the sentence. **10.** Corrected. Gap 4 now says CSV and JSON lose both shapes, differing projections and equal columns under differing filters, with your reason: `FileScan.equals` compares the normalized data filters. The V1 side is SPARK-59107, which pins four measured shapes, the three parse modes plus `ignoreCorruptFiles`. **11.** Fixed on all four. **12.** SPARK-57205's description is rewritten: the seam, the two exclusions, the two gates, the four gaps. Mutation checks on the current head: flipping the three sql/core overrides to `false` fails 8 of the 15 new tests, and dropping `&& hasStrictFileReads` fails the strictness test at its first capability assertion. CI is green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
