LuciferYang commented on PR #58409: URL: https://github.com/apache/spark/pull/58409#issuecomment-5524998839
Thanks @peter-toth, and thanks for the `DataFrameReader.schema(String)` pointer, that settles the duplicate-name thread better than my measurements did: `schema(String)` is `schema(StructType.fromDDL(...))`, so the four runs I posted were already the `schema(StructType)` path, and the narrowing had nothing left to narrow. No assertion added. All three are in at `01a676f2575`. **8.** You are right, and this was the gap I talked myself out of. Both cases read every column of a two-column file, so the merged projection was the whole data schema and `positionsInDataSchema` returned the identity, which is what the empty array already means. Three columns now, with the two subqueries over `b` and `c`, so the union is a proper subset and the answer `[100, 1000]` is only reachable through the mapping. Re-measured both ways at this head: with the mapping disabled both fail with `[10, 100]`, with the gates put back instead the V1 one fails with a scan per column and the V2 one at the capability assertion. So the pair pins both halves now rather than one. **9.** Pinned, same three configurations as the twin. `isProjectionSensitiveRead` is true for a non-strict read whatever the parser does, so the test rested on two defaults its twin already declines to rest on. **10.** Added, close to your wording: the comment now says `positionalFieldMatching` resolves a column against its position in the data schema rather than in the projection, so widening the projection does not move a column's Avro field either. That is the half `FileTable` names second, and with the paragraph I removed it was unrecorded. The description carries the new shape and both mutation results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
