subhramit opened a new pull request, #58226: URL: https://github.com/apache/spark/pull/58226
Follow-up to https://github.com/apache/spark/pull/58225 Closes [SPARK-58946](https://issues.apache.org/jira/browse/SPARK-58946) ### What changes were proposed in this pull request? Changes `&&` to `||` in the `extension` option validation in `CSVOptions.scala:127`. Depends on SPARK-58945. Until that merges this path surfaces `INTERNAL_ERROR` rather than the intended error, so the new test will not pass on its own. ### Why are the changes needed? The error message states the extension is limited to exactly three letters, which requires rejecting anything not both three characters and all letters. `&&` rejects only values failing both clauses, so these are **accepted** today: | value | why it should be rejected | |---|---| | `abcd` | four letters | | `ab1` | three characters, not all letters | | `a` | one letter | | `a/b` | three characters, contains a path separator | The value flows into the output filename (`CSVWrite.scala:61`, `CSVFileFormat.scala:91`), so `a/b` places a path separator inside a filename component. ### Does this PR introduce _any_ user-facing change? Yes. Writes that previously succeeded with an out-of-contract extension now fail with `INVALID_PARAMETER_VALUE.EXTENSION`. The option was documented as three letters only, so affected jobs were relying on validation that did not match its stated contract. Valid three-letter extensions are unaffected, and the only extension used in existing tests (`tsv`) remains valid. ### How was this patch tested? New case in `CSVSuite` covering `abcd`, `ab1`, and `a`. The existing `SPARK-50616` tsv test continues to pass. TODO: full suites: ```bash build/sbt "sql/testOnly *CSVv1Suite" build/sbt "sql/testOnly *CSVv2Suite" build/sbt "sql/testOnly *CSVLegacyTimeParserSuite" ``` should pass once https://github.com/apache/spark/pull/58225 is merged and synced here. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
