Joorgem commented on PR #57671: URL: https://github.com/apache/spark/pull/57671#issuecomment-5152542839
Added in 3a9ef31 — thanks. FAILFAST now asserts `FAILED_READ_FILE.NO_HINT` via `checkErrorMatchPVals`, matching the suite's existing `test for FAILFAST parsing mode`. While doing it I had to fix something of my own. @szehon-ho pointed out on #57608 that "both halves are malformed" is too strong — each physical line is parsed alone, univocity's default `STOP_AT_DELIMITER` does not fail on an unclosed quote, and token count is what decides corruptness. That condemned this test's *comment*, which blamed the unbalanced quote. The assertions were right; the explanation was not, and nothing catches that. So the test now pins both shapes, because they differ in the way that matters: | schema | input | leading half | trailing half | `DROPMALFORMED` | |---|---|---|---|---| | 3 col | `1,"a⏎b",c` | malformed | malformed | drops both — whole record lost | | 2 col | `1,"hello⏎world"` | `[1, hello]` — **clean, unflagged** | malformed | **keeps the truncated row** | The second one is the reason the case is worth a test rather than only a doc note: `DROPMALFORMED` retains a row that looks valid and is not, precisely because its token count still matches. The first case is the benign one — nothing silently wrong survives it. Verified green on the full `sql` matrix (`slow`, `extended`, `other`) against `master` before pushing, so it holds in `CSVv1Suite`, `CSVv2Suite` and `CSVLegacyTimeParserSuite`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
