comphead commented on PR #5974: URL: https://github.com/apache/datafusion-comet/pull/5974#issuecomment-5723155415
Thanks — all three were real and are fixed in `ed904dc`. I reproduced each before changing anything, and the first one was the serious one. **Failed parser exited 0.** Confirmed exactly as you describe on `/bin/bash` 3.2. `eval "$(...)"` reports eval's own status, so an empty result from a failed parser looked like success, `DEFAULTED` was then unbound, and the run printed a green total and exited 0 having tested nothing. That is a false green from a tool whose whole purpose is deciding whether something is safe to queue — the same class as the `SHARD_COUNT` finding in the previous round, and I should have caught the pattern the second time. The assignment is now captured and checked before evaluating: | invocation | before | after | | --- | --- | --- | | `spark nonexistent` | exit 0, `DEFAULTED: unbound variable` | exit 1, no unbound error | | `spark 9.9` | exit 0 | exit 1 | | `iceberg 9.9` | exit 0 | exit 1 | **Duplicate rows.** `spark_rows` deduplicates by name with order preserved, so `core sql_core-1` yields `catalyst, sql_core-1, sql_core-2, sql_core-3` rather than naming `sql_core-1` twice. `sql_hive-2 sql_hive-2` collapses to one, and the disjoint `core hive` control still gives seven. Worth recording why it was destructive rather than merely wasteful: a row's tree and log are keyed on its name, so the second copy's `clone_tree` deleted the directory the first was running in. **Orphaned rows on a copy failure.** I took the "prepare all trees before launching any row" option rather than tracking children in an exit path — there is no bookkeeping to get wrong, and the likely trigger, a full disk, then surfaces before anything expensive starts. With the second copy injected to fail: exit 1, zero rows launched, zero strays, against a running orphan before. The happy path is still concurrent, three rows of 2s finishing in 2s wall across three distinct trees. On your note that the heavy jobs were skipped so this is not validated end to end: agreed, and it still is not. The closest I have is the real Spark 4.1 run in my previous comment, where `sql_core-1` completed 12914 tests in 50m 36s and every failure traced to `No space left on device` on my machine. The concurrency fixes above are verified with bounded doubles establishing shell behaviour, not Spark results, same caveat as yours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
