comphead commented on PR #5974: URL: https://github.com/apache/datafusion-comet/pull/5974#issuecomment-5717027902
Rebased onto `main` (now at `58ab5f618e`) and squashed the nine iteration commits into two: one `ci:` for the tooling, one `docs:`. The content is unchanged from what you reviewed apart from the items below, which came out of using it on a real run. Worth flagging since #5973 landed in the meantime: I independently hit the same cargo-cache eviction while measuring where CI time goes, and the numbers matched yours. Good to see it fixed on `main`. Changes since your review: - **`dev/ci/local-ci-config.py`** replaces the shell's awk/sed parsing *and* the second Python implementation the preflight check used for comparison. You were right that two parsers was the wrong shape, and the requoting case proved it: both stripped only single quotes, so `spark-full: "4.1.3"` came out with quotes attached and the comparison happily agreed. There is one parser now, it validates every value against a version shape, and `check_local_ci_config` imports it rather than re-deriving. That took the check from 208 lines to 48, and it now also catches a queue-tier job rename leaving the default dangling, which the old comparison could not. - **Row-level concurrency.** Selecting more than one Spark row runs them all at once, each in its own copy of the prepared tree, which is what the workflow does with one runner per matrix row. So the per-row settings stay byte-identical to CI's instead of diverging to buy parallelism. A single row runs in the prepared tree with no copy. - **Three failure modes I hit and guarded**: partial POMs in the local repository that Coursier reports as a missing JAR it can see a POM for; purging those invalidating sbt's cached resolution, which `sbt-antlr4` reads instead of the filesystem, producing a bare `Could not find or load main class org.antlr.v4.Tool`; and editing a diff wedging the tree until it was deleted, now handled by recording the applied diff and reverting it first. On testing, so the state is clear. A real `dev/local-ci.sh spark` run on Spark 4.1 executed all seven rows concurrently and `sql_core-1` completed 12914 tests in 50m 36s with 812 ignored by `IgnoreComet`. Its four failures and one in `sql_hive-3` were all `No space left on device` / `Mkdirs failed to create` — my disk, not Comet. I have not yet seen a full green run, and the docs say so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
