dwsmith1983 commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5678443632
> Please activate `delta` for the prerequisite installs and the README's matching command, or use a build sequence that produces the current test JAR. Done: both prerequisite installs in `delta_contrib_test.yml` and the README's first command now run with `-Pspark-X,delta`, so the install produces this checkout's test-jar before the contrib suites resolve it. > Please use a setup path that elevates package installation on the plain runner while retaining Docker access. Done: the MinIO job no longer uses `setup-builder`; it installs `protobuf-compiler` and `clang` with `sudo`, sets up JDK 17 through `setup-java`, and installs the Rust toolchain with `rustup`, mirroring the action's steps. The workflow parses and `check-ci-config.py` passes; the first CI run is the proof, since the workflows here still wait on approval. > Please include a focused before/after microbenchmark for this new fast path using modern, null-heavy and mixed-age batches. Measured on a release build, 1M-row microsecond UTC timestamp batches, median of 20 runs, before against after: | batch | policy | before | after | | --- | --- | --- | --- | | modern | CheckAncient | 461 us | 199 us | | modern | Legacy(Utc) | 698 us | 199 us | | null-heavy (90% null) | CheckAncient | 215 us | 251 us | | null-heavy (90% null) | Legacy(Utc) | 194 us | 245 us | | mixed-age (half ancient) | Legacy(Utc) | 851 us | 1098 us | The first cut used a validity-aware scalar loop and lost on null-heavy batches, so the landed version takes one vectorised minimum only on null-free batches and returns the input untouched when nothing predates the cutover; batches with nulls or ancient values keep the previous per-value arms, so the cost there is the extra minimum pass. Modern null-free batches, the common case for a metadata-free file under EXCEPTION mode, are 2.3x to 3.5x faster; the mixed-age case pays about 30% more. Pruning is unchanged by this, as you noted. > Agreed follow-up issues should be linked before treating those requests as resolved. Filed, one per request: #5943 (stable injection key), #5944 (shared admission checks), #5945 (shared builder for the DV path), #5946 (explicit planning boundary), #5947 (envelope and provider contract tests), #5948 (preparation I/O through the scan's instrumented reader), #5949 (delegate schema preparation to `prepareSchemaForRead`). Each records your criteria and the tests you asked for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
