andygrove opened a new issue, #6201:
URL: https://github.com/apache/datafusion-comet/issues/6201
### What is the problem the feature request solves?
`branch-1.0` was cut at `af534e0f` on 2026-08-03, and 1.0.1 has not been
tagged yet. Bug fixes keep landing on `main`, but nothing tracks which of them
should go to `branch-1.0`, which are already there, and which open PRs will
become candidates once they merge.
#5815 triaged the first 192 commits after the cut, up to 2026-09-09, and it
remains the record of why each earlier exclusion was made. This epic carries
forward #5815's bug-fix recommendations from tiers 1 and 3. It adds the 166
commits merged since then and every open bug-fix PR on `main`. Performance
backports are out of scope; #5815's tier 2 covers those.
### Describe the potential solution
Tick a box once the fix is on `branch-1.0`, and put the backport PR next to
it. When an open PR merges, move it up to the merged list.
Candidates are ordered by severity within each group. "Clean" means the
commit cherry-picks onto `branch-1.0` (`8407d475c`) without conflicts; it does
not mean it compiles. Where one fix has to follow another, the entry says so.
#### Merged on `main`, not yet on `branch-1.0`
**Wrong results**
- [ ] #6041 A decimal `SUM` returns NULL, or raises `ARITHMETIC_OVERFLOW`
under ANSI, when an intermediate sum overflows the buffer precision but the
final result fits. Closes #6002. Conflicts in `sum_decimal.rs` and
`operators.scala` (fewer after #5746). Take #6156 with it: it limits the new
fallback to Spark 3.5+.
- [ ] #5421 After a late shuffle fallback, a native partial aggregate can
feed a Spark final aggregate: `avg` returns NULL, and the `collect_list` /
`collect_set` buffers mismatch. Closes #5419. Conflicts in
`CometExecRule.scala` and `RevertNativeForTransitionHeavyStages.scala`; drop
the Celeborn parts.
- [ ] #5470 Exchange reuse shares one shuffle between aggregates that differ
only in their result expressions, such as `COUNT(*) + 1` and `COUNT(*) - 1`.
Conflicts in `operators.scala` imports and `CometAggregateSuite`.
- [ ] #5828 Exchange reuse treats semi and anti joins, null-aware `NOT IN`,
and `explode` vs `explode_outer` as the same plan. Closes #5824. After #5470
only the tests conflict.
- [ ] #5053 Two ABFS containers in one storage account share one cached
object store, so a read can return the other container's data. Closes #4993.
Needs a manual port: it builds on URL normalization and `ObjectStoreBackend`,
which exist only on `main`. Take #5503 first.
- [ ] #5503 With asymmetric `fs.comet.libhdfs.schemes` (for example `s3`
listed, `s3a` not), Hadoop and native object stores share a cache key, and the
registry can overwrite an entry. Closes #5502. Conflicts in the
`parquet_support.rs` test module only.
- [ ] #5235 `arrays_overlap` and `array_position` over nested float arrays
treat `-0.0` and `0.0` as different. Closes #5191. Context conflicts only.
- [ ] #6073 `IN` / `InSet` and nested `=` over arrays and structs treat
`-0.0` / `0.0`, and different NaN encodings, as unequal. Closes #6019. After
#5235; context conflicts only.
- [ ] #5472 A NaN literal with non-canonical bits, and static float `IN` /
`InSet` hashing raw bits, drop rows that Spark keeps. After #6073. On
`branch-1.0` keep `withFallbackReason`, because `liftFallbackReasons` exists
only on `main`.
- [ ] #6004 With `spark.sql.parquet.fieldId.read.enabled`, a struct that
repeats a field id returns rows where Spark raises. Closes #5801. After #5866;
drop the in-memory cache hunk.
- [ ] #5684 Decimal to double / float casts are off by one ulp for
essentially every `DECIMAL(38,18)` value above 0.01. Clean.
- [ ] #5752 Rejected `TIMESTAMP_NTZ` casts return NULL under ANSI, where
Spark raises `CAST_INVALID_INPUT`. Closes #5749. Clean.
- [ ] #5682 String to timestamp casts give `2020-10-1` a spurious offset,
accept date-only zones, and panic on Unicode fractional digits. Closes #5674.
Clean after #5752.
- [ ] #5858 String to timestamp casts return NULL for years written as
`+7528`, and malformed signs bypass ANSI errors. Closes #5716. After #5682; the
inline tests in `string.rs` conflict.
- [ ] #5623 The codegen dispatcher's null short-circuit swallows ANSI errors
raised by a foldable subtree. Closes #5608. Clean.
- [ ] #5177 and #5740 An overflowing `TIMESTAMP_MILLIS` value silently wraps
in Parquet reads, at the top level and nested, where Spark throws. Conflicts in
`parquet_support.rs`.
- [ ] #5738, the `to_time` hunk only: `to_time` / `try_to_time` reject
`'T12'` and `'12:30:45.'`, which Spark accepts. Closes #5366, which is in the
1.0.1 milestone. `to_time.rs` and `to_time.sql` apply by themselves. The rest
of the PR adds native `EXTRACT(SECOND FROM TIME)` and TIME literals.
- [ ] #5789, the `unix_timestamp.rs` hunk only: native `unix_timestamp`
floors pre-epoch fractional timestamps, returning `-2` where Spark returns
`-1`. Closes #5896. The hunk applies by itself. The rest of the PR is a new
codegen-dispatch route.
- [ ] #5166 On Spark 4.2 only, `collect_set` does not deduplicate NaN and
`-0.0` the way SPARK-57298 requires. Closes #4966. Conflicts in the
`aggregates.scala` imports.
**Parquet field matching: backport these together or not at all**
#5602 cherry-picks clean on its own, but it must not go alone. On 1.0.0, a
struct with byte-identical duplicate field names panics. With #5602 applied,
the same read silently returns duplicated rows (#5783), and #5786 is the fix.
On top of #5602 alone, #5786 conflicts in 12 hunks of `schema_adapter.rs`,
because it builds on #5681, which in turn builds on #5177 and #5740. The two
options are the full stack, in the order below, or a hand port of #5786's
duplicate check onto #5602.
- [ ] #5602 Non-ASCII field names that differ only in case silently read as
NULL. Closes #5495, which is in the 1.0.1 milestone.
- [ ] #5681 Nested struct, list and map Parquet conversions return NULL on
overflow, accept values Spark rejects, or panic. Closes #5671. About 1,600
lines.
- [ ] #5845 When the JVM call that case-folds a field name fails, Comet
falls back to Rust's Unicode tables, which can select a different column than
Spark would.
- [ ] #5786 Rejects duplicate Parquet field names before decoding. Closes
#5783.
**Task failures, panics, and errors Spark does not raise**
- [ ] #5159 `collect_list` / `collect_set` over a nested argument fails with
"column types must match schema types" when nested-field nullability drifts.
Closes #5158. Clean.
- [ ] #5866 Native shuffle accepts a struct with duplicate field names, then
fails the task importing the batch into Java Arrow. With the fix, it falls
back. Closes #5605. Only the test conflicts.
- [ ] #6128 Comet memory pools panic in `grow` when Spark grants less than
requested, for example on the sort-merge join spill path. Closes #6127 and
#1733. Clean after #5494 apart from docs.
- [ ] #5133 A null calendar-interval literal passes the JVM check and then
fails native planning. Closes #5058. Clean.
- [ ] #5680 `rpad` / `lpad` panic on a NULL length. Clean.
- [ ] #5630 A sliced map fails `map_sort` with "Max offset exceeds length of
entries". Only the test conflicts.
- [ ] #5679 `concat_ws` with array arguments fails natively instead of
falling back. Clean.
- [ ] #5683 Float / double to integral casts raise a spurious
`CAST_OVERFLOW` for exactly `Int.MaxValue` / `Long.MaxValue`. Clean.
- [ ] #5357 ANSI `abs` overflow errors say `Int64 overflow` where Spark says
`long overflow`. Clean.
- [ ] #5747 `copy_array` panics on offset overflow instead of returning an
error. Closes #5706. Conflicts in `copy.rs`.
- [ ] #5561 On Spark 4.2, accelerated `mapInArrow` / `mapInPandas` fails
outright because of the Python worker configuration. Clean.
- [ ] #5759 Reading an Iceberg table partitioned by an unknown transform
fails with `Invalid Iceberg scan task`. Conflicts in
`CometIcebergNativeScan.scala`.
**Leaks, hangs and memory**
- [ ] #5748 The JVM hangs if an application returns from `main` without
calling `spark.stop()`, because Tokio runtime threads are not daemon threads.
Only the CI YAML conflicts.
- [ ] #5494 A stranded task-shared memory pool entry pins
`TaskMemoryManager` and `TaskContext` for the executor's lifetime. Conflicts in
the CI YAML and `jni_api.rs`.
- [ ] #5493 The static `CometDiskBlockWriter` spill registry lets one task
force-spill another task's buffered bytes. Clean.
- [ ] #5539 A failed Arrow vector import leaks native buffers. Closes #5534.
Only the test conflicts.
- [ ] #5461 After a failed partition write, native shuffle leaks its
reservation, its pinned buffers, and repeated-spill bytes. Clean, but its test
needs metric fields from #5370 and has to be adapted.
- [ ] #5907 The JVM shuffle sizes pointer-array growth from the data pages
instead of the array, so it over-allocates and spills early. Part of #5905.
Clean.
#### Open on `main`: candidates once merged
Each of these bugs was confirmed present in `branch-1.0`'s code. The PR
state and the cherry-pick result are as of 2026-09-24.
**Wrong results**
- [ ] #6154 Native Iceberg scans push a `bucket`, `truncate` or `days`
residual to iceberg-rust as a predicate on the source column. Combined with
`AND`, `OR` or `NOT`, this drops rows. Closes #5992. The same code is on
`branch-1.0`, but the repro has only been run on `main`. Conflicts in
`CometIcebergNativeScan.scala` and the CI YAML.
- [ ] #6069 Under ANSI or `TRY`, an integer `SUM` over a sliding window
frame wraps on overflow, because sliding frames use DataFusion's wrapping
`sum`. Closes #6043. Approved. Clean.
- [ ] #6076 `var_pop`, `var_samp` and `stddev` lose precision on large,
closely spaced doubles. Closes #6044. Approved. Drop the `regr.rs` hunk, which
is `main`-only. The PR fixes the update path only; with more than one input
partition the merge path still diverges.
- [ ] #5403 `array_min` / `array_max` pick the wrong signed zero, or the
wrong one of several equal nested elements. Closes #5401. Approved. Needs
adapting, because it uses the `main`-only `hasConditionalNativeDefault`.
- [ ] #5867 The `IS NOT NULL` guard serializes a nondeterministic child
twice in `size`, `array_append`, `arrays_zip` and `map_from_arrays`. Closes
#5781. Approved. Take only the null-guard half; the map-lookup dispatch half
closes the enhancement #5580.
- [ ] #5846 When `map_from_arrays` gets key and value arrays whose lengths
differ per row but match in total, it shifts values across rows. Approved.
Conflicts in `map_funcs/mod.rs` and `maps.scala`.
- [ ] #6178 The codegen dispatcher writes a null map key as the type's
default value, so a failing key cast under `TRY_CAST` returns `[1, 0]` instead
of `[1, NULL]`. Closes #6172. Only the test conflicts.
- [ ] #5457 A wide `DATE` cast to `TIMESTAMP_NTZ` silently wraps in release
builds, and a cast to `TIMESTAMP` panics. Closes #5456. Waiting on the author
since 09-17; 43 files.
- [ ] #5420 An empty native `AVG` partial exports `(NULL, 0)`, which erases
the result when Spark merges it, and decimal overflow does not stick. Closes
#5418. After #5421.
- [ ] #5515 When Iceberg delete-file reflection fails, the serde defaults to
position deletes with spec 0 and no equality ids, so deletes are misapplied.
Closes #5256. Approved. Only reachable when reflection fails.
- [ ] #5050 With the legacy `allowNegativeScaleOfDecimal` flag, rescaling a
negative-scale decimal wraps silently, and decimal to double loses precision.
Closes #5013. Only reachable with that flag.
**Errors where Spark succeeds, or success where Spark raises**
- [ ] #6116 Comet checks for files without field ids only at the root, and
only with `fieldId.read.enabled`. So some reads return rows where Spark raises,
and files with ids only on nested fields are rejected where Spark reads them.
Closes #5936. Approved. Overlaps the Parquet field-matching stack above;
conflicts in `parquet_support.rs` and `schema_adapter.rs`.
- [ ] #6179 (draft) A `TRY_CAST` between maps whose key cast can fail errors
natively with "Found unmasked nulls". Closes #5995. Stacked on #6178.
- [ ] #5903 `collect_list` / `collect_set` over structs with required fields
fail when the final aggregate spills. Closes #5239. Approved. Take it with
#5159.
- [ ] #5923 A scalar subquery inside a codegen-dispatched `ScalaUDF` fails
with "Subquery … has not finished". Approved. Clean.
- [ ] #5135 `hash` / `xxhash64` over `CalendarInterval` fail natively.
Closes #5059. Approved. Clean.
- [ ] #6035 A throwing literal cast in a branch that never runs fails the
query, because Comet folds it at planning time. Changes requested.
- [ ] #5872 The S3 profile name and file settings are ignored, and
`path.style.access` is applied inverted for custom endpoints. Closes #2802 and
#4245. Approved. Adds an `aws-runtime` dependency.
- [ ] #6059 Ambient `AZURE_*` environment variables outrank the configured
Hadoop Azure auth, giving a 403 or the wrong identity. Closes #5542. Clean, but
about 2,400 lines in `azure.rs`.
- [ ] #5169 Decimal ANSI errors surface as plain Arrow errors, without
Spark's error class or SQLSTATE. Closes #5072. Approved.
- [ ] #5276 (draft) The wide-decimal overflow message reports a different
value than Spark. Closes #5211. Stacked on #5169. Message text only.
- [ ] #4653 A wrapped ANSI divide-by-zero surfaces as `CometNativeException`
instead of `SparkArithmeticException`. Closes #4517. Stalled; clean on
`branch-1.0`.
- [ ] #5891 A Parquet `TIMESTAMP_MILLIS` overflow surfaces as a raw Arrow
error instead of Spark's file-read error. Closes #5517. Only meaningful after
#5177 and #5740.
**Resource use, hangs and plan quality**
- [ ] #6191 Three settings are read in the wrong unit. The native shuffle
write buffer is 1 byte by default. A `maxTempDirectorySize` with a unit
silently becomes 100 GiB. A bare `spark.memory.offHeap.size` is read as MiB.
Closes #6183, #6184 and #6185. Approved.
- [ ] #6092 The native scan loop busy-polls, pinning a core per task while
it waits on I/O with a JVM-fed input in the plan. Closes #6091. Under review;
the loop as written can lose wake-ups.
- [ ] #5483 `CometExecRule` overwrites AQE logical-stage links, so AQE loses
broadcast-join conversions. Results are correct. Closes #5482. Approved.
#### Needs a decision
**Merged on `main`**
- #5310 decodes invalid UTF-8 at the JVM to native FFI import (part of
#4764). It is a soundness fix, but it is about 600 lines, changes the output
for invalid bytes, and adds a validation pass per batch.
- #5743: `slice` over an array with non-nullable elements, such as
`slice(split(...))`, fails with a type mismatch. The fix is the `CometSlice`
hunk of #5766 plus the tests in #5839. Not yet reproduced on `branch-1.0`.
- #5815's tier 3, unchanged: #5558, #5469, #5726, #5653, #5464 and #5552.
For #5362, DataFusion 55.1 now carries the upstream `UnnestExec` batch-size fix
(apache/datafusion#24529). `branch-1.0`'s DataFusion 54.1 does not, so #5362's
1,491-line operator is still the only route on 1.0.x.
**Open on `main`**
- #6065 (approved): the JVM claims `gcs`, `abfs`, `abfss`, `wasb` and
`wasbs` Iceberg files that the native factory cannot open, so those tasks fail.
Closes #5541. The PR rewrites `main`-only code; on `branch-1.0`, narrowing the
scheme `Set` in `CometScanRule` would be a one-line fix.
- #5854 (approved): a map built with a NULL key succeeds where Spark raises,
and that half is a real bug on `branch-1.0`. The PR relies on DataFusion 55's
`map_key_dedup_policy`, so 1.0 would need a separate null-key check. Closes
#4680 and #5857.
- #5750 (approved): `branch-1.0`'s DataFusion 54.1 keeps `-0.0` and `0.0`
distinct in `array_distinct` / `array_union`. That matches Spark through 4.1.3
and diverges on 4.2.0, 4.0.5+ and 4.1.4+. The PR's gate falls back exactly
where 1.0 is already right; 1.0 would need the opposite gate. Closes #5701.
- #5048: the native scan ignores legacy Julian/Gregorian rebasing, so
pre-1582 dates and timestamps read shifted. This is documented as a limitation
on 1.0. The fix adds a config and reads file footers on the driver at planning
time. Closes #5010.
- #5533: `unbase64` can fail on rows that Spark skips under `LIMIT` or a
semi-join. The fix adds a planner policy and a config. Closes #5532.
- #5654 (changes requested): nested duplicate field ids silently take the
first match. It builds on `main`'s #5786 and #6004, and field-id reads are off
by default.
- #5613 (changes requested) and #5847 (draft): the default `fair_unified`
pool holds its lock across the JNI acquire, which can hang, and forces spills
below the fair share. #5613 builds on #6128.
- #6005 (draft): native hash partitioning of decimals above precision 18
differs from Spark's. It depends on #5421, and #5994 tracks the full fix.
- #5036 (draft): the libhdfs thread destructor can use freed memory and
crash at JVM teardown, and `hdfs-opendal` is a default feature on 1.0. The fix
pins `hdfs-sys` to a personal fork through `[patch.crates-io]`. Closes #5023.
- #4791: an aggregation task fails with `offset overflow` once the group
keys exceed 2 GiB. The fix adds a config and proto changes. Stalled. Closes
#4718.
- #6025: a transient STS throttle on IRSA becomes a storm of 403s. The fix
adds a provider module, dependencies and a config, and its Iceberg half is
`main`-only. Closes #6024.
- #6195 (approved): disables Comet when its extensions are registered
without off-heap memory. That is a behaviour change for a patch release. Closes
#6186.
- #5172 (draft): string to timestamp casts do not trim ISO control
characters. It is stacked on #5130, and `branch-1.0` lacks #5682. Not yet
verified on 1.0. Closes #5165.
- #5265 (approved): the `Input` column reads 0 with the native Iceberg scan
or native shuffle. It is metrics-only, which #5815 excluded, but it is in the
1.0.1 milestone. Closes #5264.
#### Already on `branch-1.0`
- [x] #5138, #5162, #5167 and #5282, via #5321
- [x] #5185, #5364, #5376, #5392 and #5443, via #5480, which also took #5400
to keep clippy green
- [x] #5757, via #5817
- [x] #5754, via #5823
The other backports since the cut are #5192 (#5261), #5315 (#5316), #5737
(#5820) and #5782 (#5833).
#### Not candidates
#5815 lists everything excluded before 2026-09-09. Since then:
- **Fixes code that only exists on `main`.**
- The native Iceberg writer: #5780, #5810, #5663, #5840, #5935, #6153 and
#6151.
- Native Iceberg scan metrics: #6085.
- `regr_*`: #6042.
- Join runtime filters: #6067.
- The libhdfs routing regression from #5314: #5825.
- Open PRs: #6098, #5292, #5932, #5972, #5851 and #4746.
- **Off by default, or latent on `branch-1.0`.**
- #6090, which only matters behind `sparkToColumnar`. It is also the crash
follow-up to #5442, so take it along if #5442 from #5815's tier 2 is backported.
- Experimental features: #5560 (PyArrow UDFs) and #5763 (`WriteFilesExec`).
- Open PRs: #4971, #5957, #6137 and #5982.
- **Closes an enhancement, or only changes when Comet falls back.** #5215,
#5227, #5732 and #5916, and the open #5603, #5607, #5889, #6110, #6023, #5047,
#5302 and #6182.
- **Explain output, tooling, error class or metrics only.** #5471, #5746,
#5991 and #6046, and the open #5880 and #5505.
- **A behaviour change.** #6054.
- **Blocked on an unreleased DataFusion fix.** #5958.
- **Test, CI, docs, bench or dependency only.** #5855 and the rest of the
`test:`, `ci:`, `docs:`, `chore:`, `bench:` and `deps:` commits, and the open
#6108, #6061, #5721, #4570 and #5804.
### Additional context
How the lists were built:
- **Merged.** Every `fix:` commit in `424c31aa7..67803a7a4`, plus any other
commit that closes a `bug` issue. Each one was classified by the labels on the
issue it closes, not by its prefix. Applicability was tested with `git
merge-tree` against `branch-1.0`, and every conflicting path was checked for
existence there. A conflict in a file that `branch-1.0` lacks means the fix is
for code that only exists on `main`.
- **Open.** Every open PR against `main` whose title starts with `fix`, or
that closes a `bug` issue: 67 PRs. For each one, the defective code was
confirmed on `branch-1.0` with `git grep`, and the PR head was test-merged onto
`branch-1.0`.
- **Not build-verified.** Each backport PR needs `cargo check --workspace
--all-targets` and `./mvnw test-compile` on `branch-1.0` before it goes up. A
clean cherry-pick is not a compiling one. #5461 is the known case, and the
usual causes are a test module's shared `use` block or a proto struct literal.
- **Rust version.** `branch-1.0` CI builds with Rust `stable`. If clippy
goes red on code the backport did not touch, include the newest `chore: fix
clippy warnings` commit from `main`.
- **Dependencies.** `branch-1.0` is on DataFusion 54.1.0 and Arrow 58.4.0.
`main` is on DataFusion 55.1.0 and Arrow 59.2.0.
Still open in the 1.0.1 milestone, with no fix on `main`: #4412
(`CometHashAggregateExec` and `AQEPropagateEmptyRelation`) and the enhancement
#5015.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
