andygrove opened a new issue, #5325: URL: https://github.com/apache/datafusion-comet/issues/5325
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-08-11 - Total issues processed: 44 (43 triaged, 1 skipped, 0 failed) - Type counts: 13 bugs, 30 enhancements - Priority counts applied: `priority:critical` 2, `priority:high` 3, `priority:medium` 6, `priority:low` 2 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue. Notes on label availability: the guide lists `spark 4` as a pre-existing area indicator, but the repository only has `spark 4.0` / `spark 4.1` / `spark 4.2`, so no `spark 4` label was applied in this pass. Where an issue already carried non-guide labels (`correctness`, `performance`, `test`, `array expressions`, `crash`, `area:Iceberg`), those were left untouched and are included in the area lists below for context. ## Bugs ### priority:critical - A Rust UDF silently answers calls to an ordinary Scala UDF registered under the same name ([#5295](https://github.com/apache/datafusion-comet/issues/5295)) - Area labels: `area:expressions` - Rationale: a Scala UDF registered under a name a Rust UDF already claimed is evaluated by the Rust library and returns wrong values with no error, which is the guide's "silent wrong results" case at the top of the decision tree. - arrays_overlap on flat float arrays does not canonicalize NaN like Spark ([#5270](https://github.com/apache/datafusion-comet/issues/5270)) - Area labels: `area:expressions`, `correctness`, `array expressions` - Rationale: Comet returns `false` where Spark returns `true` for non-canonical NaN bit patterns with no error, a silent correctness divergence; escalated from the reporter's `priority:medium` per the guide's "correctness over crashes" principle. ### priority:high - Native `collect_set(struct)` aggregate crashes on spill emit ([#5239](https://github.com/apache/datafusion-comet/issues/5239)) - Area labels: `area:aggregation` - Rationale: aborts in `GroupedHashAggregateStream::emit` with an ArrowError on the spill path, a crash on a supported code path per the guide's step 2. - [Bug] Preserve CalendarInterval microseconds across Comet boundaries and native kernels ([#5279](https://github.com/apache/datafusion-comet/issues/5279)) - Area labels: `area:expressions` - Rationale: an uncaught `ArithmeticException: long overflow` aborts the query even with ANSI mode disabled, where Spark returns a valid result; visible failure rather than silent, so `priority:high`. - arrays_overlap panics on nested double arrays built from expressions ([#5269](https://github.com/apache/datafusion-comet/issues/5269)) - Area labels: `area:expressions`, `array expressions`, `crash` - Rationale: native panic (index out of bounds in `GenericListArray::value`), matching the guide's crash-bug category; confirms the reporter's `priority:high`. ### priority:medium - Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure ([#5256](https://github.com/apache/datafusion-comet/issues/5256)) - Area labels: `area:scan`, `area:Iceberg`, `correctness` - Rationale: a reflection failure substitutes defaults that leak deleted rows or delete the wrong rows, which would be `priority:critical` on correctness grounds, but the issue states the paths are latent and not reachable on any supported Iceberg version today; kept at the reporter's `priority:medium` and listed under escalations. - Iceberg serde: schema and metadata reflection helpers return empty collections on failure ([#5257](https://github.com/apache/datafusion-comet/issues/5257)) - Area labels: `area:scan`, `area:Iceberg`, `correctness` - Rationale: same family as #5256 — empty mappings can resolve a column to a different field id or silently change which schema a task is serialized with, but only on a reflection failure that is not reachable today. - Iceberg write split-operator plan: CI failures when enabled by default ([#5259](https://github.com/apache/datafusion-comet/issues/5259)) - Area labels: `area:writer`, `area:ci`, `area:Iceberg` - Rationale: 8 CI jobs fail when the split-operator plan is enabled, but the feature ships off by default so nothing is broken for users today; a broken feature with a workaround is `priority:medium`. Re-labeled from `enhancement` to `bug` since the body describes concrete test failures. - Bare -0.0 literals in SQL file tests are coerced through decimal, so signed-zero coverage is vacuous ([#5271](https://github.com/apache/datafusion-comet/issues/5271)) - Area labels: `area:expressions`, `test` - Rationale: test-only defect, which the guide places at `priority:low`, but the fixtures give false confidence in signed-zero coverage across ~35 files and have already let a real divergence through review, so the reporter's `priority:medium` is retained. - Native Parquet writer derives schema nullability and field IDs from Arrow rather than Catalyst ([#5305](https://github.com/apache/datafusion-comet/issues/5305)) - Area labels: `area:writer` - Rationale: written files differ from Spark's in nullability and field IDs, a functional gap in the still-experimental native writer, which the guide's native-writer note places at lower priority. - Native Parquet write of a zero-partition RDD produces no output file ([#5303](https://github.com/apache/datafusion-comet/issues/5303)) - Area labels: `area:writer` - Rationale: missing schema-only output versus Spark, behind an off-by-default config with a fallback available, so `priority:medium` per step 3 of the decision tree. ### priority:low - Reading Iceberg tables show 0.0B for the Input column on Spark's executor UI ([#5264](https://github.com/apache/datafusion-comet/issues/5264)) - Area labels: `area:scan` - Rationale: cosmetic metrics-reporting gap in the Spark UI; query results are unaffected. - Discuss `spark.comet.enabled` parameter behavior ([#5274](https://github.com/apache/datafusion-comet/issues/5274)) - Area labels: none - Rationale: a misleading "native library initialized" log line is emitted when Comet is disabled — cosmetic, no functional impact. Re-labeled from `enhancement` to `bug` since the body reports incorrect observable behavior rather than requesting new functionality. ## Enhancements - Sort-aware Iceberg reads in Comet via a per-partition streaming merge ([#5323](https://github.com/apache/datafusion-comet/issues/5323)) - Area labels: `area:scan` - Rationale: new capability — report scan ordering and merge sorted files natively so Spark can eliminate redundant sorts. - Support Iceberg `SparkCopyOnWriteScan` in `IcebergReflection.ICEBERG_SCAN_CLASSES` ([#5319](https://github.com/apache/datafusion-comet/issues/5319)) - Area labels: `area:scan` - Rationale: extends native scan coverage to a scan class Comet does not claim to support yet; falling back to Spark is the expected behavior today. - refactor: centralize Spark-to-Arrow batch encoding in ArrowWriter ([#5317](https://github.com/apache/datafusion-comet/issues/5317)) - Area labels: `area:ffi` - Rationale: internal refactor of the Spark-to-Arrow conversion paths; the guide lists refactoring under `enhancement`. - Support S3-compliant object stores ([#5313](https://github.com/apache/datafusion-comet/issues/5313)) - Area labels: `area:scan` - Rationale: new functionality for non-AWS S3 endpoints; the current gate correctly falls back to Spark rather than misbehaving. - Evaluate extending collect_set input normalization to Spark 3.4 through 4.1 ([#5312](https://github.com/apache/datafusion-comet/issues/5312)) - Area labels: `area:aggregation` - Rationale: evaluation task about widening a documented, version-gated compatibility restriction, not a defect against the current contract. - WideDecimalBinaryExpr: write nulls directly for overflow ([#5309](https://github.com/apache/datafusion-comet/issues/5309)) - Area labels: `area:expressions` - Rationale: performance optimization of the overflow path; behavior is explicitly to be preserved. - feat: Write Iceberg Parquet data files natively via iceberg-rust (part 3 of 3) ([#5308](https://github.com/apache/datafusion-comet/issues/5308)) - Area labels: `area:writer` - Rationale: new native write capability, part 3 of the #4322 deliverable. - Native Parquet writes report row counts but not row contents to WriteTaskStatsTrackers ([#5307](https://github.com/apache/datafusion-comet/issues/5307)) - Area labels: `area:writer` - Rationale: only third-party trackers that inspect row contents are affected; Spark's only shipped tracker ignores the row argument, and the gap is warned about today, so this is hardening rather than a defect. - Reconcile the two config namespaces for native writes ([#5306](https://github.com/apache/datafusion-comet/issues/5306)) - Area labels: `area:writer` - Rationale: naming cleanup across two testing-category configs; nothing user-facing is broken. - Native Parquet writer ignores Spark's Parquet writer properties ([#5304](https://github.com/apache/datafusion-comet/issues/5304)) - Area labels: `area:writer` - Rationale: feature gap in the experimental native writer — properties are not yet plumbed through, rather than plumbed through incorrectly. - Add a C++ UDF example and a published C header for the Comet UDF ABI ([#5301](https://github.com/apache/datafusion-comet/issues/5301)) - Area labels: none - Rationale: new artifacts (header, example, CI smoke test) plus documentation. - perf: explore bulk copies for no-null fixed-width Arrow writes ([#5299](https://github.com/apache/datafusion-comet/issues/5299)) - Area labels: `area:ffi` - Rationale: performance investigation, explicitly conditional on benchmarks. - Rust UDF library cache holds its write lock across dlopen ([#5297](https://github.com/apache/datafusion-comet/issues/5297)) - Area labels: `performance` - Rationale: contention and lock-poisoning hardening in a path the reporter notes has bounded impact in practice; no incorrect results or crashes in normal operation. - Rust UDF adapter rebuilds the kernel impl and re-resolves the return type on every batch ([#5296](https://github.com/apache/datafusion-comet/issues/5296)) - Area labels: `performance` - Rationale: redundant per-batch work; an optimization with an ABI-contract decision attached. - Scope the Rust UDF registry to the session that registered the UDF ([#5294](https://github.com/apache/datafusion-comet/issues/5294)) - Area labels: none - Rationale: design change to remove a process-wide singleton per the contributor guide; the wrong-results half of the same problem is tracked separately as the bug #5295. - Determine whether "Apache Comet" is a suitable project name ([#5291](https://github.com/apache/datafusion-comet/issues/5291)) - Area labels: none - Rationale: project-administration task, not a code defect. - Comet 1.0.1 Release ([#5283](https://github.com/apache/datafusion-comet/issues/5283)) - Area labels: none - Rationale: release-tracking issue. - Remove orphaned files in comet iceberg split writers ([#5277](https://github.com/apache/datafusion-comet/issues/5277)) - Area labels: `area:writer` - Rationale: cleanup-on-abort is functionality the split writer does not implement yet, behind an off-by-default config. - Build spark_size LargeList lengths as Int32 without Int64 cast ([#5272](https://github.com/apache/datafusion-comet/issues/5272)) - Area labels: `area:expressions` - Rationale: allocation-removing optimization on the `spark_size` LargeList path; results are already correct. - Use the offset buffer for Map in spark_size ([#5266](https://github.com/apache/datafusion-comet/issues/5266)) - Area labels: `area:expressions` - Rationale: vectorization of the Map branch, explicitly droppable if the benchmark comes back flat. - IcebergReflection: Option helpers cannot distinguish an absent accessor from a failed reflective call ([#5258](https://github.com/apache/datafusion-comet/issues/5258)) - Area labels: `area:scan`, `area:Iceberg` - Rationale: API design change to the reflection layer so callers can tell the two outcomes apart; the concrete wrong-behavior consequences are tracked as the bugs #5256 and #5257. - Add a forward-flexible property slot to the Comet UDF C ABI ([#5254](https://github.com/apache/datafusion-comet/issues/5254)) - Area labels: none - Rationale: ABI extension for forward compatibility. - Relax the arrow version requirement in comet-udf-sdk ([#5253](https://github.com/apache/datafusion-comet/issues/5253)) - Area labels: none - Rationale: dependency-range relaxation to unblock future SDK consumers; nothing broken today. - Remove the per-UDF Mutex serializing Rust UDF batches ([#5252](https://github.com/apache/datafusion-comet/issues/5252)) - Area labels: `performance` - Rationale: removes defensive locking that serializes concurrent batches; an optimization, not a defect. - Implement return_field() for imported Rust UDFs ([#5251](https://github.com/apache/datafusion-comet/issues/5251)) - Area labels: none - Rationale: new capability to carry a kernel's full `Field` (non-nullability, metadata) through to the plan. - Make kernel ownership transfer explicit in the Comet UDF C ABI ([#5250](https://github.com/apache/datafusion-comet/issues/5250)) - Area labels: none - Rationale: the issue states the current behavior is correct; this makes the contract harder to get wrong. - Honor the deterministic flag for Rust UDFs ([#5249](https://github.com/apache/datafusion-comet/issues/5249)) - Area labels: none - Rationale: new support for `deterministic = false`, which #4459 currently rejects at registration so the flag cannot silently lie. - Avoid re-running DecimalPrecision.promote during recursive expression serialization ([#5248](https://github.com/apache/datafusion-comet/issues/5248)) - Area labels: `area:expressions` - Rationale: removes redundant traversals; the correctness consequence was already fixed by #5225, leaving a planner performance audit. - Add Criterion bench for native RLike (is_match loop) ([#5246](https://github.com/apache/datafusion-comet/issues/5246)) - Area labels: `area:expressions` - Rationale: adds benchmark coverage for a hot path. - Add AQE test coverage for CometInMemoryTableScanExec ([#5245](https://github.com/apache/datafusion-comet/issues/5245)) - Area labels: none - Rationale: adds test coverage; no defect is reported, only an untested interaction. ## Escalations to consider - Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure ([#5256](https://github.com/apache/datafusion-comet/issues/5256)) - Trigger: the guide places silent wrong results at `priority:critical`, and the described failure modes leak deleted rows or delete the wrong rows. Held at `priority:medium` only because the issue's own reachability analysis says the paths cannot fire on any supported Iceberg version; if that analysis is wrong, this is a `priority:critical`. - Iceberg serde: schema and metadata reflection helpers return empty collections on failure ([#5257](https://github.com/apache/datafusion-comet/issues/5257)) - Trigger: same as #5256 — silent wrong results on a currently unreachable path. Escalate together with #5256 if reachability changes. - Native `collect_set(struct)` aggregate crashes on spill emit ([#5239](https://github.com/apache/datafusion-comet/issues/5239)) - Trigger: the guide escalates a `priority:high` crash to `priority:critical` if it also produces wrong results silently. The report has no reproduction steps, so it is unknown whether a non-spilling variant of the same schema mismatch yields wrong output rather than aborting. ## Skipped — needs more info - Bug triage results: 2026-08-03 ([#5231](https://github.com/apache/datafusion-comet/issues/5231)) - This is the previous triage summary issue, awaiting human review. It is neither a bug nor an enhancement, so no type label was applied and `requires-triage` was left in place; it will drop out of the queue when a reviewer closes it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
