andygrove commented on PR #5262:
URL:
https://github.com/apache/datafusion-comet/pull/5262#issuecomment-5553465909
All six `[shuffle]` job failures on the last run (Spark 3.4/3.5/4.0/4.1/4.2
on Linux, plus macOS 4.0) were the same two tests in
`CometCelebornShuffleReaderSuite`, failing identically everywhere:
```
- JVM reader rejects nullable map keys before cast or import *** FAILED ***
- native ShuffleScan rejects nullable map keys before cast or import ***
FAILED ***
failure.exists(... contains "type mismatch")) was false
(CometCelebornShuffleReaderSuite.scala:1393)
```
This is exactly what @pingzh already fixed in #5697: Arrow 59.2 rejects a
nullable map-key field during IPC validation with `Map key field must not be
nullable`, which now fires before Comet's own `validate_remote_schema` check
that produces `type mismatch`. The reader still rejects the malformed frame
correctly and reports the fetch failure once; only the expected message changed.
#5697 was squash-merged into `df55` on 2026-09-04 as
f536ca7f9bcd49b1012f2017cdcf67555b38410d, on top of 3ee7fa65b. The branch was
then rewritten: 3ee7fa65b came back as cb23c2598 with the same commit message
and author date but a new hash, and the squash-merge commit was not carried
over. So the fix was silently dropped from the branch even though the
description still lists it.
I have cherry-picked it back as 2e0a757d976645201819c1ab0b5d9f3d0729603b
(`git cherry-pick -x`, authorship preserved) and pushed to `df55`.
Worth watching on the next run: several jobs were still pending when the
failures were triaged (Iceberg extensions, the Spark SQL `sql_core`/`sql_hive`
shards, macOS `[expressions]`), so this may not have been the complete failure
set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
