andygrove commented on issue #3079:
URL:
https://github.com/apache/datafusion-comet/issues/3079#issuecomment-5783001281
Reopening. I closed this saying we don't need to allocate the same
partitions as Spark, and that was wrong.
We have two shuffle implementations and both can appear in the same plan.
`applyCometShuffle` picks native or columnar per exchange with no harmonization
across the plan, so the two sides of a join can end up with different ones. The
columnar path uses Spark's `HashPartitioning.partitionIdExpression` and the
native path uses the Rust murmur3. On a decimal key with precision above 18
those disagree. So the native path does have to match Spark, not for its own
sake, but to stay consistent with its sibling.
On main at `9a4d5f283`, joining a Parquet table to a JSON view on a
`DECIMAL(38,0)` key:
```
CometSortMergeJoin [k#37], [k#40], Inner
:- CometSort [k#37, a#38], [k#37 ASC NULLS FIRST]
: +- CometExchange hashpartitioning(k#37, 7), ENSURE_REQUIREMENTS,
CometNativeShuffle
: +- CometNativeScan parquet spark_catalog.default.wd_parquet[k#37,a#38]
+- CometSort [k#40, b#41], [k#40 ASC NULLS FIRST]
+- CometColumnarExchange hashpartitioning(k#40, 7), ENSURE_REQUIREMENTS,
CometColumnarShuffle
+- FileScan json [k#40,b#41]
```
Comet returns zero rows where Spark returns four. Default
`spark.comet.shuffle.mode=auto`, nothing exotic set. It reproduces with AQE
enabled too, as long as coalescing doesn't collapse the fixture into a single
partition, which is what you'd get at any real data volume. That's most likely
why the suites have never caught it.
The mismatch itself is in `hash_array_decimal!`, which hashes
`value.to_le_bytes()`, a fixed sixteen-byte little-endian i128. Spark hashes
`toJavaBigDecimal.unscaledValue.toByteArray()`, which is minimal-length
big-endian two's complement. Byte order and length both differ, and murmur3
mixes length into `fmix`, so every value diverges. Unscaled `1` in
`DECIMAL(38,0)` at seed 42 hashes to `-386724586` in Spark and `-680163996` in
Comet, which lands in partition 4 against partition 6 out of 7.
#6005 implements the guard described in the original report. #5994 tracks
making the native encoding match Spark, which would remove the need for the
guard and also unblock `hash` and `xxhash64` for these types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
