andygrove commented on issue #3079:
URL:
https://github.com/apache/datafusion-comet/issues/3079#issuecomment-5818110339
A cached relation makes this reachable from a query that has Comet turned
off. The cache keeps the partitions native shuffle produced, and the cache scan
reports the cached plan's `hashpartitioning(k, 200)` to every later query. A
later join on `k` then shuffles only the other side, so if that side goes
through Spark's hash, the join silently drops rows. That happens with Comet off
for the query, or with Comet on when the other side's shuffle is columnar.
On `main` at 67803a7a4, default Spark 4.1 profile, with
`spark.sql.autoBroadcastJoinThreshold=-1`:
```scala
spark.range(0, 10000, 1, 4)
.selectExpr("cast(id as decimal(38,0)) as k", "id as v")
.write.parquet(pq)
spark.range(0, 10000, 1, 3)
.selectExpr("cast(id as decimal(38,0)) as k2", "id as w")
.write.parquet(pq2)
val cached = spark.read.parquet(pq).repartition($"k").cache()
cached.count() // the cached plan shuffles with CometNativeShuffle into 200
partitions
spark.conf.set("spark.comet.enabled", "false")
cached.join(spark.read.parquet(pq2), $"k" === $"k2").count() // 57, expected
10000
```
| Other side of the join | Rows |
| ---------------------------------------- | ------------- |
| Comet off, Parquet (Spark shuffle) | 57 / 10000 |
| Comet on, JSON (`CometColumnarShuffle`) | 57 / 10000 |
| Comet on, Parquet (`CometNativeShuffle`) | 10000 / 10000 |
It is the same with AQE on or off, and with
`spark.comet.exec.inMemoryCache.enabled` on or off, since the partitioning
comes from the cached plan rather than the cache format. `decimal(20,2)` keys
fail too. Decimal keys of precision 18 or less, and integer, string, binary,
date, timestamp, timestamp_ntz, boolean, float and double keys, all join
correctly after a native-shuffled cache.
The first row is the new part: without a cache, a query with Comet off never
sees a native shuffle. #6005 should fix this too, since it keeps wide-decimal
keys off native shuffle, and a cached case might be worth adding to its tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
