andygrove opened a new issue, #2625:
URL: https://github.com/apache/datafusion-comet/issues/2625
### Describe the bug
## Schema
```
c36: ArrayType(FloatType,true)
c51: ArrayType(DoubleType,true)
```
## SQL
```
SELECT c51, c36, array_intersect(c51, c36) AS x FROM test1 ORDER BY c51, c36;
```
### Spark Plan
```
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(2) Sort [c51#173 ASC NULLS FIRST, c36#158 ASC NULLS FIRST], true, 0
+- AQEShuffleRead coalesced
+- ShuffleQueryStage 0
+- Exchange rangepartitioning(c51#173 ASC NULLS FIRST, c36#158 ASC
NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=3778]
+- *(1) Project [c51#173, c36#158, array_intersect(c51#173,
cast(c36#158 as array<double>)) AS x#1135]
+- *(1) ColumnarToRow
+- FileScan parquet [c36#158,c51#173] Batched: true,
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1
paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet],
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<c36:array<float>,c51:array<double>>
+- == Initial Plan ==
Sort [c51#173 ASC NULLS FIRST, c36#158 ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(c51#173 ASC NULLS FIRST, c36#158 ASC NULLS
FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=3764]
+- Project [c51#173, c36#158, array_intersect(c51#173, cast(c36#158 as
array<double>)) AS x#1135]
+- FileScan parquet [c36#158,c51#173] Batched: true, DataFilters:
[], Format: Parquet, Location: InMemoryFileIndex(1
paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet],
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<c36:array<float>,c51:array<double>>
```
### Comet Plan
```
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(2) CometColumnarToRow
+- CometSort [c51#173, c36#158, x#1143], [c51#173 ASC NULLS FIRST,
c36#158 ASC NULLS FIRST]
+- AQEShuffleRead coalesced
+- ShuffleQueryStage 0
+- CometColumnarExchange rangepartitioning(c51#173 ASC NULLS
FIRST, c36#158 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS,
CometColumnarShuffle, [plan_id=3850]
+- *(1) Project [c51#173, c36#158, array_intersect(c51#173,
cast(c36#158 as array<double>)) AS x#1143]
+- *(1) CometColumnarToRow
+- CometScan [native_iceberg_compat] parquet
[c36#158,c51#173] Batched: true, DataFilters: [], Format: CometParquet,
Location: InMemoryFileIndex(1
paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet],
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<c36:array<float>,c51:array<double>>
+- == Initial Plan ==
CometSort [c51#173, c36#158, x#1143], [c51#173 ASC NULLS FIRST, c36#158
ASC NULLS FIRST]
+- CometColumnarExchange rangepartitioning(c51#173 ASC NULLS FIRST,
c36#158 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, CometColumnarShuffle,
[plan_id=3823]
+- Project [c51#173, c36#158, array_intersect(c51#173, cast(c36#158 as
array<double>)) AS x#1143]
+- CometScan [native_iceberg_compat] parquet [c36#158,c51#173]
Batched: true, DataFilters: [], Format: CometParquet, Location:
InMemoryFileIndex(1
paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet],
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<c36:array<float>,c51:array<double>>
```
First difference at row 81:
Spark: `[],[0.0,0.08175886,0.83853054],[]`
Comet: `[],[-0.0,0.90196264],[]`
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
