pchintar opened a new issue, #4457:
URL: https://github.com/apache/datafusion-comet/issues/4457
### Describe the bug
## Description
Queries containing a `NullType` column can crash when Comet native
`LocalTableScanExec` support is enabled.
Example:
```sql
SELECT max(col) FROM VALUES
(NULL),
(NULL)
AS t(col)
```
with:
```text
spark.comet.exec.localTableScan.enabled=true
```
Observed fallback plan:
```text
HashAggregate
+- Exchange [COMET: unsupported shuffle data type NullType for input max#10]
+- HashAggregate [COMET: Unsupported data type: NullType, Unsupported
aggregate expression(s)]
+- CometLocalTableScan
```
During execution, the query can fail with:
```text
java.lang.UnsupportedOperationException:
Unsupported data type: [org.apache.spark.sql.types.NullType$] void
at org.apache.spark.sql.comet.util.Utils$.toArrowType(Utils.scala:155)
at org.apache.spark.sql.comet.util.Utils$.toArrowSchema(Utils.scala:202)
at
org.apache.spark.sql.comet.execution.arrow.CometArrowConverters$.rowToArrowBatchIter(...)
at org.apache.spark.sql.comet.CometLocalTableScanExec.doExecuteColumnar(...)
```
<img width="2318" height="796" alt="Image"
src="https://github.com/user-attachments/assets/835f3cfe-75d2-4f23-8fbf-889bfbadb8b4";
/>
The issue occurs because `CometLocalTableScanExec` still attempts Arrow
conversion for `NullType` schemas even though `NullType` is unsupported in
downstream aggregate/shuffle native paths.
## Expected behavior
Unsupported `NullType` local table scans should gracefully fall back to
Spark execution instead of failing during Arrow schema conversion.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
