parthchandra opened a new issue, #5391:
URL: https://github.com/apache/datafusion-comet/issues/5391
### Describe the bug
A job with a huge number of scan partitions fails at stage submission:
```
Task serialization failed: java.lang.OutOfMemoryError: Required array
length 2147483639 + 794 is too large
at java.io.ByteArrayOutputStream.ensureCapacity(...)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(...)
at
org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1528)
```
A single `byte[]` can't exceed ~2GB, and the driver is serializing one
object bigger than that. The failing job had ~38.7M partitions (multi-petabyte
dataset), native scan into a native shuffle. The same job runs fine on plain
Spark, so it's something Comet does differently, not just scale.
### Analysis
`DAGScheduler.submitMissingTasks` serializes the `(RDD,
ShuffleDependency)` pair once into a single broadcast byte array, which must
fit in ~2GB. On the native shuffle path,
`CometShuffleDependency.nativeShuffleSpec` is a non-transient field holding a
`NativeExecContext` whose `perPartitionByKey: Map[String, Array[Array[Byte]]]`
carries one
serialized scan plan-data blob (the partition's file list) per map
partition. With ~38.7M partitions that's ~38.7M protobufs baked into the
broadcast task binary, so it exceeds 2GB — even though each write task only
ever reads its own partition's slice.
Spark's `FileScanRDD` keeps per-partition file lists `@transient`; each
partition's files ride in its own `Partition` object and are serialized
separately per task, so the full list is never included in the shared
broadcast blob.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
