ArnavBalyan opened a new issue, #8453: URL: https://github.com/apache/incubator-gluten/issues/8453
### Backend VL (Velox) ### Bug description Currently the Columnar Cache Serializer has an implicit expection for receiving light batches. When upstream operator produces a heavy batch it fails. `2025-01-02T15:58:41.7526781Z org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 310.0 failed 1 times, most recent failure: Lost task 0.0 in stage 310.0 (TID 462) (5dc5d78faedf executor driver): java.lang.UnsupportedOperationException: Input batch is not light batch 2025-01-02T15:58:41.7527897Z at org.apache.gluten.columnarbatch.ColumnarBatches.getIndicatorVector(ColumnarBatches.java:385) 2025-01-02T15:58:41.7528539Z at org.apache.gluten.columnarbatch.ColumnarBatches.getNativeHandle(ColumnarBatches.java:397) 2025-01-02T15:58:41.7529248Z at org.apache.spark.sql.execution.ColumnarCachedBatchSerializer$$anon$1.next(ColumnarCachedBatchSerializer.scala:184) 2025-01-02T15:58:41.7530071Z at org.apache.spark.sql.execution.ColumnarCachedBatchSerializer$$anon$1.next(ColumnarCachedBatchSerializer.scala:172) 2025-01-02T15:58:41.7530659Z at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) 2025-01-02T15:58:41.7531112Z at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:224) 2025-01-02T15:58:41.7531765Z at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:302) 2025-01-02T15:58:41.7532346Z at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1531) 2025-01-02T15:58:41.7532969Z at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1458) 2025-01-02T15:58:41.7533561Z at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1522) 2025-01-02T15:58:41.7534142Z at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1349) 2025-01-02T15:58:41.7534584Z at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:378) 2025-01-02T15:58:41.7534979Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:329) 2025-01-02T15:58:41.7535635Z at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 2025-01-02T15:58:41.7536316Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7536694Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7537081Z at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 2025-01-02T15:58:41.7537525Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7537893Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7538275Z at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 2025-01-02T15:58:41.7538719Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7539081Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7539611Z at org.apache.gluten.execution.ColumnarInputRDDsWrapper.$anonfun$getIterators$1(WholeStageTransformer.scala:559) 2025-01-02T15:58:41.7540176Z at scala.collection.immutable.List.flatMap(List.scala:366) 2025-01-02T15:58:41.7540708Z at org.apache.gluten.execution.ColumnarInputRDDsWrapper.getIterators(WholeStageTransformer.scala:550) 2025-01-02T15:58:41.7541467Z at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:48) 2025-01-02T15:58:41.7542048Z at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25) 2025-01-02T15:58:41.7542475Z at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37) 2025-01-02T15:58:41.7543110Z at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46) 2025-01-02T15:58:41.7543697Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7544066Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7544454Z at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 2025-01-02T15:58:41.7545001Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7545368Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7545754Z at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 2025-01-02T15:58:41.7546191Z at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367) 2025-01-02T15:58:41.7546551Z at org.apache.spark.rdd.RDD.iterator(RDD.scala:331) 2025-01-02T15:58:41.7546986Z at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) 2025-01-02T15:58:41.7547640Z at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101) 2025-01-02T15:58:41.7548226Z at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) 2025-01-02T15:58:41.7548723Z at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) 2025-01-02T15:58:41.7549126Z at org.apache.spark.scheduler.Task.run(Task.scala:139) 2025-01-02T15:58:41.7549545Z at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554) 2025-01-02T15:58:41.7550015Z at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529) 2025-01-02T15:58:41.7550435Z at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557) 2025-01-02T15:58:41.7551036Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 2025-01-02T15:58:41.7551572Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 2025-01-02T15:58:41.7551971Z at java.lang.Thread.run(Thread.java:748)` ### Spark version None ### Spark configurations _No response_ ### System information _No response_ ### Relevant logs _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
