andygrove commented on issue #7267: URL: https://github.com/apache/datafusion-comet/issues/7267#issuecomment-6644384904
I went through the parts of the code this audit didn't cover (expressions, aggregates, Comet's own operators, the shuffle read and row conversion paths, and the JVM side) and filed what I found as #7494 to #7503. #7494 is the most serious: with native columnar-to-row enabled (it's opt-in), its row offsets wrap past 2 GiB and the JVM reads rows from before the buffer. #7504 fixes it. #7495 (explode) and #7496 (dictionary blocks on shuffle read) are on by default, and I reproduced both with unit tests. On the "Not yet checked" list: Celeborn shuffle read framing looks fine, since frame lengths are validated as Long and frames are bounded; the only exposure there is the dictionary expansion in #7496. Sort-merge join spill read-back restores one batch per spill file without concatenating, although the join's coalescer panics instead of returning an error on overflow (listed in #7503). Python UDF output is passed through without merging, and the input side already plans byte-bounded ranges. The Iceberg Java fallback reader goes through `SparkColumnarArrowReader`, which merges batches by row count only. That's #7502. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
