advancedxy commented on issue #67: URL: https://github.com/apache/arrow-datafusion-comet/issues/67#issuecomment-1964319590
> Probably we can print java/scala test durations in the CI and then we can decided which parts are most time consuming After enable test duration, the most time consuming tests(exceeds one minute) on X86 mac runners with JDK 1.8 are: ``` - Comet columnar shuffle shuffle: different data type (8 minutes, 5 seconds) - all types, with nulls (23 minutes, 55 seconds) - Comet columnar shuffle shuffle: different data type (8 minutes, 18 seconds) - Comet columnar shuffle shuffle: different data type (5 minutes, 59 seconds) - Comet columnar shuffle shuffle: different data type (6 minutes, 50 seconds) - fix: Too many task completion listener of ArrowReaderIterator causes OOM (34 minutes, 33 seconds) - Comet columnar shuffle shuffle: different data type (7 minutes, 22 seconds) - Comet columnar shuffle shuffle: different data type (6 minutes, 59 seconds) ``` The source is https://github.com/apache/arrow-datafusion-comet/actions/runs/8047518357/job/21976917198?pr=116 The `fix: Too many task completion listener of ArrowReaderIterator causes OOM (34 minutes, 33 seconds)` case looks extreme. Maybe we should fix it first, cc @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
