Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

baibaichen closed issue #4943: [CH] New byte buffer takes most of time in SourceFromJavalter::generate URL: https://github.com/apache/incubator-gluten/issues/4943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

zzcclp commented on issue #4943: URL: https://github.com/apache/incubator-gluten/issues/4943#issuecomment-2003077901 > 原因：查询运行过程中，有26200次new byte[1024*1024] 操作，平均每个task有78次，总耗时8s, 而查询耗时也就30+s > > 问题：为什么会走带copy的OnHeapCopyShuffleInputStream，没走zero-copy的LowCopyNettyShuffleInputStream

Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

taiyang-li closed issue #4943: [CH] New byte buffer takes most of time in SourceFromJavalter::generate URL: https://github.com/apache/incubator-gluten/issues/4943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

taiyang-li commented on issue #4943: URL: https://github.com/apache/incubator-gluten/issues/4943#issuecomment-1993717645 修改配置后 " --conf spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager" 火焰图如下：

Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

taiyang-li commented on issue #4943: URL: https://github.com/apache/incubator-gluten/issues/4943#issuecomment-1993693855 原因：查询运行过程中，有26200次new byte[1024*1024] 操作，平均每个task有78次，总耗时8s, 而查询耗时也就30+s 问题：为什么会走带copy的OnHeapCopyShuffleInputStream，没走zero-copy的LowCopyNettyShuffleInputStream

Re: [I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

zhanglistar commented on issue #4943: URL: https://github.com/apache/incubator-gluten/issues/4943#issuecomment-1993376883 optoruntime::new_array_c可能是传入的`memory.m_capacity`过大，另外jdk中会对内存进行memset，导致该函数占用过多的时间。 -- This is an automated message from the Apache Git Service. To respond to the

[I] [CH] New byte buffer takes most of time in SourceFromJavalter::generate [incubator-gluten]

taiyang-li opened a new issue, #4943: URL: https://github.com/apache/incubator-gluten/issues/4943 ### Description  ``` bool