Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116876079 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116875761 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha opened a new pull request, #5789: URL: https://github.com/apache/incubator-gluten/pull/5789 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [VL][CI] Cache native libraries to re-use them in Spark test jobs [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on code in PR #5768: URL: https://github.com/apache/incubator-gluten/pull/5768#discussion_r1604430830 ## .github/workflows/velox_docker.yml: ## @@ -494,13 +494,74 @@ jobs: GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-17 Thread via GitHub
exmy commented on code in PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#discussion_r1604328001 ## gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java: ## @@ -125,13 +139,10 @@ public static synchronized void unloadFromPath(String

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on code in PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#discussion_r1604422499 ## shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala: ## @@ -208,7 +208,7 @@ class GlutenConfig(conf: SQLConf) extends Logging { def

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-17 Thread via GitHub
exmy commented on PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#issuecomment-2116857239 @baibaichen @zhztheplayer Could you help review this change? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116856661 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-17 Thread via GitHub
GlutenPerfBot commented on PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778#issuecomment-2116854197 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [VL][CI] Cache native libraries to re-use them in Spark test jobs [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on code in PR #5768: URL: https://github.com/apache/incubator-gluten/pull/5768#discussion_r1604415212 ## .github/workflows/velox_docker.yml: ## @@ -494,13 +494,74 @@ jobs: GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#issuecomment-2116789351 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-17 Thread via GitHub
yaooqinn merged PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Optimize buildbundle-veloxbe.sh to be able to build for one specified Spark version [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming commented on issue #5777: URL: https://github.com/apache/incubator-gluten/issues/5777#issuecomment-2116771953 Closing it as PR merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Optimize buildbundle-veloxbe.sh to be able to build for one specified Spark version [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming closed issue #5777: Optimize buildbundle-veloxbe.sh to be able to build for one specified Spark version URL: https://github.com/apache/incubator-gluten/issues/5777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-16 Thread via GitHub
zhztheplayer commented on code in PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#discussion_r1604381360 ## gluten-core/src/test/scala/org/apache/gluten/execution/WholeStageTransformerSuite.scala: ## @@ -252,6 +252,34 @@ abstract class

Re: [PR] [VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma merged PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Arrow CSV reader peak memory is very large [incubator-gluten]

2024-05-16 Thread via GitHub
zhztheplayer commented on issue #5766: URL: https://github.com/apache/incubator-gluten/issues/5766#issuecomment-2116709386 > @zhztheplayer do you remember? I can't recall that. But it doesn't make sense to buffer all data for a reader. I suppose @jinchengchenghh is looking

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116700672 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-16 Thread via GitHub
FelixYBW merged PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116680656 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [CH] Make sure that all processors are cancelled before releasing global resources in native engine and after yarn kill or driver shutdown command comes [incubator-gluten]

2024-05-16 Thread via GitHub
taiyang-li commented on issue #5788: URL: https://github.com/apache/incubator-gluten/issues/5788#issuecomment-2116595445 猜想:当driver发送shutdown command到executor时,executor执行ShutdownHookManager中注册过的hook函数,其中包括释放native engine中的全局资源。但是从上面的log看,此时native engine中,source算子还在执行,它会依赖已经被释放的全局资源(global

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-16 Thread via GitHub
exmy commented on code in PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#discussion_r1604328001 ## gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java: ## @@ -125,13 +139,10 @@ public static synchronized void unloadFromPath(String

[I] [CH] Make sure that all processors are cancelled before releasing global resources in native engine and after yarn kill or driver shutdown command comes [incubator-gluten]

2024-05-16 Thread via GitHub
taiyang-li opened a new issue, #5788: URL: https://github.com/apache/incubator-gluten/issues/5788 ### Backend CH (ClickHouse) ### Bug description

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-16 Thread via GitHub
GlutenPerfBot commented on PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2116587518 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [I] [CELEBORN] Fix invoke celebornShuffleId exception [incubator-gluten]

2024-05-16 Thread via GitHub
kerwin-zk closed issue #5775: [CELEBORN] Fix invoke celebornShuffleId exception URL: https://github.com/apache/incubator-gluten/issues/5775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-5775][CELEBORN] Fix invoke celebornShuffleId exception [incubator-gluten]

2024-05-16 Thread via GitHub
kerwin-zk merged PR #5776: URL: https://github.com/apache/incubator-gluten/pull/5776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [GLUTEN-5775][CELEBORN] Fix invoke celebornShuffleId exception [incubator-gluten]

2024-05-16 Thread via GitHub
kerwin-zk commented on PR #5776: URL: https://github.com/apache/incubator-gluten/pull/5776#issuecomment-2116585046 LGTM. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#issuecomment-2116584604 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#issuecomment-2116584383 https://github.com/apache/incubator-gluten/issues/5741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-16 Thread via GitHub
kecookier commented on PR #5784: URL: https://github.com/apache/incubator-gluten/pull/5784#issuecomment-2116574343 @zuochunwei @zhztheplayer @Yohahaha Could you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma commented on PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#issuecomment-2116566997 @kerwin-zk Could you help to review commit https://github.com/apache/incubator-gluten/pull/5786/commits/8c195f2b45e7e302014b25da630040e870d5bac0? Thanks! -- This is an

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma commented on PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#issuecomment-2116566604 Could you help to review commit https://github.com/apache/incubator-gluten/pull/5786/commits/8c195f2b45e7e302014b25da630040e870d5bac0? Thanks! -- This is an automated

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#issuecomment-2116566558 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#issuecomment-2116566430 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma opened a new pull request, #5786: URL: https://github.com/apache/incubator-gluten/pull/5786 A follow-up PR of https://github.com/apache/incubator-gluten/pull/5675 Remove the mapping from input vector to row index ranges. The process can be done in one loop. -- This is an

Re: [PR] [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5785: URL: https://github.com/apache/incubator-gluten/pull/5785#issuecomment-2116561475 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5785: URL: https://github.com/apache/incubator-gluten/pull/5785#issuecomment-2116561319 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116561005 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma commented on PR #5785: URL: https://github.com/apache/incubator-gluten/pull/5785#issuecomment-2116561672 @kerwin-zk Could you help to review commit https://github.com/apache/incubator-gluten/pull/5785/commits/8efc88258c6274205efc6ef3bec4bb6064ba995b ? Thanks! -- This is an

[PR] [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer. [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma opened a new pull request, #5785: URL: https://github.com/apache/incubator-gluten/pull/5785 A follow-up PR of https://github.com/apache/incubator-gluten/pull/5675 Discussion: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602744985 -- This is an automated

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116552304 > > > @Yohahaha Seems to be that the block reservation code can be very suitable to become a individual `AllocationListener`. Would you like to check on that? > > > >

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116552140 > ListenableMemoryAllocator make sense for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116550590 After some tries I would like to say ReservationListener is a hook to another memory management system and should be as simple as possible, MemoryAllocator is a common class

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-16 Thread via GitHub
ulysses-you merged PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Dynamically sizing off-heap memory [incubator-gluten]

2024-05-16 Thread via GitHub
ulysses-you closed issue #5438: Dynamically sizing off-heap memory URL: https://github.com/apache/incubator-gluten/issues/5438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5784: URL: https://github.com/apache/incubator-gluten/pull/5784#issuecomment-2116541000 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-16 Thread via GitHub
kecookier opened a new pull request, #5784: URL: https://github.com/apache/incubator-gluten/pull/5784 ## What changes were proposed in this pull request? Currently, in the Gluten code, once `velox::core::MemConfigMutable` is initialized, the key-value pairs are not modified anymore. This

Re: [PR] [VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2116540085 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116535745 > > @Yohahaha Seems to be that the block reservation code can be very suitable to become a individual `AllocationListener`. Would you like to check on that? > > I

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1604269082 ## cpp/core/shuffle/HashPartitioner.cc: ## @@ -16,9 +16,28 @@ */ #include "shuffle/HashPartitioner.h" +#include Review Comment: Please remove.

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116530587 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2116518948 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS setting logic [incubator-gluten]

2024-05-16 Thread via GitHub
PHILO-HE merged PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS generation logic [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on code in PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769#discussion_r1604256354 ## cpp/velox/CMakeLists.txt: ## @@ -52,6 +47,19 @@ if (NOT DEFINED VELOX_HOME) message(STATUS "Set VELOX_HOME to ${VELOX_HOME}") endif()

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116501561 > @Yohahaha Seems to be that the block reservation code can be very suitable to become a individual `AllocationListener`. Would you like to check on that? I see it,

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116499339 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
zhztheplayer commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116497653 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-5773] Update aws-sdk-cpp version to 1.11.285 (from 1.11.169) [incubator-gluten]

2024-05-16 Thread via GitHub
yma11 commented on PR #5774: URL: https://github.com/apache/incubator-gluten/pull/5774#issuecomment-2116493470 > Looks good! Thanks but wait for a while. May need some extra change. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-16 Thread via GitHub
jinchengchenghh commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1604244527 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS generation logic [incubator-gluten]

2024-05-16 Thread via GitHub
marin-ma commented on code in PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769#discussion_r1604244812 ## cpp/velox/CMakeLists.txt: ## @@ -52,6 +47,19 @@ if (NOT DEFINED VELOX_HOME) message(STATUS "Set VELOX_HOME to ${VELOX_HOME}") endif()

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-16 Thread via GitHub
jinchengchenghh commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1604244527 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-16 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116483438 @Yohahaha Seems to be that the block reservation code can be very suitable to become a individual `AllocationListener`. Would you like to check on that? -- This is an

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS generation logic [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769#issuecomment-2116481325 @marin-ma @PHILO-HE please help take a look, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [GLUTEN-4844] [CH] support native write csv file [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #4846: URL: https://github.com/apache/incubator-gluten/pull/4846#issuecomment-2116479483 This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks. -- This is an automated

Re: [PR] [GLUTEN-5192][VL] Copy bundle jars to dist dir [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5193: URL: https://github.com/apache/incubator-gluten/pull/5193#issuecomment-2116479419 This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. -- This is an automated

Re: [PR] [GLUTEN-4844] [CH] support native write csv file [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] closed pull request #4846: [GLUTEN-4844] [CH] support native write csv file URL: https://github.com/apache/incubator-gluten/pull/4846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-16 Thread via GitHub
ulysses-you commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1604240574 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116473765 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS generation logic [incubator-gluten]

2024-05-16 Thread via GitHub
Yohahaha commented on PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769#issuecomment-2116473739 > Is there impact to build process? no changes, just a little refactor. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116473526 https://github.com/apache/incubator-gluten/issues/5757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming opened a new pull request, #5782: URL: https://github.com/apache/incubator-gluten/pull/5782 ## What changes were proposed in this pull request? If generator function's input is already Attribute reference, we omit the introduction of the ProjectExec. (Fixes: \#5757)

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2116472976 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-16 Thread via GitHub
jinchengchenghh commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1604232341 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [VL][CI] disable Velox UT [incubator-gluten]

2024-05-16 Thread via GitHub
GlutenPerfBot commented on PR #5780: URL: https://github.com/apache/incubator-gluten/pull/5780#issuecomment-2116463325 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [CORE] Avoid copy in ByteLiteralNode [incubator-gluten]

2024-05-16 Thread via GitHub
jinchengchenghh commented on PR #5763: URL: https://github.com/apache/incubator-gluten/pull/5763#issuecomment-2116455255 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2116449714 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming commented on PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778#issuecomment-2116443676 @FelixYBW made the following changes: 1. Added the valid versions into doc, it makes the description a little bit long, the whole table for options is reformatted, but

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2116442532 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Daily Update Velox Version (2024_05_17) [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5781: URL: https://github.com/apache/incubator-gluten/pull/5781#issuecomment-2116431246 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Daily Update Velox Version (2024_05_17) [incubator-gluten]

2024-05-16 Thread via GitHub
rui-mo opened a new pull request, #5781: URL: https://github.com/apache/incubator-gluten/pull/5781 Upstream Velox's New Commits: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-5773] Update aws-sdk-cpp version to 1.11.285 (from 1.11.169) [incubator-gluten]

2024-05-16 Thread via GitHub
PHILO-HE commented on PR #5774: URL: https://github.com/apache/incubator-gluten/pull/5774#issuecomment-2116430637 @yma11, I note this lib is upgraded to 1.11.321 in upstream velox. Do we need to align with it in Gluten? https://github.com/facebookincubator/velox/pull/9756 -- This is

Re: [PR] [GLUTEN-5773] Update aws-sdk-cpp version to 1.11.285 (from 1.11.169) [incubator-gluten]

2024-05-16 Thread via GitHub
PHILO-HE commented on PR #5774: URL: https://github.com/apache/incubator-gluten/pull/5774#issuecomment-2116429191 > So we only updated the the aws-sdk-cpp in vcpkg. > > @PHILO-HE Is there any risk if we only update individual components in vcpkg but not vcpkg itself in long term?

Re: [PR] [GLUTEN-5771][VL] Add metrics for ColumnarArrowEvalPythonExec [incubator-gluten]

2024-05-16 Thread via GitHub
yma11 commented on PR #5772: URL: https://github.com/apache/incubator-gluten/pull/5772#issuecomment-2116423660 > @yma11 can you add a UI chart for the pyarrow UDF? Also add some implementation details? > > In theory we can convert Velox to Arrow in Velox pipeline, then pass the

Re: [PR] [VL][CI] disable Velox UT [incubator-gluten]

2024-05-16 Thread via GitHub
PHILO-HE merged PR #5780: URL: https://github.com/apache/incubator-gluten/pull/5780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Reduce GitHub Action usage according to ASF project allowance [incubator-gluten]

2024-05-16 Thread via GitHub
zhouyuan commented on issue #5765: URL: https://github.com/apache/incubator-gluten/issues/5765#issuecomment-2116419427 https://github.com/apache/incubator-gluten/pull/5780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [VL][CI] disable Velox UT [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5780: URL: https://github.com/apache/incubator-gluten/pull/5780#issuecomment-2116417902 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL][CI] disable Velox UT [incubator-gluten]

2024-05-16 Thread via GitHub
zhouyuan opened a new pull request, #5780: URL: https://github.com/apache/incubator-gluten/pull/5780 ## What changes were proposed in this pull request? This patch disabled Velox related cache/UT ## How was this patch tested? pass GHA -- This is an automated message

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2116416276 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-16 Thread via GitHub
FelixYBW commented on code in PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778#discussion_r1604203126 ## docs/get-started/build-guide.md: ## @@ -31,7 +31,8 @@ Please set them via `--`, e.g. `--build_type=Release`. | velox_home | Specify your own

Re: [I] Build failed on macOS(Apple Silicon) [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming closed issue #5709: Build failed on macOS(Apple Silicon) URL: https://github.com/apache/incubator-gluten/issues/5709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Build failed on macOS(Apple Silicon) [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming commented on issue #5709: URL: https://github.com/apache/incubator-gluten/issues/5709#issuecomment-2116409653 Related PR are all merged, closing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Reduce GitHub Action usage according to ASF project allowance [incubator-gluten]

2024-05-16 Thread via GitHub
zhouyuan commented on issue #5765: URL: https://github.com/apache/incubator-gluten/issues/5765#issuecomment-2116401390 @FelixYBW yes, we are now moving back some of the tests to our internal CI and also is re-constructing the CI to reduce the usage

Re: [PR] [GLUTEN-5709][VL] Fix macOS build [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming closed pull request #5710: [GLUTEN-5709][VL] Fix macOS build URL: https://github.com/apache/incubator-gluten/pull/5710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [GLUTEN-5709][VL] Fix macOS build [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming commented on PR #5710: URL: https://github.com/apache/incubator-gluten/pull/5710#issuecomment-2116369979 Already solved by other PR, closing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-16 Thread via GitHub
xumingming commented on PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778#issuecomment-2116366850 > The valid command is > > ``` > ./dev/buildbundle-veloxbe.sh ALL > ./dev/buildbundle-veloxbe.sh 3.2 > ./dev/buildbundle-veloxbe.sh 3.3 >

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240517) [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5779: URL: https://github.com/apache/incubator-gluten/pull/5779#issuecomment-2116337470 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240517) [incubator-gluten]

2024-05-16 Thread via GitHub
github-actions[bot] commented on PR #5779: URL: https://github.com/apache/incubator-gluten/pull/5779#issuecomment-2116337258 https://github.com/apache/incubator-gluten/issues/1632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240517) [incubator-gluten]

2024-05-16 Thread via GitHub
kyligence-git opened a new pull request, #5779: URL: https://github.com/apache/incubator-gluten/pull/5779 Auto commit by gluten daily build, please check the build status and merge it if it's green. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [GLUTEN-5771][VL] Add metrics for ColumnarArrowEvalPythonExec [incubator-gluten]

2024-05-16 Thread via GitHub
FelixYBW commented on PR #5772: URL: https://github.com/apache/incubator-gluten/pull/5772#issuecomment-2116238446 @yma11 can you add a UI chart for the pyarrow UDF? Also add some implementation details? In theory we can convert Velox to Arrow in Velox pipeline, then pass the arrow

Re: [PR] [VL] Refine CMAKE_CXX_FLAGS generation logic [incubator-gluten]

2024-05-16 Thread via GitHub
FelixYBW commented on PR #5769: URL: https://github.com/apache/incubator-gluten/pull/5769#issuecomment-2116232572 Is there impact to build process? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

  1   2   3   4   5   6   7   8   9   10   >