Re: [PR] [VL] Daily Update Velox Version (2024_05_17) [incubator-gluten]

2024-05-17 Thread via GitHub
rui-mo commented on PR #5781: URL: https://github.com/apache/incubator-gluten/pull/5781#issuecomment-2118652961 @FelixYBW Velox accepts a raw pointer instead of a shared_ptr now. We need to keep its lifecycle. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Use conf to control C2R occupied memory [incubator-gluten]

2024-05-17 Thread via GitHub
FelixYBW commented on PR #5799: URL: https://github.com/apache/incubator-gluten/pull/5799#issuecomment-2118639820 @jinchengchenghh -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [VL] Daily Update Velox Version (2024_05_17) [incubator-gluten]

2024-05-17 Thread via GitHub
FelixYBW commented on PR #5781: URL: https://github.com/apache/incubator-gluten/pull/5781#issuecomment-2118638004 why the spill executor must be hold in Gluten now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

(incubator-gluten) branch main updated: [VL][CI] disable nightly job on GHA (#5803)

2024-05-17 Thread yuanzhou
This is an automated email from the ASF dual-hosted git repository. yuanzhou pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new c12380d4f [VL][CI] disable nightly job

Re: [PR] [VL][CI] disable nightly job [incubator-gluten]

2024-05-17 Thread via GitHub
zhouyuan merged PR #5803: URL: https://github.com/apache/incubator-gluten/pull/5803 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL][CI] disable nightly job [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5803: URL: https://github.com/apache/incubator-gluten/pull/5803#issuecomment-2118620150 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL][CI] disable nightly job [incubator-gluten]

2024-05-17 Thread via GitHub
zhouyuan opened a new pull request, #5803: URL: https://github.com/apache/incubator-gluten/pull/5803 ## What changes were proposed in this pull request? disable nightly job on GHA #5765 ## How was this patch tested? no tests required -- This is an automated

Re: [PR] [VL] Adding PySpark related tool/scripts [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #4572: URL: https://github.com/apache/incubator-gluten/pull/4572#issuecomment-2118560083 This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. -- This is an automated

Re: [PR] [GLUTEN-5194][CH] support native overlay function [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5257: URL: https://github.com/apache/incubator-gluten/pull/5257#issuecomment-2118559949 This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. -- This is an automated

Re: [PR] [GLUTEN-5792][CORE] Fix build on macOS [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming commented on code in PR #5800: URL: https://github.com/apache/incubator-gluten/pull/5800#discussion_r1605642800 ## cpp/core/CMakeLists.txt: ## @@ -323,7 +326,7 @@ else() endif() target_link_libraries(gluten -PUBLIC Arrow::arrow Arrow::parquet) +PUBLIC

Re: [PR] [VL] Daily Update Velox Version (2024_05_18) [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5802: URL: https://github.com/apache/incubator-gluten/pull/5802#issuecomment-2118512659 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Daily Update Velox Version (2024_05_18) [incubator-gluten]

2024-05-17 Thread via GitHub
GlutenPerfBot opened a new pull request, #5802: URL: https://github.com/apache/incubator-gluten/pull/5802 Upstream Velox's New Commits: ```txt f9f8cd7b1 by Daniel Munoz, CoalesceIfDistanceLE supports reporting coalesced distance (9859) e8244c2e0 by Jialiang Tan, Fix flaky test

Re: [PR] [GLUTEN-5792][CORE] Fix build on macOS [incubator-gluten]

2024-05-17 Thread via GitHub
kecookier commented on code in PR #5800: URL: https://github.com/apache/incubator-gluten/pull/5800#discussion_r1605620631 ## cpp/core/CMakeLists.txt: ## @@ -323,7 +326,7 @@ else() endif() target_link_libraries(gluten -PUBLIC Arrow::arrow Arrow::parquet) +PUBLIC

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240518) [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5801: URL: https://github.com/apache/incubator-gluten/pull/5801#issuecomment-2118472950 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240518) [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5801: URL: https://github.com/apache/incubator-gluten/pull/5801#issuecomment-2118472807 https://github.com/apache/incubator-gluten/issues/1632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240518) [incubator-gluten]

2024-05-17 Thread via GitHub
kyligence-git opened a new pull request, #5801: URL: https://github.com/apache/incubator-gluten/pull/5801 Auto commit by gluten daily build, please check the build status and merge it if it's green. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [DNM] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
FelixYBW commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2118207602 > please do not merge, GetArrayItemTransformer need refactor. marked in title. Feel free to remove once refactor is done. -- This is an automated message from the

Re: [I] Gluten 1.1.1 does not work on Spark 3.3.1 on EMR [incubator-gluten]

2024-05-17 Thread via GitHub
FelixYBW commented on issue #5798: URL: https://github.com/apache/incubator-gluten/issues/5798#issuecomment-2118170925 Do you mean AWS EMR? I don't think Gluten can work with AWS EMR. Gluten relies on many Spark APIs and hacked several Spark files, AWS EMR have much optimizations to Spark

Re: [PR] [VL] Support simulate task spilling in GenericBenchmark [incubator-gluten]

2024-05-17 Thread via GitHub
GlutenPerfBot commented on PR #5795: URL: https://github.com/apache/incubator-gluten/pull/5795#issuecomment-2118169593 = Performance report for TPCDS SF2000 with Velox backend, for reference only query

Re: [I] Gluten 1.1.1 does not work on Spark 3.3.1 on EMR [incubator-gluten]

2024-05-17 Thread via GitHub
astarrr commented on issue #5798: URL: https://github.com/apache/incubator-gluten/issues/5798#issuecomment-2118027598 I upload some screenshots. Here with the Amazon jar there is no method `isSubquery`.

Re: [I] Reduce GitHub Action usage according to ASF project allowance [incubator-gluten]

2024-05-17 Thread via GitHub
FelixYBW commented on issue #5765: URL: https://github.com/apache/incubator-gluten/issues/5765#issuecomment-2117966550 > One thought is, we can make some PR-triggered jobs only run nightly, such as Java-17 tests, etc., assuming they rarely fail. make sense. -- This is an

(incubator-gluten) branch main updated: [VL] Support simulate task spilling in GenericBenchmark (#5795)

2024-05-17 Thread marong
This is an automated email from the ASF dual-hosted git repository. marong pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 63c166d62 [VL] Support simulate task

Re: [PR] [VL] Support simulate task spilling in GenericBenchmark [incubator-gluten]

2024-05-17 Thread via GitHub
marin-ma merged PR #5795: URL: https://github.com/apache/incubator-gluten/pull/5795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [VL] Refine evict logic in sort shuffle writer (#5786)

2024-05-17 Thread marong
This is an automated email from the ASF dual-hosted git repository. marong pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new b8f8154a6 [VL] Refine evict logic in sort

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-17 Thread via GitHub
marin-ma merged PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [GLUTEN-5792][CORE] Fix build on macOS [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5800: URL: https://github.com/apache/incubator-gluten/pull/5800#issuecomment-2117715229 https://github.com/apache/incubator-gluten/issues/5792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-5792][CORE] Fix build on macOS [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming opened a new pull request, #5800: URL: https://github.com/apache/incubator-gluten/pull/5800 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) (Fixes: \#5792) -- This is an automated message from the Apache

Re: [PR] Use conf to control C2R occupied memory [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5799: URL: https://github.com/apache/incubator-gluten/pull/5799#issuecomment-2117676334 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] Use conf to control C2R occupied memory [incubator-gluten]

2024-05-17 Thread via GitHub
XinShuoWang opened a new pull request, #5799: URL: https://github.com/apache/incubator-gluten/pull/5799 ## What changes were proposed in this pull request? In the current design, the Column2Row operation is completed in one go, which consumes a lot of memory and causes the program

Re: [PR] Use conf to control C2R occupied memory [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5799: URL: https://github.com/apache/incubator-gluten/pull/5799#issuecomment-2117676832 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Reduce GitHub Action usage according to ASF project allowance [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on issue #5765: URL: https://github.com/apache/incubator-gluten/issues/5765#issuecomment-2117608570 One thought is, we can make some PR-triggered jobs only run nightly, such as Java-17 tests, etc., assuming they rarely fail. -- This is an automated message from the

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2117551983 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(incubator-gluten) branch main updated: [CORE] Unify the transforming for shuffle expression (#5793)

2024-05-17 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new b6c6e264d [CORE] Unify the transforming

Re: [PR] [CORE] Unify the transforming for shuffle expression [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE merged PR #5793: URL: https://github.com/apache/incubator-gluten/pull/5793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2117523138 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2117521216 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

(incubator-gluten) branch main updated: [VL][CI] Cache native libraries to re-use them in Spark test jobs (#5768)

2024-05-17 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 1454aaafe [VL][CI] Cache native libraries

Re: [I] [VL] Reuse native build artifact for different spark version to save resources and time [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE closed issue #5721: [VL] Reuse native build artifact for different spark version to save resources and time URL: https://github.com/apache/incubator-gluten/issues/5721 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [VL][CI] Cache native libraries to re-use them in Spark test jobs [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE merged PR #5768: URL: https://github.com/apache/incubator-gluten/pull/5768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2117417987 > I'd like to take unix_date, thanks. We have supported it in both Gluten & Velox. Just changed its state in the list. Thanks!

Re: [PR] Extend adapter setup to support Debian distributions [incubator-gluten]

2024-05-17 Thread via GitHub
Preetesh2110 closed pull request #5540: Extend adapter setup to support Debian distributions URL: https://github.com/apache/incubator-gluten/pull/5540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]

2024-05-17 Thread via GitHub
NEUpanning commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2117366460 I'd like to take unix_date, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [VL] Fix warnings caused by -Wunused-but-set-variable [incubator-gluten]

2024-05-17 Thread via GitHub
kecookier commented on PR #5797: URL: https://github.com/apache/incubator-gluten/pull/5797#issuecomment-2117352501 @zhztheplayer @zuochunwei Could you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [VL] Fix warnings caused by -Wunused-but-set-variable [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5797: URL: https://github.com/apache/incubator-gluten/pull/5797#issuecomment-2117351007 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Fix warnings caused by -Wunused-but-set-variable [incubator-gluten]

2024-05-17 Thread via GitHub
kecookier opened a new pull request, #5797: URL: https://github.com/apache/incubator-gluten/pull/5797 ## What changes were proposed in this pull request? Fix warnings caused by -Wunused-but-set-variable ``` /root/gluten-ups/cpp/velox/substrait/SubstraitToVeloxPlan.cc:2224:90:

(incubator-gluten) branch main updated: Revert "fix warnings by -Wunused-but-set-variable"

2024-05-17 Thread zhaokuo
This is an automated email from the ASF dual-hosted git repository. zhaokuo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new aa9221af8 Revert "fix warnings by

(incubator-gluten) branch main updated: fix warnings by -Wunused-but-set-variable

2024-05-17 Thread zhaokuo
This is an automated email from the ASF dual-hosted git repository. zhaokuo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 30cc3b6cf fix warnings by

Re: [PR] [CORE] Refactor ExpressionTransformer [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5796: URL: https://github.com/apache/incubator-gluten/pull/5796#issuecomment-2117260775 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Refactor ExpressionTransformer [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5796: URL: https://github.com/apache/incubator-gluten/pull/5796#issuecomment-2117260432 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1604684880 ## gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala: ## @@ -164,20 +165,71 @@ private[gluten] class GlutenDriverPlugin extends DriverPlugin

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2117149320 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2117147525 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Rework planner C2R / R2C code with new transition facilities [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5767: URL: https://github.com/apache/incubator-gluten/pull/5767#issuecomment-2117143742 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Support simulate task spilling in GenericBenchmark [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5795: URL: https://github.com/apache/incubator-gluten/pull/5795#issuecomment-2117127942 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Support simulate task spilling in GenericBenchmark [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha opened a new pull request, #5795: URL: https://github.com/apache/incubator-gluten/pull/5795 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [CORE] Unify shuffle expression transformer [incubator-gluten]

2024-05-17 Thread via GitHub
exmy commented on PR #5793: URL: https://github.com/apache/incubator-gluten/pull/5793#issuecomment-2117109887 cc @PHILO-HE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [CORE] Unify shuffle expression transformer [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5793: URL: https://github.com/apache/incubator-gluten/pull/5793#issuecomment-2117109869 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Unify shuffle expression transformer [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5793: URL: https://github.com/apache/incubator-gluten/pull/5793#issuecomment-2117109456 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [CORE] Unify shuffle expression transformer [incubator-gluten]

2024-05-17 Thread via GitHub
exmy opened a new pull request, #5793: URL: https://github.com/apache/incubator-gluten/pull/5793 ## What changes were proposed in this pull request? ## How was this patch tested? PASS CI -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-17 Thread via GitHub
kerwin-zk commented on code in PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#discussion_r1604611639 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.cc: ## @@ -154,83 +127,71 @@ arrow::Status VeloxSortBasedShuffleWriter::write(std::shared_ptr return

Re: [I] [VL] Build failed for macOS [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming commented on issue #5792: URL: https://github.com/apache/incubator-gluten/issues/5792#issuecomment-2117106521 Seems introduced by https://github.com/apache/incubator-gluten/pull/5675 . I'd suggest add a CI build check for macOS, Gluten's build fail on macOS quite often. --

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-17 Thread via GitHub
marin-ma commented on code in PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#discussion_r1604605226 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.cc: ## @@ -154,83 +127,71 @@ arrow::Status VeloxSortBasedShuffleWriter::write(std::shared_ptr return

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-17 Thread via GitHub
kerwin-zk commented on code in PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#discussion_r1604600289 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.cc: ## @@ -154,83 +127,71 @@ arrow::Status VeloxSortBasedShuffleWriter::write(std::shared_ptr return

Re: [PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-17 Thread via GitHub
kecookier commented on PR #5784: URL: https://github.com/apache/incubator-gluten/pull/5784#issuecomment-2117088979 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-17 Thread via GitHub
kecookier commented on PR #5784: URL: https://github.com/apache/incubator-gluten/pull/5784#issuecomment-2117085647 Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [VL] Refine evict logic in sort shuffle writer [incubator-gluten]

2024-05-17 Thread via GitHub
marin-ma commented on PR #5786: URL: https://github.com/apache/incubator-gluten/pull/5786#issuecomment-2117081711 @kerwin-zk Just rebased. Could you help to review again? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

(incubator-gluten) branch main updated: [VL] Use MemConfig to replace MemConfigMutable to make the code cleaner and to improve performance (#5784)

2024-05-17 Thread hongze
This is an automated email from the ASF dual-hosted git repository. hongze pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 038d9cbaa [VL] Use MemConfig to replace

Re: [PR] [VL] Use MemConfig to replace MemConfigMutable, which makes the code cleaner and may also lead to some performance improvements. [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer merged PR #5784: URL: https://github.com/apache/incubator-gluten/pull/5784 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2117039157 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

(incubator-gluten) branch main updated: [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer (#5785)

2024-05-17 Thread marong
This is an automated email from the ASF dual-hosted git repository. marong pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 70ee0f411 [VL] Add BufferedOutputStream

Re: [PR] [VL] Add BufferedOutputStream to track the memory usage in PrestoSerializer [incubator-gluten]

2024-05-17 Thread via GitHub
marin-ma merged PR #5785: URL: https://github.com/apache/incubator-gluten/pull/5785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2117021394 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116996342 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116991395 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] Build failed for macOS [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming opened a new issue, #5792: URL: https://github.com/apache/incubator-gluten/issues/5792 ### Problem description ``` gluten/cpp/core/shuffle/Partitioner.h:21:10: fatal error: 'folly/container/F14Map.h' file not found #include ^~

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2116951408 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116949559 please do not merge, GetArrayItemTransformer need refactor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [VL] Enable rint function [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5791: URL: https://github.com/apache/incubator-gluten/pull/5791#issuecomment-2116947051 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Enable rint function [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5791: URL: https://github.com/apache/incubator-gluten/pull/5791#issuecomment-2116946693 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Enable rint function [incubator-gluten]

2024-05-17 Thread via GitHub
zhli1142015 opened a new pull request, #5791: URL: https://github.com/apache/incubator-gluten/pull/5791 ## What changes were proposed in this pull request? Enable rint function. ## How was this patch tested? UT. -- This is an automated message from the Apache

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-17 Thread via GitHub
xumingming commented on code in PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#discussion_r1604487543 ## gluten-core/src/test/scala/org/apache/gluten/execution/WholeStageTransformerSuite.scala: ## @@ -252,6 +252,34 @@ abstract class

Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5782: URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116944578 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-17 Thread via GitHub
GlutenPerfBot commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2116943967 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
jackylee-ch commented on code in PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#discussion_r1604473825 ## backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala: ## @@ -978,6 +978,23 @@ class

Re: [PR] [GLUTEN-4039][VL] Add array insert function for spark 3.4+ [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5790: URL: https://github.com/apache/incubator-gluten/pull/5790#issuecomment-2116923711 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-4039][VL] Add array insert function for spark 3.4+ [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5790: URL: https://github.com/apache/incubator-gluten/pull/5790#issuecomment-2116923340 https://github.com/apache/incubator-gluten/issues/4039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-4039][VL] Add array insert function for spark 3.4+ [incubator-gluten]

2024-05-17 Thread via GitHub
ivoson opened a new pull request, #5790: URL: https://github.com/apache/incubator-gluten/pull/5790 ## What changes were proposed in this pull request? Add support of `array_insert` function for velox backend. Velox side changes:

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha commented on code in PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#discussion_r1604464386 ## backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala: ## @@ -978,6 +978,23 @@ class ScalarFunctionsValidateSuite

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha commented on code in PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#discussion_r1604463462 ## backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala: ## @@ -978,6 +978,23 @@ class ScalarFunctionsValidateSuite

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
jackylee-ch commented on code in PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#discussion_r1604451514 ## backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala: ## @@ -978,6 +978,23 @@ class

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116876079 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #5789: URL: https://github.com/apache/incubator-gluten/pull/5789#issuecomment-2116875761 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Ensure get(GetArrayItem) function offloaded [incubator-gluten]

2024-05-17 Thread via GitHub
Yohahaha opened a new pull request, #5789: URL: https://github.com/apache/incubator-gluten/pull/5789 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [VL][CI] Cache native libraries to re-use them in Spark test jobs [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on code in PR #5768: URL: https://github.com/apache/incubator-gluten/pull/5768#discussion_r1604430830 ## .github/workflows/velox_docker.yml: ## @@ -494,13 +494,74 @@ jobs: GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-17 Thread via GitHub
exmy commented on code in PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#discussion_r1604328001 ## gluten-core/src/main/java/org/apache/gluten/vectorized/JniLibLoader.java: ## @@ -125,13 +139,10 @@ public static synchronized void unloadFromPath(String

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-17 Thread via GitHub
PHILO-HE commented on code in PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#discussion_r1604422499 ## shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala: ## @@ -208,7 +208,7 @@ class GlutenConfig(conf: SQLConf) extends Logging { def

Re: [PR] [GLUTEN-5741][CH] Fix core dump when executor exits [incubator-gluten]

2024-05-17 Thread via GitHub
exmy commented on PR #5787: URL: https://github.com/apache/incubator-gluten/pull/5787#issuecomment-2116857239 @baibaichen @zhztheplayer Could you help review this change? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Move memory reservation block computation logic into ListenableAllocator [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on PR #5770: URL: https://github.com/apache/incubator-gluten/pull/5770#issuecomment-2116856661 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-5777][VL] Supporting specify spark version when build [incubator-gluten]

2024-05-17 Thread via GitHub
GlutenPerfBot commented on PR #5778: URL: https://github.com/apache/incubator-gluten/pull/5778#issuecomment-2116854197 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [VL][CI] Cache native libraries to re-use them in Spark test jobs [incubator-gluten]

2024-05-17 Thread via GitHub
zhztheplayer commented on code in PR #5768: URL: https://github.com/apache/incubator-gluten/pull/5768#discussion_r1604415212 ## .github/workflows/velox_docker.yml: ## @@ -494,13 +494,74 @@ jobs: GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \

  1   2   >