(incubator-gluten) branch main updated: [VL] Enable length function for binary type (#5761)

2024-05-15 Thread rui
This is an automated email from the ASF dual-hosted git repository. rui pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new cb02cdb0a [VL] Enable length function for

Re: [PR] [VL] Enable length function for binary type [incubator-gluten]

2024-05-15 Thread via GitHub
rui-mo merged PR #5761: URL: https://github.com/apache/incubator-gluten/pull/5761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [GLUTEN-5414] [VL] Support datasource v2 scan csv [incubator-gluten]

2024-05-15 Thread via GitHub
liujiayi771 commented on PR #5717: URL: https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2114084000 @jinchengchenghh Thanks! I will create an issue to trace this peak memory issue. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2114076185 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

(incubator-gluten) branch main updated: [VL] Daily Update Velox Version (2024_05_15) (#5748)

2024-05-15 Thread rui
This is an automated email from the ASF dual-hosted git repository. rui pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 888e1e244 [VL] Daily Update Velox Version

Re: [PR] [VL] Daily Update Velox Version (2024_05_15) [incubator-gluten]

2024-05-15 Thread via GitHub
rui-mo merged PR #5748: URL: https://github.com/apache/incubator-gluten/pull/5748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
ulysses-you commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1602624507 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [CORE] Optimize plan to use ByteBuffer and avoid copy in ByteLiteralNode [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5763: URL: https://github.com/apache/incubator-gluten/pull/5763#issuecomment-2113980115 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [CORE] Optimize plan to use ByteBuffer and avoid copy in ByteLiteralNode [incubator-gluten]

2024-05-15 Thread via GitHub
jinchengchenghh opened a new pull request, #5763: URL: https://github.com/apache/incubator-gluten/pull/5763 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [CORE] Optimize plan to use ByteBuffer and avoid copy in ByteLiteralNode [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5763: URL: https://github.com/apache/incubator-gluten/pull/5763#issuecomment-2113979987 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
jinchengchenghh commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1602573722 ## gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala: ## @@ -18,6 +18,15 @@ package org.apache.gluten.expression

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
ulysses-you commented on PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#issuecomment-2113972798 cc @jinchengchenghh @baibaichen @PHILO-HE thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602561639 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.cc: ## @@ -0,0 +1,349 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602560233 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.h: ## @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [PR] [GLUTEN-5414] [VL] Support datasource v2 scan csv [incubator-gluten]

2024-05-15 Thread via GitHub
jinchengchenghh commented on PR #5717: URL: https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2113952674 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [GLUTEN-5744][CH] Release native global resources manually on SIGTERM [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5762: URL: https://github.com/apache/incubator-gluten/pull/5762#issuecomment-2113932804 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5744][CH] Release native global resources manually on SIGTERM [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5762: URL: https://github.com/apache/incubator-gluten/pull/5762#issuecomment-2113932595 https://github.com/apache/incubator-gluten/issues/5744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-5744][CH] Release native global resources manually on SIGTERM [incubator-gluten]

2024-05-15 Thread via GitHub
taiyang-li opened a new pull request, #5762: URL: https://github.com/apache/incubator-gluten/pull/5762 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) (Fixes: \#5744) ## How was this patch tested? Production env.

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
ulysses-you commented on PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2113930583 I'm fine with this pr, looking forward some best practices, thank you @supermem613 @zhztheplayer -- This is an automated message from the Apache Git Service. To respond

(incubator-gluten) branch main updated: [VL] Move velox related configs to VeloxConfig.h (#5743)

2024-05-15 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 9d2a13bff [VL] Move velox related configs

Re: [PR] [VL] Move velox related configs to VeloxConfig.h [incubator-gluten]

2024-05-15 Thread via GitHub
PHILO-HE merged PR #5743: URL: https://github.com/apache/incubator-gluten/pull/5743 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602527162 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.h: ## @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602519192 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.h: ## @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2113907004 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2113906603 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1602507034 ## cpp/velox/shuffle/VeloxShuffleWriter.h: ## @@ -132,6 +132,8 @@ class VeloxShuffleWriter final : public ShuffleWriter { arrow::Status

Re: [PR] [VL] Enable length function for binary type [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5761: URL: https://github.com/apache/incubator-gluten/pull/5761#issuecomment-2113819533 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
zml1206 commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2113809806 Test failure is unrelated. cc @rui-mo @zhztheplayer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [VL] Move velox related configs to VeloxConfig.h [incubator-gluten]

2024-05-15 Thread via GitHub
Yohahaha commented on PR #5743: URL: https://github.com/apache/incubator-gluten/pull/5743#issuecomment-2113800172 @FelixYBW @zhztheplayer please help review, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [VL] Not fallback for function spark_partition_id and monotonically_increasing_id [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5046: URL: https://github.com/apache/incubator-gluten/pull/5046#issuecomment-2113783174 This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks. -- This is an automated

Re: [PR] [VL] Not fallback for function spark_partition_id and monotonically_increasing_id [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] closed pull request #5046: [VL] Not fallback for function spark_partition_id and monotonically_increasing_id URL: https://github.com/apache/incubator-gluten/pull/5046 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [GLUTEN-5759][CORE] Optimze checkGlutenOperatorMatch to show clearer error message [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5760: URL: https://github.com/apache/incubator-gluten/pull/5760#issuecomment-2113774762 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [GLUTEN-5759][CORE] Optimze checkGlutenOperatorMatch to show clearer error message [incubator-gluten]

2024-05-15 Thread via GitHub
xumingming opened a new pull request, #5760: URL: https://github.com/apache/incubator-gluten/pull/5760 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) (Fixes: \#5759) ## How was this patch tested? Existing UT.

Re: [PR] [GLUTEN-5759][CORE] Optimze checkGlutenOperatorMatch to show clearer error message [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5760: URL: https://github.com/apache/incubator-gluten/pull/5760#issuecomment-2113773427 https://github.com/apache/incubator-gluten/issues/5759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[I] Optimize checkGlutenOperatorMatch [incubator-gluten]

2024-05-15 Thread via GitHub
xumingming opened a new issue, #5759: URL: https://github.com/apache/incubator-gluten/issues/5759 ### Backend VL (Velox) ### Bug description With the following test case(which will fail, not important here, we want to see what the error message is): ```scala

Re: [PR] [VL][DNM] Test complex hash pr [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma commented on PR #5758: URL: https://github.com/apache/incubator-gluten/pull/5758#issuecomment-2113733429 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [VL][DNM] Test complex hash pr [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma opened a new pull request, #5758: URL: https://github.com/apache/incubator-gluten/pull/5758 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#issuecomment-2113731716 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2113718355 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5414] [VL] Support datasource v2 scan csv [incubator-gluten]

2024-05-15 Thread via GitHub
jinchengchenghh commented on PR #5717: URL: https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2113698935 > > I have found an issue: when reading large CSV files, for example, when a single CSV file in a table is 300M, the peak memory usage of arrow memory pool during

[I] Unnecessary ProjectExec generated for Generate function [incubator-gluten]

2024-05-15 Thread via GitHub
xumingming opened a new issue, #5757: URL: https://github.com/apache/incubator-gluten/issues/5757 ### Description For the following test case: ```scala test("test explode1") { withTempView("t1") { sql( """select * from values (array(1)),

Re: [PR] [VL] Daily Update Velox Version (2024_05_16) [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5756: URL: https://github.com/apache/incubator-gluten/pull/5756#issuecomment-2113678748 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2113678904 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [VL] Daily Update Velox Version (2024_05_16) [incubator-gluten]

2024-05-15 Thread via GitHub
GlutenPerfBot opened a new pull request, #5756: URL: https://github.com/apache/incubator-gluten/pull/5756 Upstream Velox's New Commits: ```txt 08ffe2207 by rui-mo, Add custom argument generators for Presto decimal functions (9715) b470e8521 by Ankita Victor, Remove global

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240516) [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5755: URL: https://github.com/apache/incubator-gluten/pull/5755#issuecomment-2113622004 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240516) [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5755: URL: https://github.com/apache/incubator-gluten/pull/5755#issuecomment-2113621784 https://github.com/apache/incubator-gluten/issues/1632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240516) [incubator-gluten]

2024-05-15 Thread via GitHub
kyligence-git opened a new pull request, #5755: URL: https://github.com/apache/incubator-gluten/pull/5755 Auto commit by gluten daily build, please check the build status and merge it if it's green. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [GLUTEN-4836][VL]Add support for WindowGroupLimitExec in gluten [incubator-gluten]

2024-05-15 Thread via GitHub
EpsilonPrime commented on code in PR #5398: URL: https://github.com/apache/incubator-gluten/pull/5398#discussion_r1602259409 ## gluten-core/src/main/resources/substrait/proto/substrait/algebra.proto: ## @@ -495,6 +504,7 @@ message Rel { GenerateRel generate = 17;

Re: [I] [VL] Support customize options for parquet native write [incubator-gluten]

2024-05-15 Thread via GitHub
FelixYBW commented on issue #5751: URL: https://github.com/apache/incubator-gluten/issues/5751#issuecomment-2113434045 @gaoyangxiaozhu can you list all the parquet write parameters Spark supports and velox/arrow supports? Let's pass all supported params to Velox -- This is an automated

Re: [PR] [GLUTEN-5731][CORE] Fix the logic to calculate rss shuffle write time [incubator-gluten]

2024-05-15 Thread via GitHub
GlutenPerfBot commented on PR #5742: URL: https://github.com/apache/incubator-gluten/pull/5742#issuecomment-2113179749 = Performance report for TPCDS SF2000 with Velox backend, for reference only query

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2113017041 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2112981961 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2112908284 > Could you add cpp UT for sort-based shuffle? If it's possible to implement another `LocalRssClient` for sort-based shuffle and use it in native tests, we can reuse the

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1599300738 ## cpp/core/jni/JniWrapper.cc: ## @@ -148,6 +149,69 @@ class JavaInputStreamAdaptor final : public arrow::io::InputStream { bool closed_ = false; };

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1599300738 ## cpp/core/jni/JniWrapper.cc: ## @@ -148,6 +149,69 @@ class JavaInputStreamAdaptor final : public arrow::io::InputStream { bool closed_ = false; };

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1601882027 ## cpp/core/shuffle/HashPartitioner.cc: ## @@ -52,4 +52,33 @@ arrow::Status gluten::HashPartitioner::compute( return arrow::Status::OK(); }

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2112893168 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2112869952 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2112587570 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#issuecomment-2112586665 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Enable more types and partial merge mode for HLL [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5754: URL: https://github.com/apache/incubator-gluten/pull/5754#issuecomment-2112582372 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2112575602 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
supermem613 commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1601668026 ## shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala: ## @@ -1821,4 +1832,42 @@ object GlutenConfig { .internal()

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
supermem613 commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1601663164 ## gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-15 Thread via GitHub
supermem613 commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1601661282 ## gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache

Re: [PR] [GLUTEN-5696] Add preprojection support for ArrowEvalPythonExec [incubator-gluten]

2024-05-15 Thread via GitHub
yma11 commented on code in PR #5697: URL: https://github.com/apache/incubator-gluten/pull/5697#discussion_r1601549409 ## backends-velox/src/main/scala/org/apache/gluten/execution/python/ColumnarArrowEvalPythonExec.scala: ## @@ -299,15 +316,20 @@ case class

Re: [PR] [GLUTEN-5696] Add preprojection support for ArrowEvalPythonExec [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5697: URL: https://github.com/apache/incubator-gluten/pull/5697#issuecomment-2112389784 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5696] Add preprojection support for ArrowEvalPythonExec [incubator-gluten]

2024-05-15 Thread via GitHub
yma11 commented on code in PR #5697: URL: https://github.com/apache/incubator-gluten/pull/5697#discussion_r1601545797 ## backends-velox/src/main/scala/org/apache/gluten/execution/python/ColumnarArrowEvalPythonExec.scala: ## @@ -335,6 +357,65 @@ case class

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#issuecomment-2112214162 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Support "Complete" Hash aggregations [incubator-gluten]

2024-05-15 Thread via GitHub
zhouyuan commented on issue #1250: URL: https://github.com/apache/incubator-gluten/issues/1250#issuecomment-2112166470 @jackylee-ch please go head -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Support "Complete" Hash aggregations [incubator-gluten]

2024-05-15 Thread via GitHub
jackylee-ch commented on issue #1250: URL: https://github.com/apache/incubator-gluten/issues/1250#issuecomment-2112149714 @zhouyuan We meet some performance problem with this issue. Are you working on this issue now? If not, I'm glad to continue this work. -- This is an automated

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#issuecomment-2112147258 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2112129226 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#issuecomment-2112123946 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add a config to ignore fallback cost for scan [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5617: URL: https://github.com/apache/incubator-gluten/pull/5617#issuecomment-2112112941 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#issuecomment-2112072687 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2112056029 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2112028151 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [CH] Crash on exit with Poco exception [incubator-gluten]

2024-05-15 Thread via GitHub
taiyang-li commented on issue #5744: URL: https://github.com/apache/incubator-gluten/issues/5744#issuecomment-2111940597 ``` (gdb) bt #0 0x7fd07b010428 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x7fd07b01202a in abort () from /lib/x86_64-linux-gnu/libc.so.6

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2111932103 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5668][CH] Support mixed conditions in shuffle hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2111926090 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1601210868 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.cc: ## @@ -0,0 +1,349 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [I] Shuffle write time metric is wrong [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma closed issue #5731: Shuffle write time metric is wrong URL: https://github.com/apache/incubator-gluten/issues/5731 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [GLUTEN-5731][CORE] Fix the logic to calculate rss shuffle write time [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma merged PR #5742: URL: https://github.com/apache/incubator-gluten/pull/5742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [GLUTEN-5731][CORE] Fix the logic to calculate shuffle write time in RssPartitionWriter (#5742)

2024-05-15 Thread marong
This is an automated email from the ASF dual-hosted git repository. marong pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 2984bd740 [GLUTEN-5731][CORE] Fix the

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
ulysses-you commented on code in PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#discussion_r1601172081 ## gluten-core/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala: ## @@ -450,14 +450,6 @@ trait SparkPlanExecApi {

Re: [PR] [CORE] Add decimal precision tests [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5752: URL: https://github.com/apache/incubator-gluten/pull/5752#issuecomment-2111857742 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5668][CH] Support mixed inequal conditions in hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5735: URL: https://github.com/apache/incubator-gluten/pull/5735#issuecomment-2111827071 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2111798856 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Use the smaller table to build hashmap in shuffled hash join [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5750: URL: https://github.com/apache/incubator-gluten/pull/5750#issuecomment-2111792299 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Enable GlutenParqutRowIndexSuite for Spark 3.4/3.5 [incubator-gluten]

2024-05-15 Thread via GitHub
GlutenPerfBot commented on PR #5740: URL: https://github.com/apache/incubator-gluten/pull/5740#issuecomment-2111791559 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2111791910 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5691][CH] Enable merge on local disk first after insert into mergetree [incubator-gluten]

2024-05-15 Thread via GitHub
github-actions[bot] commented on PR #5692: URL: https://github.com/apache/incubator-gluten/pull/5692#issuecomment-2111737667 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5731][CORE] Fix the logic to calculate rss shuffle write time [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma commented on PR #5742: URL: https://github.com/apache/incubator-gluten/pull/5742#issuecomment-2111736168 @kerwin-zk Could you help to confirm this change? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [GLUTEN-5414] [VL] Support datasource v2 scan csv [incubator-gluten]

2024-05-15 Thread via GitHub
liujiayi771 commented on PR #5717: URL: https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2111724082 I have added some codes in the `release` method of `ArrowNativeMemoryPool` to check the peak memory. ```java @Override public void release() throws Exception {

Re: [PR] [GLUTEN-5414] [VL] Support datasource v2 scan csv [incubator-gluten]

2024-05-15 Thread via GitHub
zhztheplayer commented on PR #5717: URL: https://github.com/apache/incubator-gluten/pull/5717#issuecomment-2111708885 > I have found an issue: when reading large CSV files, for example, when a single CSV file in a table is 300M, the peak memory usage of arrow memory pool during

Re: [PR] [GLUTEN-5696] Add preprojection support for ArrowEvalPythonExec [incubator-gluten]

2024-05-15 Thread via GitHub
jinchengchenghh commented on code in PR #5697: URL: https://github.com/apache/incubator-gluten/pull/5697#discussion_r1601013958 ## backends-velox/src/main/scala/org/apache/gluten/execution/python/ColumnarArrowEvalPythonExec.scala: ## @@ -279,16 +282,30 @@ case class

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-15 Thread via GitHub
marin-ma commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1601021801 ## cpp/velox/shuffle/VeloxSortBasedShuffleWriter.h: ## @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

(incubator-gluten) branch main updated: [VL] Enable GlutenParqutRowIndexSuite for Spark 3.4/3.5 (#5740)

2024-05-15 Thread kejia
This is an automated email from the ASF dual-hosted git repository. kejia pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new a53ecc4a1 [VL] Enable

Re: [I] [VL] Spark 3.5 Unit Tests track [incubator-gluten]

2024-05-15 Thread via GitHub
JkSelf closed issue #5309: [VL] Spark 3.5 Unit Tests track URL: https://github.com/apache/incubator-gluten/issues/5309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

  1   2   >