Re: [PR] [VL] Fallback Window operator if there is a mismatch between the functions's input and output types. [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5554: URL: https://github.com/apache/incubator-gluten/pull/5554#issuecomment-2103920800 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#issuecomment-2103919865 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [GLUTEN-4039][VL] Add flatten function support [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5551: URL: https://github.com/apache/incubator-gluten/pull/5551#issuecomment-2103914699 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5414] [VL] Fix arrow native memory pool track [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5683: URL: https://github.com/apache/incubator-gluten/pull/5683#issuecomment-2103890360 https://github.com/apache/incubator-gluten/issues/5414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-5414] [VL] Fix arrow native memory pool track [incubator-gluten]

2024-05-09 Thread via GitHub
jinchengchenghh opened a new pull request, #5683: URL: https://github.com/apache/incubator-gluten/pull/5683 Before this patch, will throw exception ```

Re: [PR] [VL] Use the default pip3.6 from the alinux3 in the build of velox [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5676: URL: https://github.com/apache/incubator-gluten/pull/5676#issuecomment-2103870044 = Performance report for TPCH SF2000 with Velox backend, for reference only query

(incubator-gluten) branch main updated: [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request (#5674)

2024-05-09 Thread yangzy
This is an automated email from the ASF dual-hosted git repository. yangzy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 6fba94991 [GLUTEN-5673][VL] Fix

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha merged PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [VL] Unexpected memory issue found in CI [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha closed issue #5673: [VL] Unexpected memory issue found in CI URL: https://github.com/apache/incubator-gluten/issues/5673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [VL] Result mismatch issues Tracker [incubator-gluten]

2024-05-09 Thread via GitHub
zjuwangg commented on issue #4652: URL: https://github.com/apache/incubator-gluten/issues/4652#issuecomment-2103850120 https://github.com/apache/incubator-gluten/issues/5682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [GLUTEN-5682][VL]Fix isNull and isNotNull coexsit is filter causing result not correct [incubator-gluten]

2024-05-09 Thread via GitHub
zjuwangg commented on PR #5670: URL: https://github.com/apache/incubator-gluten/pull/5670#issuecomment-2103845903 velox related change has been merged via https://github.com/facebookincubator/velox/pull/9718 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
kerwin-zk commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1596242830 ## cpp/velox/shuffle/VeloxShuffleWriter.cc: ## @@ -376,8 +386,145 @@ arrow::Status VeloxShuffleWriter::partitioningAndDoSplit(facebook::velox::RowVec

Re: [PR] [GLUTEN-5682][VL]Fix isNull and isNotNull coexsit is filter causing result not correct [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5670: URL: https://github.com/apache/incubator-gluten/pull/5670#issuecomment-2103843271 https://github.com/apache/incubator-gluten/issues/5682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[I] [VL]isNull and isNotNull both exist in filter condition, the result is not correct [incubator-gluten]

2024-05-09 Thread via GitHub
zjuwangg opened a new issue, #5682: URL: https://github.com/apache/incubator-gluten/issues/5682 ### Backend VL (Velox) ### Bug description run sql `select l_orderkey from lineitem where l_comment is null and l_comment is not null` in velox-module ut, we can reproduce

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103833029 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
zhouyuan commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1596226525 ## cpp/velox/shuffle/VeloxShuffleWriter.cc: ## @@ -376,8 +386,145 @@ arrow::Status VeloxShuffleWriter::partitioningAndDoSplit(facebook::velox::RowVec

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103824355 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Use the default pip3.6 from the alinux3 in the build of velox [incubator-gluten]

2024-05-09 Thread via GitHub
zhouyuan merged PR #5676: URL: https://github.com/apache/incubator-gluten/pull/5676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [VL] Use the default pip3.6 from the alinux3 in the build of velox (#5676)

2024-05-09 Thread yuanzhou
This is an automated email from the ASF dual-hosted git repository. yuanzhou pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new a7e7f6578 [VL] Use the default pip3.6

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596222302 ## backends-velox/src/main/scala/org/apache/gluten/expression/aggregate/VeloxCollect.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software

Re: [PR] [WIP][VL]isNull and isNotNull coexsit causing result not correct [incubator-gluten]

2024-05-09 Thread via GitHub
zjuwangg commented on PR #5670: URL: https://github.com/apache/incubator-gluten/pull/5670#issuecomment-2103802042 > Is it only a Spark UT issue? or observed in some SQL? It's not just a Spark UT issue. We encounter this bug causing result not mismatch. We can reproduce this bug

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596208826 ## gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDataFrameAggregateSuite.scala: ## @@ -279,4 +283,55 @@ class GlutenDataFrameAggregateSuite

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596212647 ## backends-velox/src/main/scala/org/apache/gluten/expression/aggregate/VeloxCollect.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596211736 ## backends-velox/src/main/scala/org/apache/gluten/expression/aggregate/VeloxCollect.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software

Re: [PR] [VL] Daily Update Velox Version (2024_05_10) [incubator-gluten]

2024-05-09 Thread via GitHub
zhouyuan commented on PR #5678: URL: https://github.com/apache/incubator-gluten/pull/5678#issuecomment-2103797710 may related: https://github.com/facebookincubator/velox/pull/9734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103794412 > thank you @zhztheplayer , can you also provide the npe case to help reproduce ? It seems this pr did not contain that case ? NPE can be repeated from current main

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596209504 ## backends-velox/src/test/scala/org/apache/gluten/execution/FallbackSuite.scala: ## @@ -106,15 +112,38 @@ class FallbackSuite extends

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596208826 ## gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDataFrameAggregateSuite.scala: ## @@ -279,4 +283,55 @@ class GlutenDataFrameAggregateSuite

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1596207765 ## gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache

Re: [I] Correctness issue converting literal array with empty array or map as head element [incubator-gluten]

2024-05-09 Thread via GitHub
PHILO-HE closed issue #5662: Correctness issue converting literal array with empty array or map as head element URL: https://github.com/apache/incubator-gluten/issues/5662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

(incubator-gluten) branch main updated: [GLUTEN-5662][VL] Fix literal array conversion with nested empty array/map ahead of non-empty (#5663)

2024-05-09 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new f6e882335 [GLUTEN-5662][VL] Fix literal

Re: [PR] [GLUTEN-5662][VL] Fix literal array conversion with nested empty array/map ahead of non-empty [incubator-gluten]

2024-05-09 Thread via GitHub
PHILO-HE merged PR #5663: URL: https://github.com/apache/incubator-gluten/pull/5663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [DNM][GLUTEN-5548][VL] test dynamic setting [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5506: URL: https://github.com/apache/incubator-gluten/pull/5506#issuecomment-2103788167 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1596205549 ## gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1596202734 ## gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache

Re: [PR] [VL] Daily Update Velox Version (2024_05_10) [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5678: URL: https://github.com/apache/incubator-gluten/pull/5678#issuecomment-2103776291 @xiaoxmeng We got another regression by Velox's change on arbitrator ``` 24/05/10 02:07:04 ERROR ManagedReservationListener: Error reserving memory from target

Re: [PR] [GLUTEN-5438] feat: Dynamically sizing off-heap memory [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on code in PR #5439: URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1596197989 ## shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala: ## @@ -1821,4 +1832,42 @@ object GlutenConfig { .internal()

(incubator-gluten) branch main updated: [DOC]add Gluten logo (#5680)

2024-05-09 Thread weitingchen
This is an automated email from the ASF dual-hosted git repository. weitingchen pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 6c1a7478c [DOC]add Gluten logo

Re: [PR] [DOC]add Gluten logo [incubator-gluten]

2024-05-09 Thread via GitHub
weiting-chen merged PR #5680: URL: https://github.com/apache/incubator-gluten/pull/5680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596188187 ## backends-velox/src/main/scala/org/apache/gluten/expression/aggregate/VeloxCollect.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software

Re: [PR] [VL] Support InsertIntoHadoopFsRelationCommand for csv format [incubator-gluten]

2024-05-09 Thread via GitHub
liujiayi771 commented on PR #5681: URL: https://github.com/apache/incubator-gluten/pull/5681#issuecomment-2103751578 cc @jinchengchenghh @zhouyuan, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [VL] Support InsertIntoHadoopFsRelationCommand for csv format [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5681: URL: https://github.com/apache/incubator-gluten/pull/5681#issuecomment-2103751338 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Support InsertIntoHadoopFsRelationCommand for csv format [incubator-gluten]

2024-05-09 Thread via GitHub
liujiayi771 opened a new pull request, #5681: URL: https://github.com/apache/incubator-gluten/pull/5681 ## What changes were proposed in this pull request? `ArrowConvertorRule` needs to be adapted to `InsertIntoHadoopFsRelationCommand` in order to support `insert into table select *

Re: [PR] [DNM][GLUTEN-5548][VL] test dynamic setting [incubator-gluten]

2024-05-09 Thread via GitHub
zhli1142015 commented on PR #5506: URL: https://github.com/apache/incubator-gluten/pull/5506#issuecomment-2103749798 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
PHILO-HE commented on code in PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#discussion_r1596181272 ## backends-velox/src/test/scala/org/apache/gluten/execution/FallbackSuite.scala: ## @@ -106,15 +112,38 @@ class FallbackSuite extends

Re: [PR] [DOC]add Gluten logo [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5680: URL: https://github.com/apache/incubator-gluten/pull/5680#issuecomment-2103747633 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [DOC]add Gluten logo [incubator-gluten]

2024-05-09 Thread via GitHub
weiting-chen opened a new pull request, #5680: URL: https://github.com/apache/incubator-gluten/pull/5680 ## What changes were proposed in this pull request? Add Gluten logo in github ## How was this patch tested? N/A -- This is an automated message from the

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
liujiayi771 commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103729011 @zhztheplayer Do you think this approach is better, or is it better for us to convert it to binary in the project during row extraction? -- This is an automated message

Re: [PR] [WIP][DNM][VL] Support Row Index Metadata Column [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5351: URL: https://github.com/apache/incubator-gluten/pull/5351#issuecomment-2103726944 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha commented on code in PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#discussion_r1596173675 ## cpp/velox/memory/VeloxMemoryManager.cc: ## @@ -94,20 +95,22 @@ class ListenableArbitrator : public velox::memory::MemoryArbitrator { // Since

Re: [PR] [VL] Fix async io coredump [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5657: URL: https://github.com/apache/incubator-gluten/pull/5657#discussion_r1596171031 ## backends-velox/src/main/java/org/apache/gluten/backend/VeloxBackendJniWrapper.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software

Re: [PR] [GLUTEN-5662][VL] Fix issue converting literal array with empty array/map as head to velox expr [incubator-gluten]

2024-05-09 Thread via GitHub
ivoson commented on PR #5663: URL: https://github.com/apache/incubator-gluten/pull/5663#issuecomment-2103716237 cc @rui-mo @PHILO-HE can you please take a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#issuecomment-2103715824 > > Thanks for helping investigate this @Yohahaha ! > > What I am curious about is that, since we are on single-threaded execution, each `ListenableArbitrator` should be

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on code in PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#discussion_r1596167623 ## cpp/velox/memory/VeloxMemoryManager.cc: ## @@ -94,20 +95,22 @@ class ListenableArbitrator : public velox::memory::MemoryArbitrator { // Since

[I] [VL] random issue on GHA: terminate called without an active exception [incubator-gluten]

2024-05-09 Thread via GitHub
zhouyuan opened a new issue, #5679: URL: https://github.com/apache/incubator-gluten/issues/5679 ### Backend VL (Velox) ### Bug description ``` VeloxOrcDataTypeValidationSuite: E20240509 09:27:33.494535 61272 Exceptions.h:69] Line:

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha commented on PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#issuecomment-2103710252 > Thanks for helping investigate this @Yohahaha ! > > What I am curious about is that, since we are on single-threaded execution, each `ListenableArbitrator` should be

Re: [PR] [GLUTEN-4896] Upgrade Spark33 version to Spark3.3.4 [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] closed pull request #4897: [GLUTEN-4896] Upgrade Spark33 version to Spark3.3.4 URL: https://github.com/apache/incubator-gluten/pull/4897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [GLUTEN-4896] Upgrade Spark33 version to Spark3.3.4 [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #4897: URL: https://github.com/apache/incubator-gluten/pull/4897#issuecomment-2103701028 This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks. -- This is an automated

Re: [PR] [GLUTEN-5414] [VL] Move ArrowCsvFileFormat class match to its module and enable arrow memory pool track [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5667: URL: https://github.com/apache/incubator-gluten/pull/5667#issuecomment-2103698607 > https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/utils/PlanUtil.scala#L53 > > If I move `ArrowFileSourceScanExec`

Re: [PR] [VL] Fix async io coredump [incubator-gluten]

2024-05-09 Thread via GitHub
marin-ma commented on code in PR #5657: URL: https://github.com/apache/incubator-gluten/pull/5657#discussion_r1596152758 ## backends-velox/src/main/java/org/apache/gluten/backend/VeloxBackendJniWrapper.java: ## @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#issuecomment-2103686872 Thanks for helping investigate this @Yohahaha ! What I am curious about is that, since we are on single-threaded execution, each `ListenableArbitrator` should be

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103681867 @PHILO-HE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [GLUTEN-5414] [VL] Move ArrowCsvFileFormat class match to its module [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5667: URL: https://github.com/apache/incubator-gluten/pull/5667#issuecomment-2103680597 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2103668516 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Generate hdfs-client.xml for libhdfs [incubator-gluten]

2024-05-09 Thread via GitHub
ulysses-you commented on PR #5661: URL: https://github.com/apache/incubator-gluten/pull/5661#issuecomment-2103667265 @PHILO-HE , no, we do not need it. libhdfs would read hdfs-client.xml by default in working directory and that's where this pr generates hdfs-client.xml. -- This is an

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2103667167 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [VL] Spark 3.5 Unit Tests track [incubator-gluten]

2024-05-09 Thread via GitHub
yma11 commented on issue #5309: URL: https://github.com/apache/incubator-gluten/issues/5309#issuecomment-2103659349 Hi @gaoyangxiaozhu, the left failed UTs for Spark3.5 are all about row index metadata and I see your support

Re: [PR] [VL] Daily Update Velox Version (2024_05_10) [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5678: URL: https://github.com/apache/incubator-gluten/pull/5678#issuecomment-2103646212 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Daily Update Velox Version (2024_05_10) [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot opened a new pull request, #5678: URL: https://github.com/apache/incubator-gluten/pull/5678 Upstream Velox's New Commits: ```txt 7d76b1d9e by Jialiang Tan, Propagate additional dwrf writer options (9752) 49c3ebb20 by gaoyangxiaozhu, Row index metadata column

Re: [PR] [VL] Fix NullPointerException when collect_list / collect_set are partially fallen back [incubator-gluten]

2024-05-09 Thread via GitHub
zhztheplayer commented on PR #5655: URL: https://github.com/apache/incubator-gluten/pull/5655#issuecomment-2103637430 @liujiayi771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [GLUTEN-5414] [VL] Move ArrowCsvFileFormat class match to its module [incubator-gluten]

2024-05-09 Thread via GitHub
jinchengchenghh commented on PR #5667: URL: https://github.com/apache/incubator-gluten/pull/5667#issuecomment-2103629940 And the genExtendedColumnarValidationRules may take effect after `add c2r r2c`, so the test failed -- This is an automated message from the Apache Git Service. To

Re: [PR] [GLUTEN-5620][CORE] Remove check_overflow and refactor code [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5654: URL: https://github.com/apache/incubator-gluten/pull/5654#issuecomment-2103627644 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [DNM][GLUTEN-5548][VL] test dynamic setting [incubator-gluten]

2024-05-09 Thread via GitHub
zhli1142015 commented on PR #5506: URL: https://github.com/apache/incubator-gluten/pull/5506#issuecomment-2103620812 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240510) [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5677: URL: https://github.com/apache/incubator-gluten/pull/5677#issuecomment-2103577310 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240510) [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5677: URL: https://github.com/apache/incubator-gluten/pull/5677#issuecomment-2103577135 https://github.com/apache/incubator-gluten/issues/1632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240510) [incubator-gluten]

2024-05-09 Thread via GitHub
kyligence-git opened a new pull request, #5677: URL: https://github.com/apache/incubator-gluten/pull/5677 Auto commit by gluten daily build, please check the build status and merge it if it's green. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [VL] Daily Update Velox Version (2024_05_09) [incubator-gluten]

2024-05-09 Thread via GitHub
FelixYBW merged PR #5664: URL: https://github.com/apache/incubator-gluten/pull/5664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [VL] Daily Update Velox Version (2024_05_09) (#5664)

2024-05-09 Thread felixybw
This is an automated email from the ASF dual-hosted git repository. felixybw pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 24d949e61 [VL] Daily Update Velox

Re: [PR] [WIP][VL]isNull and isNotNull coexsit causing result not correct [incubator-gluten]

2024-05-09 Thread via GitHub
FelixYBW commented on PR #5670: URL: https://github.com/apache/incubator-gluten/pull/5670#issuecomment-2103320140 Is it only a Spark UT issue? or observed in some SQL? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
FelixYBW commented on code in PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#discussion_r1595874723 ## cpp/velox/shuffle/VeloxShuffleWriter.h: ## @@ -132,6 +132,8 @@ class VeloxShuffleWriter final : public ShuffleWriter { arrow::Status

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
FelixYBW commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2103267810 Thank you for the PR! Waiting for long time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] [VL] Unexpected memory issue found in CI [incubator-gluten]

2024-05-09 Thread via GitHub
FelixYBW commented on issue #5673: URL: https://github.com/apache/incubator-gluten/issues/5673#issuecomment-2103266340 @zhztheplayer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [VL] Defer debug log generation [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5672: URL: https://github.com/apache/incubator-gluten/pull/5672#issuecomment-2103260981 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2103029195 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Use the default pip3.6 from the alinux3 in the build of velox [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5676: URL: https://github.com/apache/incubator-gluten/pull/5676#issuecomment-2102960452 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [VL] Use the default pip3.6 from the alinux3 in the build of velox [incubator-gluten]

2024-05-09 Thread via GitHub
kerwin-zk opened a new pull request, #5676: URL: https://github.com/apache/incubator-gluten/pull/5676 ## What changes were proposed in this pull request? Use the default pip3.6 from the alinux3 in the build of velox ## How was this patch tested? CI -- This is

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2102892979 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Defer debug log generation [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5672: URL: https://github.com/apache/incubator-gluten/pull/5672#issuecomment-2102672700 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [I] [VL] Add SampleExec operator support [incubator-gluten]

2024-05-09 Thread via GitHub
gaoyangxiaozhu commented on issue #5315: URL: https://github.com/apache/incubator-gluten/issues/5315#issuecomment-2102615417 Hey @zhouyuan / @FelixYBW / @rui-mo , I'm starting to work on this feature, and here's a quick draft to share my ideas and one basic but crucial question need

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2102594110 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5673][VL] Fix arbitrator grow logic when exist concurrent memory request [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5674: URL: https://github.com/apache/incubator-gluten/pull/5674#issuecomment-2102589290 https://github.com/apache/incubator-gluten/issues/5673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [VL] Defer debug log generation [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha merged PR #5672: URL: https://github.com/apache/incubator-gluten/pull/5672 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [VL] Defer debug log generation (#5672)

2024-05-09 Thread yangzy
This is an automated email from the ASF dual-hosted git repository. yangzy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 949650969 [VL] Defer debug log generation

Re: [PR] [VL] Daily Update Velox Version (2024_05_09) [incubator-gluten]

2024-05-09 Thread via GitHub
Yohahaha commented on PR #5664: URL: https://github.com/apache/incubator-gluten/pull/5664#issuecomment-2102583144 After https://github.com/apache/incubator-gluten/pull/5456, current gluten arbitrator will encounter concurrent memory request issue, we can temporary disable

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2102577681 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2102577287 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][VL] Support celeborn sort based shuffle [incubator-gluten]

2024-05-09 Thread via GitHub
github-actions[bot] commented on PR #5675: URL: https://github.com/apache/incubator-gluten/pull/5675#issuecomment-2102576899 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]

2024-05-09 Thread via GitHub
ivoson commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2102428935 I'd like to implement `array_append` and `array_insert` for spark 3.4+ -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [GLUTEN-5620][CORE] Remove check_overflow and refactor code [incubator-gluten]

2024-05-09 Thread via GitHub
GlutenPerfBot commented on PR #5654: URL: https://github.com/apache/incubator-gluten/pull/5654#issuecomment-2102398755 = Performance report for TPCH SF2000 with Velox backend, for reference only query

Re: [PR] [GLUTEN-5620][CORE] Remove check_overflow and refactor code [incubator-gluten]

2024-05-09 Thread via GitHub
jinchengchenghh commented on PR #5654: URL: https://github.com/apache/incubator-gluten/pull/5654#issuecomment-2102309196 /Benchmark Velox -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

  1   2   >