(incubator-gluten) branch main updated: [GLUTEN-5331] Update the doc of build Gluten with Velox Backend (#5332)

2024-04-08 Thread yangzy
This is an automated email from the ASF dual-hosted git repository. yangzy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new de57cf5f6 [GLUTEN-5331] Update the doc of

Re: [PR] [GLUTEN-5331]Update the doc of Build Gluten with Velox Backend #5331 [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha merged PR #5332: URL: https://github.com/apache/incubator-gluten/pull/5332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [GLUTEN-5331]Update the doc of Build Gluten with Velox Backend [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha closed issue #5331: [GLUTEN-5331]Update the doc of Build Gluten with Velox Backend URL: https://github.com/apache/incubator-gluten/issues/5331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [VL] Add uniffle integration [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #3767: URL: https://github.com/apache/incubator-gluten/pull/3767#issuecomment-2044161671 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
taiyang-li commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556866284 ## cpp-ch/local-engine/Parser/AggregateFunctionParser.h: ## @@ -117,6 +117,12 @@ class AggregateFunctionParser DB::ActionsDAGPtr & actions_dag,

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
taiyang-li commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556865941 ## cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp: ## @@ -174,6 +174,28 @@ const DB::ActionsDAG::Node *

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
taiyang-li commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556863971 ## cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp: ## @@ -174,6 +174,28 @@ const DB::ActionsDAG::Node *

Re: [PR] [GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
marin-ma commented on PR #5326: URL: https://github.com/apache/incubator-gluten/pull/5326#issuecomment-2044118499 @yangzhg Thanks for your detailed explanation. There are configurations in Gluten to explicitly limit the batch size within 32k

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
taiyang-li commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556853664 ## cpp-ch/local-engine/Functions/FunctionIfInfinite.cpp: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + *

Re: [PR] [GLUTEN-5331]Update the doc of Build Gluten with Velox Backend #5331 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5332: URL: https://github.com/apache/incubator-gluten/pull/5332#issuecomment-2044114819 https://github.com/apache/incubator-gluten/issues/5331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
taiyang-li commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556847102 ## cpp-ch/local-engine/Parser/aggregate_function_parser/CommonAggregateFunctionParser.cpp: ## @@ -25,17 +25,17 @@

Re: [I] [VL] Potential Overflow Issue in `VeloxShuffleWriter` Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
FelixYBW commented on issue #5307: URL: https://github.com/apache/incubator-gluten/issues/5307#issuecomment-2044103018 So the root cause is that Velox may output rowvector with unlimited batch size. If so we need raise the issue to Velox. -- This is an automated message from the

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
lgbo-ustc commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556831739 ## backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHHashAggregateExecTransformer.scala: ## @@ -349,13 +349,18 @@ case class

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
lgbo-ustc commented on PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#issuecomment-2044094122 Is there any chance to modify the implementation in `CH` ? It will bring additional costs to add a`ifInfinite` at the end. Or could you give some performance test result to

Re: [PR] [CH] Fix diff of factorial function [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5330: URL: https://github.com/apache/incubator-gluten/pull/5330#issuecomment-2044093849 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CH] Fix diff of factorial function [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5330: URL: https://github.com/apache/incubator-gluten/pull/5330#issuecomment-2044093724 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [CH] Fix diff of factorial function [incubator-gluten]

2024-04-08 Thread via GitHub
exmy opened a new pull request, #5330: URL: https://github.com/apache/incubator-gluten/pull/5330 ## What changes were proposed in this pull request? * Fix diff of factorial function, `factorial(x)` should return null when x > 20 or x < 0 * enable spark ut about `ceil` and

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240409) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5329: URL: https://github.com/apache/incubator-gluten/pull/5329#issuecomment-2044092108 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240409) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5329: URL: https://github.com/apache/incubator-gluten/pull/5329#issuecomment-2044090488 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
lgbo-ustc commented on code in PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#discussion_r1556817411 ## backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHHashAggregateExecTransformer.scala: ## @@ -349,13 +349,18 @@ case class

Re: [PR] [GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
yangzhg commented on PR #5326: URL: https://github.com/apache/incubator-gluten/pull/5326#issuecomment-2044079819 > Thanks. Does this cause overflow in real cases? The uint16_t is used on purpose to benefit L2 cache, and is not supposed to cause the overflow with the 32k

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240409) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5329: URL: https://github.com/apache/incubator-gluten/pull/5329#issuecomment-2044078209 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240409) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5329: URL: https://github.com/apache/incubator-gluten/pull/5329#issuecomment-2044078031 https://github.com/apache/incubator-gluten/issues/1632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240409) [incubator-gluten]

2024-04-08 Thread via GitHub
kyligence-git opened a new pull request, #5329: URL: https://github.com/apache/incubator-gluten/pull/5329 Auto commit by gluten daily build, please check the build status and merge it if it's green. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [VL] Add uniffle integration [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #3767: URL: https://github.com/apache/incubator-gluten/pull/3767#issuecomment-2044075707 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-4990] [CH] fix data loss for dynamic patition [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #4991: URL: https://github.com/apache/incubator-gluten/pull/4991#issuecomment-2044061418 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5225][CH] Add mergetree index filter on driver [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5308: URL: https://github.com/apache/incubator-gluten/pull/5308#issuecomment-2044053857 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add uniffle integration [incubator-gluten]

2024-04-08 Thread via GitHub
PHILO-HE commented on code in PR #3767: URL: https://github.com/apache/incubator-gluten/pull/3767#discussion_r1556751289 ## cpp/core/jni/JniWrapper.cc: ## @@ -915,6 +915,23 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_vectorized_ShuffleWriterJniWrappe

Re: [PR] [VL] Add uniffle integration [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #3767: URL: https://github.com/apache/incubator-gluten/pull/3767#issuecomment-2044021508 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Add uniffle integration [incubator-gluten]

2024-04-08 Thread via GitHub
summaryzb commented on code in PR #3767: URL: https://github.com/apache/incubator-gluten/pull/3767#discussion_r1556740563 ## cpp/core/jni/JniWrapper.cc: ## @@ -915,6 +915,23 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_vectorized_ShuffleWriterJniWrappe

(incubator-gluten) branch main updated: [VL] Fix Alinux3 velox compilation exception (#5318)

2024-04-08 Thread yangzy
This is an automated email from the ASF dual-hosted git repository. yangzy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new c60b5d90e [VL] Fix Alinux3 velox

Re: [PR] [VL] Fix Alinux3 velox compilation exception [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha merged PR #5318: URL: https://github.com/apache/incubator-gluten/pull/5318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [GLUTEN-5324] Error message prompted during LocalLimitExec conversion (#5325)

2024-04-08 Thread yangzy
This is an automated email from the ASF dual-hosted git repository. yangzy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 570b93759 [GLUTEN-5324] Error message

Re: [I] [GLUTEN-5324]Error message prompted during LocalLimitExec conversion [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha closed issue #5324: [GLUTEN-5324]Error message prompted during LocalLimitExec conversion URL: https://github.com/apache/incubator-gluten/issues/5324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [GLUTEN-5324]Error message prompted during LocalLimitExec conversion … [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha merged PR #5325: URL: https://github.com/apache/incubator-gluten/pull/5325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [VL] Fix split function [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #4752: URL: https://github.com/apache/incubator-gluten/pull/4752#issuecomment-2043993669 This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. -- This is an automated

Re: [PR] [VL] Daily Update Velox Version (2024_04_09) [incubator-gluten]

2024-04-08 Thread via GitHub
zhztheplayer commented on PR #5328: URL: https://github.com/apache/incubator-gluten/pull/5328#issuecomment-2043990472 https://github.com/oap-project/velox/tree/2024_04_09 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
marin-ma commented on PR #5326: URL: https://github.com/apache/incubator-gluten/pull/5326#issuecomment-2043984990 Thanks. Does this cause overflow in real cases? The uint16_t is used on purpose to benefit L2 cache, and is not supposed to cause the overflow with the 32k limitation of the

Re: [PR] [DNM] Test window optimization in Velox [incubator-gluten]

2024-04-08 Thread via GitHub
zhztheplayer commented on PR #4929: URL: https://github.com/apache/incubator-gluten/pull/4929#issuecomment-2043984318 Committed Velox change in https://github.com/oap-project/velox/tree/2024_04_09 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [VL] Daily Update Velox Version (2024_04_09) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5328: URL: https://github.com/apache/incubator-gluten/pull/5328#issuecomment-2043972465 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043972017 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [VL] Daily Update Velox Version (2024_04_09) [incubator-gluten]

2024-04-08 Thread via GitHub
zhztheplayer opened a new pull request, #5328: URL: https://github.com/apache/incubator-gluten/pull/5328 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [VL] Daily Update Velox Version (2024_04_09) [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5328: URL: https://github.com/apache/incubator-gluten/pull/5328#issuecomment-2043968116 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [I] [CORE] Mark an overridden method with `@Override` annotation [incubator-gluten]

2024-04-08 Thread via GitHub
PHILO-HE closed issue #5316: [CORE] Mark an overridden method with `@Override` annotation URL: https://github.com/apache/incubator-gluten/issues/5316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [GLUTEN-5316][CORE] Add @Override annotation for some methods [incubator-gluten]

2024-04-08 Thread via GitHub
PHILO-HE merged PR #5317: URL: https://github.com/apache/incubator-gluten/pull/5317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [GLUTEN-5316][CORE] Add @Override annotation for some methods (#5317)

2024-04-08 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 0560ddd7b [GLUTEN-5316][CORE] Add

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043423122 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043099830 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043090951 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5215][CH]stable precision in statistical aggregation functions [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5311: URL: https://github.com/apache/incubator-gluten/pull/5311#issuecomment-2043066617 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043062480 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2043007059 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042896700 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
Surbhi-Vijay commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042803002 I was also trying this function. But array_sort introduces lambda function for comparator and looks like lambda functions are still falling back to Spark. Also check

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha commented on PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#issuecomment-2042751379 thank you for the improvements, this idea works for me, just few comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha commented on code in PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#discussion_r1555835707 ## gluten-core/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala: ## @@ -91,4 +91,6 @@ trait IteratorApi { numOutputRows: SQLMetric,

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
Yohahaha commented on code in PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#discussion_r1555825799 ## gluten-core/src/main/java/org/apache/gluten/substrait/rel/RawSplitInfo.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [GLUTEN-5309] Add UTs for Spark 3.5 [incubator-gluten]

2024-04-08 Thread via GitHub
zhouyuan merged PR #5310: URL: https://github.com/apache/incubator-gluten/pull/5310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042681282 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042593307 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] gluten-velox-bundle.jar NOT working with provided --conf spark.plugins=org.apache.gluten.GlutenPlugin \ with spark-shell tried io.glutenproject.GlutenPlugin instead (this exists) [incubator-gluten

2024-04-08 Thread via GitHub
surajchoubey opened a new issue, #5327: URL: https://github.com/apache/incubator-gluten/issues/5327 ### Problem description ``` $SPARK_HOME/bin/spark-shell \

Re: [PR] [GLUTEN-5316][CORE] Mark overridden methods with @Override annotation [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5317: URL: https://github.com/apache/incubator-gluten/pull/5317#issuecomment-2042434521 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5316][CORE] Mark overridden methods with @Override annotation [incubator-gluten]

2024-04-08 Thread via GitHub
dcoliversun commented on PR #5317: URL: https://github.com/apache/incubator-gluten/pull/5317#issuecomment-2042435065 cc @PHILO-HE would you please review this PR? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] [VL] support partial window in Spark 3.5 [incubator-gluten]

2024-04-08 Thread via GitHub
ayushi-agarwal commented on issue #4836: URL: https://github.com/apache/incubator-gluten/issues/4836#issuecomment-2042392888 @JkSelf Sure, I will try to raise a PR for the first task by end of Thursday. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [CORE][VL] RAS: Group expansion support [incubator-gluten]

2024-04-08 Thread via GitHub
zhztheplayer merged PR #5323: URL: https://github.com/apache/incubator-gluten/pull/5323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(incubator-gluten) branch main updated: [CORE][VL] RAS: Group expansion support (#5323)

2024-04-08 Thread hongze
This is an automated email from the ASF dual-hosted git repository. hongze pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new 19da3bb1c [CORE][VL] RAS: Group expansion

Re: [PR] [GLUTEN-5225][CH] Add mergetree index filter index on driver [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5308: URL: https://github.com/apache/incubator-gluten/pull/5308#issuecomment-2042343727 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5326: URL: https://github.com/apache/incubator-gluten/pull/5326#issuecomment-2042333905 https://github.com/apache/incubator-gluten/issues/5307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber [incubator-gluten]

2024-04-08 Thread via GitHub
yangzhg opened a new pull request, #5326: URL: https://github.com/apache/incubator-gluten/pull/5326 ## What changes were proposed in this pull request? Fixes: #5307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042323274 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2042316490 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5324]Error message prompted during LocalLimitExec conversion … [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5325: URL: https://github.com/apache/incubator-gluten/pull/5325#issuecomment-2042299017 https://github.com/apache/incubator-gluten/issues/5324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [GLUTEN-5324]Error message prompted during LocalLimitExec conversion … [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5325: URL: https://github.com/apache/incubator-gluten/pull/5325#issuecomment-2042299479 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2042283220 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE][VL] RAS: Group expansion support [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5323: URL: https://github.com/apache/incubator-gluten/pull/5323#issuecomment-2042253286 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

[PR] [CORE][VL] RAS: Group expansion support [incubator-gluten]

2024-04-08 Thread via GitHub
zhztheplayer opened a new pull request, #5323: URL: https://github.com/apache/incubator-gluten/pull/5323 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
liujiayi771 commented on code in PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#discussion_r1555475093 ## backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/IteratorApiImpl.scala: ## @@ -44,6 +42,7 @@ import

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
WangGuangxin commented on PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#issuecomment-2042210511 There are still some cases to fix, for example: 1. velox backend with iceberg format 2. clickhouse backend(which is not planed in this PR) But it works now on velox

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042197202 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] WIP [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5322: URL: https://github.com/apache/incubator-gluten/pull/5322#issuecomment-2042196729 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [GLUTEN-5125]Support CollectLimitExec #5125 [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5269: URL: https://github.com/apache/incubator-gluten/pull/5269#issuecomment-2042180507 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#issuecomment-2042170919 https://github.com/apache/incubator-gluten/issues/5320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5321: URL: https://github.com/apache/incubator-gluten/pull/5321#issuecomment-2042171507 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [GLUTEN-5320][VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
WangGuangxin opened a new pull request, #5321: URL: https://github.com/apache/incubator-gluten/pull/5321 ## What changes were proposed in this pull request? Currently, driver generate `GlutenPartition` based on spark's `FilePartitions`, and then convert to `LocalFilesNode` and

[I] [VL] Reduce driver memory footprint by postpone the creation and serialization of LocalFilesNode [incubator-gluten]

2024-04-08 Thread via GitHub
WangGuangxin opened a new issue, #5320: URL: https://github.com/apache/incubator-gluten/issues/5320 ### Description Currently, driver generate `GlutenPartition` based on spark's `FilePartitions`, and then convert to `LocalFilesNode` and serialized to byte array in pb format.

(incubator-gluten) branch main updated: [CELEBORN] Check `spark.shuffle.compress` first to decide whether to compress shuffle data (#5228)

2024-04-08 Thread philo
This is an automated email from the ASF dual-hosted git repository. philo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git The following commit(s) were added to refs/heads/main by this push: new e182d659b [CELEBORN] Check

Re: [PR] [CELEBORN] Check `spark.shuffle.compress` first to decide whether to compress shuffle data [incubator-gluten]

2024-04-08 Thread via GitHub
PHILO-HE merged PR #5228: URL: https://github.com/apache/incubator-gluten/pull/5228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [CELEBORN] Whether to compress shuffle data should be aligned with spark.shuffle.compress first [incubator-gluten]

2024-04-08 Thread via GitHub
kerwin-zk commented on PR #5228: URL: https://github.com/apache/incubator-gluten/pull/5228#issuecomment-2042134174 > Hi @kerwin-zk, I note velox backend CI is not triggered. Could you help fix it? You can just add "/**" for the three modules in the below link. Thanks!

Re: [PR] [GLUTEN-4990] [CH] fix data loss for dynamic patition [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #4991: URL: https://github.com/apache/incubator-gluten/pull/4991#issuecomment-2042130083 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [VL] Enable make_date Spark function [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5286: URL: https://github.com/apache/incubator-gluten/pull/5286#issuecomment-2042124088 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Remove useless GlutenFilePartition [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5319: URL: https://github.com/apache/incubator-gluten/pull/5319#issuecomment-2042079308 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]

2024-04-08 Thread via GitHub
ivoson commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2042062087 Hi @PHILO-HE I'd like to take filter (array filter), thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [CORE] Remove useless GlutenFilePartition [incubator-gluten]

2024-04-08 Thread via GitHub
WangGuangxin opened a new pull request, #5319: URL: https://github.com/apache/incubator-gluten/pull/5319 Remove useless code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [CORE] Remove useless GlutenFilePartition [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5319: URL: https://github.com/apache/incubator-gluten/pull/5319#issuecomment-2042058167 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [CORE] Remove useless GlutenFilePartition [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5319: URL: https://github.com/apache/incubator-gluten/pull/5319#issuecomment-2042057755 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [DNM] Test window optimization in Velox [incubator-gluten]

2024-04-08 Thread via GitHub
JkSelf commented on PR #4929: URL: https://github.com/apache/incubator-gluten/pull/4929#issuecomment-2042025917 @zhztheplayer The CI is passed. Please help to pick this [commit](https://github.com/oap-project/velox/commit/d90f777ed0d6d0148173e67e707b230dc5b9ccb1) into oap velox in next

Re: [PR] [VL] Fix alinux3 velox compilation exception [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5318: URL: https://github.com/apache/incubator-gluten/pull/5318#issuecomment-2042002763 Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues?

Re: [PR] [GLUTEN-5316][CORE] Mark overridden methods with @Override annotation [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5317: URL: https://github.com/apache/incubator-gluten/pull/5317#issuecomment-2042001567 Run Gluten Clickhouse CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [VL] Fix alinux3 velox compilation exception [incubator-gluten]

2024-04-08 Thread via GitHub
kerwin-zk opened a new pull request, #5318: URL: https://github.com/apache/incubator-gluten/pull/5318 ## What changes were proposed in this pull request? When building Velox, use the Alinux3 built-in gcc and python3 ## How was this patch tested? CI --

Re: [PR] [GLUTEN-5316][CORE] Mark overridden methods with @Override annotation [incubator-gluten]

2024-04-08 Thread via GitHub
github-actions[bot] commented on PR #5317: URL: https://github.com/apache/incubator-gluten/pull/5317#issuecomment-2042001255 https://github.com/apache/incubator-gluten/issues/5316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

  1   2   >