Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
planga82 commented on PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#issuecomment-2155834873 Thanks!! I am a rust beginner, I appreciate any comments! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] feat: Add HashJoin support for BuildRight [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #437: URL: https://github.com/apache/datafusion-comet/pull/437#issuecomment-2155825768 > Hmm, it's weird. I've updated the plan stability results. However, `CometTPCDSV2_7_PlanStabilitySuite` still fails on q5a for Spark 4.0 only in CI. And I cannot reproduce it local

Re: [PR] Debug print as many fields as possible for `SessionState` [datafusion]

2024-06-07 Thread via GitHub
lewiszlw commented on code in PR #10818: URL: https://github.com/apache/datafusion/pull/10818#discussion_r1631848699 ## datafusion/core/src/execution/session_state.rs: ## @@ -141,7 +141,13 @@ impl Debug for SessionState { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> st

Re: [PR] [functions]support current_timestamp [datafusion]

2024-06-07 Thread via GitHub
github-actions[bot] commented on PR #6873: URL: https://github.com/apache/datafusion/pull/6873#issuecomment-2155751219 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or th

[PR] Remove builtin Sum [datafusion]

2024-06-07 Thread via GitHub
jayzhan211 opened a new pull request, #10831: URL: https://github.com/apache/datafusion/pull/10831 ## Which issue does this PR close? Part of #10731 . ## Rationale for this change ## What changes are included in this PR? Since we need stub f

Re: [PR] feat: Add HashJoin support for BuildRight [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #437: URL: https://github.com/apache/datafusion-comet/pull/437#issuecomment-2155734913 Hmm, it's weird. I've updated the plan stability results. However, `CometTPCDSV2_7_PlanStabilitySuite` still fails on q5a for Spark 4.0 only in CI. And I cannot reproduce it locally

Re: [PR] chore: Create initial release process scripts for official ASF source release [datafusion-comet]

2024-06-07 Thread via GitHub
parthchandra commented on PR #429: URL: https://github.com/apache/datafusion-comet/pull/429#issuecomment-2155732503 > @parthchandra @advancedxy I think I have addressed all of the feedback now LGTM -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Move sum test to slt or optimizer_integration [datafusion]

2024-06-07 Thread via GitHub
jayzhan211 closed pull request #10807: Move sum test to slt or optimizer_integration URL: https://github.com/apache/datafusion/pull/10807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Remove expr_fn::sum and replace them with function stub [datafusion]

2024-06-07 Thread via GitHub
jayzhan211 merged PR #10816: URL: https://github.com/apache/datafusion/pull/10816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [I] shuffle write fails with out of range slice [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on issue #540: URL: https://github.com/apache/datafusion-comet/issues/540#issuecomment-2155702600 Btw, it doesn't fail in shuffle writer. From the stack trace, it looks like when the writer to pull next batch from its upstream, the error happened. -- This is an au

Re: [I] shuffle write fails with out of range slice [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on issue #540: URL: https://github.com/apache/datafusion-comet/issues/540#issuecomment-2155702042 @andygrove Is there any reproducible example? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Add spilling in SortMergeJoin [datafusion]

2024-06-07 Thread via GitHub
comphead commented on issue #9359: URL: https://github.com/apache/datafusion/issues/9359#issuecomment-2155697663 the first use case is to try spilling for buffered data, as the buffered data comes in full size and eats the memory. This is proved by query below ``` > select * from (s

[I] shuffle write fails with out of range slice [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove opened a new issue, #540: URL: https://github.com/apache/datafusion-comet/issues/540 ### Describe the bug I just saw this when running benchmarks with latest from main and with xxhash64 disabled. ``` Caused by: org.apache.comet.CometNativeException: General executi

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631706529 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631705684 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove merged PR #539: URL: https://github.com/apache/datafusion-comet/pull/539 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove merged PR #520: URL: https://github.com/apache/datafusion-comet/pull/520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631685596 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #539: URL: https://github.com/apache/datafusion-comet/pull/539#discussion_r1631668088 ## spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala: ## @@ -27,7 +28,19 @@ trait CometExprShim { /** * Returns a tuple of express

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631661598 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631660778 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] Debug print as many fields as possible for `SessionState` [datafusion]

2024-06-07 Thread via GitHub
comphead commented on code in PR #10818: URL: https://github.com/apache/datafusion/pull/10818#discussion_r1631653847 ## datafusion/core/src/execution/session_state.rs: ## @@ -141,7 +141,13 @@ impl Debug for SessionState { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> st

Re: [I] DataFusion 37.0.0 upgrade [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove closed issue #222: DataFusion 37.0.0 upgrade URL: https://github.com/apache/datafusion-comet/issues/222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [I] DataFusion 37.0.0 upgrade [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on issue #222: URL: https://github.com/apache/datafusion-comet/issues/222#issuecomment-2155464917 Comet is now using 39.0.0-rc1 (and will upgrade to 39.0.0 when it is available, probably early next week) so I think we can close this issue now -- This is an automated m

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631622607 ## docs/source/user-guide/expressions.md: ## @@ -19,99 +19,175 @@ # Supported Spark Expressions -The following Spark expressions are currently available:

Re: [PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #539: URL: https://github.com/apache/datafusion-comet/pull/539#discussion_r1631611199 ## spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala: ## @@ -27,7 +28,19 @@ trait CometExprShim { /** * Returns a tuple of express

Re: [PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #539: URL: https://github.com/apache/datafusion-comet/pull/539#discussion_r1631608977 ## spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala: ## @@ -27,7 +28,19 @@ trait CometExprShim { /** * Returns a tuple of

[PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove opened a new pull request, #539: URL: https://github.com/apache/datafusion-comet/pull/539 ## Which issue does this PR close? N/A ## Rationale for this change Code cleanup. ## What changes are included in this PR? - Add new `

Re: [PR] chore: Add CometEvalMode enum to replace string literals [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on PR #539: URL: https://github.com/apache/datafusion-comet/pull/539#issuecomment-2155378687 These changes were originally part of https://github.com/apache/datafusion-comet/pull/509 but it looks like we won't be merging that one for some time -- This is an automated

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631596537 ## docs/source/user-guide/expressions.md: ## @@ -19,99 +19,175 @@ # Supported Spark Expressions -The following Spark expressions are currently avai

Re: [PR] build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2155372197 Thanks @advancedxy @kazuyukitanimura @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove merged PR #403: URL: https://github.com/apache/datafusion-comet/pull/403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [I] Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove closed issue #248: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released URL: https://github.com/apache/datafusion-comet/issues/248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#discussion_r1631588805 ## core/src/execution/datafusion/expressions/regexp.rs: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] bug: Add tests for Scalar and Inverval values for UnaryMinus [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #538: URL: https://github.com/apache/datafusion-comet/pull/538#discussion_r1631572354 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1583,66 +1585,61 @@ class CometExpressionSuite extends CometTestBase with A

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
kazuyukitanimura commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631556263 ## core/src/execution/datafusion/expressions/scalar_funcs.rs: ## @@ -130,6 +133,10 @@ pub fn create_comet_physical_fun( let func = Arc::new

Re: [PR] build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2155312001 cc @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2155311450 All tests are passed! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631548388 ## docs/source/user-guide/operators.md: ## @@ -19,15 +19,18 @@ # Supported Spark Operators -The following Spark operators are currently available: +The follo

Re: [I] Add ANSI support for Add [datafusion-comet]

2024-06-07 Thread via GitHub
planga82 commented on issue #536: URL: https://github.com/apache/datafusion-comet/issues/536#issuecomment-2155294788 I want to try this one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Debug print as many fields as possible for `SessionState` [datafusion]

2024-06-07 Thread via GitHub
alamb commented on code in PR #10818: URL: https://github.com/apache/datafusion/pull/10818#discussion_r1631537285 ## datafusion/core/src/execution/session_state.rs: ## @@ -141,7 +141,13 @@ impl Debug for SessionState { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::

Re: [I] Supported python versions (drop 3.7 and add 3.12?) [datafusion-python]

2024-06-07 Thread via GitHub
Michael-J-Ward commented on issue #726: URL: https://github.com/apache/datafusion-python/issues/726#issuecomment-2155286703 Additional context: Python 3.7 is officially in "end-of-life" status. Python 3.8 will reach EOL in 2024-10. https://devguide.python.org/versions/#python-

Re: [I] Add ability to receive an iterator over the inputs of a LogicalPlan instead of a Vec. [datafusion]

2024-06-07 Thread via GitHub
alamb commented on issue #10808: URL: https://github.com/apache/datafusion/issues/10808#issuecomment-2155283319 > Oh yeah, you are right, I don't know why I didn't think about that. I will try it anyway, maybe I will come up with a different solution. Thanks Peter, for letting me know!

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631522983 ## docs/source/user-guide/operators.md: ## @@ -19,15 +19,18 @@ # Supported Spark Operators -The following Spark operators are currently available: +The fo

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631518775 ## core/src/execution/datafusion/expressions/scalar_funcs.rs: ## @@ -130,6 +133,10 @@ pub fn create_comet_physical_fun( let func = Arc::new(spark_

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631517608 ## core/src/execution/datafusion/expressions/scalar_funcs.rs: ## @@ -130,6 +133,10 @@ pub fn create_comet_physical_fun( let func = Arc::new(spark_

Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#discussion_r1631514068 ## core/src/execution/datafusion/expressions/abs.rs: ## @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#discussion_r1631513639 ## core/src/execution/datafusion/expressions/abs.rs: ## @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] build(deps): bump actions/cache from 3 to 4 [datafusion-python]

2024-06-07 Thread via GitHub
Michael-J-Ward commented on PR #566: URL: https://github.com/apache/datafusion-python/pull/566#issuecomment-2155257613 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] build(deps): bump actions/cache from 3 to 4 [datafusion-python]

2024-06-07 Thread via GitHub
dependabot[bot] commented on PR #566: URL: https://github.com/apache/datafusion-python/pull/566#issuecomment-2155257658 Sorry, only users with push access can use that command. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
vaibhawvipul commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631510549 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -980,6 +980,29 @@ class CometExpressionSuite extends CometTestBase with Adaptive

[I] Supported python versions (drop 3.7 and add 3.12?) [datafusion-python]

2024-06-07 Thread via GitHub
Michael-J-Ward opened a new issue, #726: URL: https://github.com/apache/datafusion-python/issues/726 In upgrading maturin in #725, I noticed that nixos no longer includes python 3.7 in its packages. Hence, I was able to upgrade requirements.txt for 3.10 and 3.11 but left python 3.7 unchange

Re: [I] Example for building an external index for parquet files [datafusion]

2024-06-07 Thread via GitHub
adriangb commented on issue #10546: URL: https://github.com/apache/datafusion/issues/10546#issuecomment-2155239748 https://github.com/datafusion-contrib/datafusion-async-parquet-index/pull/1 πŸ˜ƒ -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#discussion_r1631494622 ## core/src/execution/datafusion/planner.rs: ## @@ -499,7 +497,12 @@ impl PhysicalPlanner { let child = self.create_expr(expr.child.as_ref().u

Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#discussion_r1631494622 ## core/src/execution/datafusion/planner.rs: ## @@ -499,7 +497,12 @@ impl PhysicalPlanner { let child = self.create_expr(expr.child.as_ref().u

Re: [PR] feat: Support Ansi mode in abs function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #500: URL: https://github.com/apache/datafusion-comet/pull/500#discussion_r1631494071 ## core/src/execution/datafusion/planner.rs: ## @@ -356,11 +358,7 @@ impl PhysicalPlanner { let child = self.create_expr(expr.child.as_ref().u

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
comphead commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631491538 ## docs/source/user-guide/operators.md: ## @@ -19,15 +19,18 @@ # Supported Spark Operators -The following Spark operators are currently available: +The fol

Re: [I] Add ANSI support for Subtract [datafusion-comet]

2024-06-07 Thread via GitHub
planga82 commented on issue #535: URL: https://github.com/apache/datafusion-comet/issues/535#issuecomment-2155221660 I want to try this one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released [datafusion-comet]

2024-06-07 Thread via GitHub
viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2155219115 Using 39.0.0-rc1 tag now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631485590 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -980,6 +980,29 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631481199 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -980,6 +980,29 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631476538 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -980,6 +980,29 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] chore: Make ANSI fallback more granular [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #509: URL: https://github.com/apache/datafusion-comet/pull/509#discussion_r1631456108 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -712,17 +712,6 @@ class CometSparkSessionExtensions } override de

Re: [PR] chore: Make ANSI fallback more granular [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #509: URL: https://github.com/apache/datafusion-comet/pull/509#discussion_r1631442398 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -701,17 +695,17 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde wi

Re: [PR] chore: Make ANSI fallback more granular [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #509: URL: https://github.com/apache/datafusion-comet/pull/509#discussion_r1631433344 ## spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala: ## @@ -69,7 +69,7 @@ abstract class CometTestBase conf.set("spark.hadoop.fs.file.impl",

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#issuecomment-2155140117 @comphead This is the user guide version of the list of supported expressions with notes about known issues. I think this is separate from the list that you are generating for th

Re: [I] Projects require unique expressions names error in substrait producer/consumer [datafusion]

2024-06-07 Thread via GitHub
richtia commented on issue #10815: URL: https://github.com/apache/datafusion/issues/10815#issuecomment-2155139457 > > same column names with different aliases > > Isn't the repro trying to alias _different column names_ (PS_PARTKEY, PS_SUPPKEY) to _same alias_ (K1)? Why would

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#issuecomment-2155132495 @parthchandra @kazuyukitanimura @huaxingao @viirya This is ready for review now. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] docs: Improve user documentation for supported operators and expressions [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #520: URL: https://github.com/apache/datafusion-comet/pull/520#discussion_r1631425039 ## core/src/execution/datafusion/planner.rs: ## @@ -563,7 +563,7 @@ impl PhysicalPlanner { let child = self.create_expr(expr.child.as_ref().un

[I] x NOT IN y works but NOT (x IN y) doesn't [datafusion]

2024-06-07 Thread via GitHub
lostmygithubaccount opened a new issue, #10830: URL: https://github.com/apache/datafusion/issues/10830 ### Describe the bug in running some benchmarking, was getting errors like: ``` Exception: This feature is not implemented: Physical plan does not support logical expressio

Re: [PR] chore: Create initial release process scripts for official ASF source release [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #429: URL: https://github.com/apache/datafusion-comet/pull/429#discussion_r1631380186 ## dev/release/verify-release-candidate.sh: ## @@ -0,0 +1,132 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contr

Re: [PR] chore: Create initial release process scripts for official ASF source release [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on PR #429: URL: https://github.com/apache/datafusion-comet/pull/429#issuecomment-2155062685 @parthchandra @advancedxy I think I have addressed all of the feedback now -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] chore: Create initial release process scripts for official ASF source release [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #429: URL: https://github.com/apache/datafusion-comet/pull/429#discussion_r1631377169 ## dev/release/run-rat.sh: ## @@ -0,0 +1,43 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

Re: [I] Efficiently and correctly Extract Page Index statistics into `ArrayRef`s [datafusion]

2024-06-07 Thread via GitHub
marvinlanhenke commented on issue #10806: URL: https://github.com/apache/datafusion/issues/10806#issuecomment-2155061713 The proposed Api looks nice πŸ‘ŒUntil the merge I can use the time to explore and prototype. Thanks for the pointers -- This is an automated message from the Apache Git Se

Re: [PR] chore: Create initial release process scripts for official ASF source release [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on code in PR #429: URL: https://github.com/apache/datafusion-comet/pull/429#discussion_r1631376735 ## core/Cargo.toml: ## @@ -16,17 +16,21 @@ # under the License. [package] -name = "comet" +name = "datafusion-comet" version = "0.1.0" +homepage = "https:

Re: [I] Add ability to receive an iterator over the inputs of a LogicalPlan instead of a Vec. [datafusion]

2024-06-07 Thread via GitHub
LorrensP-2158466 commented on issue #10808: URL: https://github.com/apache/datafusion/issues/10808#issuecomment-2155054911 Oh yeah, you are right, I don't know why I didn't think about that. I will try it anyway, maybe I will come up with a different solution. Thanks Peter, for letting me k

Re: [PR] feat: Add support for RLike [datafusion-comet]

2024-06-07 Thread via GitHub
andygrove commented on PR #469: URL: https://github.com/apache/datafusion-comet/pull/469#issuecomment-2155043489 Could I get more feedback on this PR @parthchandra @kazuyukitanimura @huaxingao -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] Support correct output column names and struct field names when consuming/producing Substrait [datafusion]

2024-06-07 Thread via GitHub
Blizzara opened a new pull request, #10829: URL: https://github.com/apache/datafusion/pull/10829 ## Which issue does this PR close? Closes #10817 ## Rationale for this change ## What changes are included in this PR? - fix Substrait producer to inclu

Re: [PR] chore: Prepare 39.0.0-rc1 [datafusion]

2024-06-07 Thread via GitHub
andygrove merged PR #10828: URL: https://github.com/apache/datafusion/pull/10828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [I] Efficiently and correctly Extract Page Index statistics into `ArrayRef`s [datafusion]

2024-06-07 Thread via GitHub
alamb commented on issue #10806: URL: https://github.com/apache/datafusion/issues/10806#issuecomment-2154976973 Thanks @marvinlanhenke πŸ™ To write the relevant structues into Parquet, the [statistics_enable](https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertie

Re: [I] Release DataFusion `39.0.0` [datafusion]

2024-06-07 Thread via GitHub
alamb commented on issue #10517: URL: https://github.com/apache/datafusion/issues/10517#issuecomment-2154972223 Would it be possible to try and include https://github.com/apache/datafusion/pull/10802 in this release (so that we don't have to release a subsequent breaking API change next rel

Re: [PR] fix: use total ordering in the min & max accumulator for floats [datafusion]

2024-06-07 Thread via GitHub
alamb commented on code in PR #10627: URL: https://github.com/apache/datafusion/pull/10627#discussion_r1631304835 ## datafusion/physical-expr/src/aggregate/min_max.rs: ## @@ -1103,3 +1117,41 @@ impl Accumulator for SlidingMinAccumulator { std::mem::size_of_val(self) - s

Re: [I] Inconsistent null handling in min/max [datafusion]

2024-06-07 Thread via GitHub
alamb closed issue #8031: Inconsistent null handling in min/max URL: https://github.com/apache/datafusion/issues/8031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] fix: use total ordering in the min & max accumulator for floats [datafusion]

2024-06-07 Thread via GitHub
alamb merged PR #10627: URL: https://github.com/apache/datafusion/pull/10627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Rewrite `CommonSubexprEliminate` to avoid copies using TreeNode [datafusion]

2024-06-07 Thread via GitHub
alamb commented on PR #10067: URL: https://github.com/apache/datafusion/pull/10067#issuecomment-2154956308 > Sure, please work on it if you have some time during the weekend as I can't do it earlier than next week. Let me know how it goes and I can take this next week if you don't have

Re: [I] Efficiently and correctly Extract Page Index statistics into `ArrayRef`s [datafusion]

2024-06-07 Thread via GitHub
marvinlanhenke commented on issue #10806: URL: https://github.com/apache/datafusion/issues/10806#issuecomment-2154931485 @alamb I was briefly looking at this, trying to understand whats needed here (since I'm very unfamiliar here). Do we already have a helper fn at place to write

Re: [I] Feedback request for providing configurable UDF functions [datafusion]

2024-06-07 Thread via GitHub
Omega359 commented on issue #10744: URL: https://github.com/apache/datafusion/issues/10744#issuecomment-2154914120 After thinking about this a fair bit the builder approach like what @jayzhan211 did with aggregate functions seems to be the best way forward on this feature imho. While I do l

Re: [PR] bug: null character not permitted in chr function [datafusion-comet]

2024-06-07 Thread via GitHub
vaibhawvipul commented on code in PR #513: URL: https://github.com/apache/datafusion-comet/pull/513#discussion_r1631256710 ## core/src/execution/datafusion/expressions/scalar_funcs.rs: ## @@ -130,6 +133,10 @@ pub fn create_comet_physical_fun( let func = Arc::new(spa

Re: [I] Release DataFusion `39.0.0` [datafusion]

2024-06-07 Thread via GitHub
andygrove commented on issue #10517: URL: https://github.com/apache/datafusion/issues/10517#issuecomment-2154890538 PR to prepare 39.0.0-rc1: https://github.com/apache/datafusion/pull/10828 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Add ability to receive an iterator over the inputs of a LogicalPlan instead of a Vec. [datafusion]

2024-06-07 Thread via GitHub
peter-toth commented on issue #10808: URL: https://github.com/apache/datafusion/issues/10808#issuecomment-2154884925 @LorrensP-2158466, I as far as I get you just want to change the return type of `LogicalPlan ::inputs()` to use iterators instead of the current `Vec`. I think that is not re

[PR] Prepare 39.0.0-rc1 [datafusion]

2024-06-07 Thread via GitHub
andygrove opened a new pull request, #10828: URL: https://github.com/apache/datafusion/pull/10828 ## Which issue does this PR close? Part of https://github.com/apache/datafusion/issues/10517 ## Rationale for this change Scheduled release. ## What ch

Re: [I] Adopt temporalio/snipsync for documentation [datafusion]

2024-06-07 Thread via GitHub
edmondop commented on issue #10768: URL: https://github.com/apache/datafusion/issues/10768#issuecomment-2154864487 There are other such as https://github.com/SimonCropp/MarkdownSnippets, but I know Temporal is a popular OSS product, with excellent documentation, see https://temporal.io/blog

Re: [PR] Introduce expr builder for aggregate function [datafusion]

2024-06-07 Thread via GitHub
jayzhan211 commented on PR #10560: URL: https://github.com/apache/datafusion/pull/10560#issuecomment-2154855300 > I have some ideas about additional comments / documentation that I would be happy to help add Sure! -- This is an automated message from the Apache Git Service. To res

Re: [PR] Rewrite `CommonSubexprEliminate` to avoid copies using TreeNode [datafusion]

2024-06-07 Thread via GitHub
peter-toth commented on PR #10067: URL: https://github.com/apache/datafusion/pull/10067#issuecomment-2154839896 Sure, please work on it if you have some time during the weekend as I can't do it earlier than next week. Let me know how it goes and I can take this next week if you don't have

Re: [PR] fix: use total ordering in the min & max accumulator for floats [datafusion]

2024-06-07 Thread via GitHub
westonpace commented on code in PR #10627: URL: https://github.com/apache/datafusion/pull/10627#discussion_r1631196582 ## datafusion/physical-expr/src/aggregate/min_max.rs: ## @@ -1103,3 +1117,36 @@ impl Accumulator for SlidingMinAccumulator { std::mem::size_of_val(self

Re: [PR] Remove expr_fn::sum and replace them with function stub [datafusion]

2024-06-07 Thread via GitHub
jayzhan211 commented on PR #10816: URL: https://github.com/apache/datafusion/pull/10816#issuecomment-2154828340 Thanks @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] build(deps): update Arrow/Parquet to `52.0`, object-store to `0.10` [datafusion]

2024-06-07 Thread via GitHub
waynexia commented on PR #10765: URL: https://github.com/apache/datafusion/pull/10765#issuecomment-2154822425 Thanks for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] build(deps): update arrow-flight requirement from 51.0.0 to 52.0.0 [datafusion]

2024-06-07 Thread via GitHub
dependabot[bot] commented on PR #10822: URL: https://github.com/apache/datafusion/pull/10822#issuecomment-2154819969 Looks like arrow-flight is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] build(deps): update parquet requirement from 51.0.0 to 52.0.0 [datafusion]

2024-06-07 Thread via GitHub
dependabot[bot] commented on PR #10825: URL: https://github.com/apache/datafusion/pull/10825#issuecomment-2154819773 Looks like parquet is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] build(deps): update parquet requirement from 51.0.0 to 52.0.0 [datafusion]

2024-06-07 Thread via GitHub
dependabot[bot] closed pull request #10825: build(deps): update parquet requirement from 51.0.0 to 52.0.0 URL: https://github.com/apache/datafusion/pull/10825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

  1   2   >