Re: [PR] build: bump spark version to 3.4.3 [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #292: URL: https://github.com/apache/datafusion-comet/pull/292#issuecomment-2136555674 > @viirya I didn't see any changes between 3.4.2.diff and 3.4.3.diff except some index changes, line number changes and the changes I added Okay, that's great. I thought that

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
vaibhawvipul commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1618206450 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1469,5 +1469,36 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
MohamedAbdeen21 commented on code in PR #10700: URL: https://github.com/apache/datafusion/pull/10700#discussion_r1618201733 ## datafusion/functions/src/macros.rs: ## @@ -59,6 +59,30 @@ macro_rules! export_functions { }; } +macro_rules! make_function { +// single

Re: [PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 commented on code in PR #10700: URL: https://github.com/apache/datafusion/pull/10700#discussion_r1618187445 ## datafusion/functions/src/macros.rs: ## @@ -59,6 +59,30 @@ macro_rules! export_functions { }; } +macro_rules! make_function { +// single vector

Re: [PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 commented on code in PR #10700: URL: https://github.com/apache/datafusion/pull/10700#discussion_r1618187445 ## datafusion/functions/src/macros.rs: ## @@ -59,6 +59,30 @@ macro_rules! export_functions { }; } +macro_rules! make_function { +// single vector

Re: [I] Remove redundant upper case aliases for `FIRST_VALUE` and `LAST_VALUE` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed issue #10695: Remove redundant upper case aliases for `FIRST_VALUE` and `LAST_VALUE` URL: https://github.com/apache/datafusion/issues/10695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 merged PR #10696: URL: https://github.com/apache/datafusion/pull/10696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
vaibhawvipul commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1618170030 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -1959,9 +1959,19 @@ object QueryPlanSerde extends Logging with

Re: [I] Suport unparsing `LogicalPlan::Window` to SQL [datafusion]

2024-05-28 Thread via GitHub
yyy1000 commented on issue #10664: URL: https://github.com/apache/datafusion/issues/10664#issuecomment-2136478200 I tried to implement for a while and I have a question, for selecting a window function, there would be an projection. ``` > explain SELECT first_value(y) OVER (ORDER

[PR] Fix incorrect statistics read for unsigned integers columns in parquet [datafusion]

2024-05-28 Thread via GitHub
xinlifoobar opened a new pull request, #10704: URL: https://github.com/apache/datafusion/pull/10704 ## Which issue does this PR close? Closes #10604 ## Rationale for this change ## What changes are included in this PR? ## Are these changes

Re: [PR] build: bump spark version to 3.4.3 [datafusion-comet]

2024-05-28 Thread via GitHub
huaxingao commented on PR #292: URL: https://github.com/apache/datafusion-comet/pull/292#issuecomment-2136444508 @viirya I didn't see any changes between 3.4.2.diff and 3.4.3.diff except some index changes, line number changes and the changes I added ``` @@ -1128,7 +1130,7 @@

Re: [PR] support serialization and deserialization limit in the aggregation exec [datafusion]

2024-05-28 Thread via GitHub
liukun4515 commented on PR #10692: URL: https://github.com/apache/datafusion/pull/10692#issuecomment-2136417448 > Looks good to me -- thank you @liukun4515 Thanks for your such amazing quick feedback. -- This is an automated message from the Apache Git Service. To respond to the

[PR] Profile spark3.5.1 and centos7 for compatible on spark 3.5.1 and centos7 old glic 2.7 [datafusion-comet]

2024-05-28 Thread via GitHub
awol2005ex opened a new pull request, #491: URL: https://github.com/apache/datafusion-comet/pull/491 Profile spark3.5.1 and centos7 for compatible on spark 3.5.1 and centos7 old glic 2.7 Run: ./build_for_centos7.sh And the jar in path :

Re: [PR] fix: Removing copying data from dictionary values into CometDictionary [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #490: URL: https://github.com/apache/datafusion-comet/pull/490#discussion_r1618088185 ## common/src/main/java/org/apache/comet/vector/CometDictionary.java: ## @@ -59,121 +46,56 @@ public ValueVector getValueVector() { } public boolean

[PR] fix: Removing copying data from dictionary values into CometDictionary [datafusion-comet]

2024-05-28 Thread via GitHub
viirya opened a new pull request, #490: URL: https://github.com/apache/datafusion-comet/pull/490 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
vaibhawvipul commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1618086496 ## core/src/execution/datafusion/expressions/negative.rs: ## @@ -0,0 +1,350 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] build: bump spark version to 3.4.3 [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #292: URL: https://github.com/apache/datafusion-comet/pull/292#discussion_r1618068217 ## dev/diffs/3.4.3.diff: ## @@ -0,0 +1,2564 @@ +diff --git a/pom.xml b/pom.xml +index d3544881af1..47382e29b5a 100644 +--- a/pom.xml b/pom.xml +@@ -148,6

Re: [PR] test: show stats in explain of two representative queries [datafusion]

2024-05-28 Thread via GitHub
github-actions[bot] closed pull request #8173: test: show stats in explain of two representative queries URL: https://github.com/apache/datafusion/pull/8173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: track memory consumers for GreedyMemoryPoolState [datafusion]

2024-05-28 Thread via GitHub
github-actions[bot] commented on PR #9015: URL: https://github.com/apache/datafusion/pull/9015#issuecomment-2136370655 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Add GroupValuesFullyOrdered mode to GroupValues trait for aggregate grouping. [datafusion]

2024-05-28 Thread via GitHub
github-actions[bot] commented on PR #9662: URL: https://github.com/apache/datafusion/pull/9662#issuecomment-2136370624 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Draft: Optimize to_timestamp [datafusion]

2024-05-28 Thread via GitHub
github-actions[bot] closed pull request #9694: Draft: Optimize to_timestamp URL: https://github.com/apache/datafusion/pull/9694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] fix: move array_except in SetOp and support Null columnar in `array_except`, `array_union` and `array_intersect` [datafusion]

2024-05-28 Thread via GitHub
github-actions[bot] commented on PR #9710: URL: https://github.com/apache/datafusion/pull/9710#issuecomment-2136370571 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 commented on code in PR #10696: URL: https://github.com/apache/datafusion/pull/10696#discussion_r1618065968 ## datafusion/functions-aggregate/src/lib.rs: ## @@ -109,13 +109,13 @@ mod tests { let mut names = HashSet::new(); for func in

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
goldmedal commented on code in PR #10696: URL: https://github.com/apache/datafusion/pull/10696#discussion_r1618064435 ## datafusion/functions-aggregate/src/lib.rs: ## @@ -109,13 +109,13 @@ mod tests { let mut names = HashSet::new(); for func in

Re: [PR] Aggregate UDF Redesign: Introduce `expr-common` and `funcitons-aggregate-common` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10327: Aggregate UDF Redesign: Introduce `expr-common` and `funcitons-aggregate-common` URL: https://github.com/apache/datafusion/pull/10327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Move create_physical_expr to phy-expr-common #3 [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10188: Move create_physical_expr to phy-expr-common #3 URL: https://github.com/apache/datafusion/pull/10188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Introduce `reverse_expr` for UDAF [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10214: Introduce `reverse_expr` for UDAF URL: https://github.com/apache/datafusion/pull/10214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Minor: Switch LastValue SQL workflow to UDAF version [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10062: Minor: Switch LastValue SQL workflow to UDAF version URL: https://github.com/apache/datafusion/pull/10062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Move `create_physical_expr` to `physical-expr-common` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed issue #10074: Move `create_physical_expr` to `physical-expr-common` URL: https://github.com/apache/datafusion/issues/10074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Move create_physical_expr to phy-expr-common #2 [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10176: Move create_physical_expr to phy-expr-common #2 URL: https://github.com/apache/datafusion/pull/10176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Move `create_physical_expr` to `phy-expr-common` #1 [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10144: Move `create_physical_expr` to `phy-expr-common` #1 URL: https://github.com/apache/datafusion/pull/10144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Deprecate builtin first/last aggregate function and use UDAF [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 closed pull request #10091: Deprecate builtin first/last aggregate function and use UDAF URL: https://github.com/apache/datafusion/pull/10091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 commented on code in PR #10696: URL: https://github.com/apache/datafusion/pull/10696#discussion_r1618012266 ## datafusion/functions-aggregate/src/lib.rs: ## @@ -109,13 +109,13 @@ mod tests { let mut names = HashSet::new(); for func in

[I] Decouple builtin aggregate function from optimizer [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 opened a new issue, #10703: URL: https://github.com/apache/datafusion/issues/10703 ### Is your feature request related to a problem or challenge? If we move the built-in function out of the optimizer, it is more easy to extend/adapt the optimizer rule to other UDAFs. For

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
jayzhan211 commented on code in PR #10696: URL: https://github.com/apache/datafusion/pull/10696#discussion_r1618012266 ## datafusion/functions-aggregate/src/lib.rs: ## @@ -109,13 +109,13 @@ mod tests { let mut names = HashSet::new(); for func in

Re: [PR] fix: Fallback to Spark for LIKE with custom escape character [datafusion-comet]

2024-05-28 Thread via GitHub
andygrove merged PR #478: URL: https://github.com/apache/datafusion-comet/pull/478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on PR #10392: URL: https://github.com/apache/datafusion/pull/10392#issuecomment-2136253325 Ensure all the sqllogictests passed without the array index cases: Details ``` ### Array index ## array[i] # single index with scalars #1 (positive

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on code in PR #10392: URL: https://github.com/apache/datafusion/pull/10392#discussion_r1617982950 ## datafusion/sqllogictest/test_files/array.slt: ## @@ -689,7 +689,7 @@ select column1, column2, column3, column4, column5 from nested_arrays; # values table

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on code in PR #10392: URL: https://github.com/apache/datafusion/pull/10392#discussion_r1617979884 ## datafusion/sql/src/expr/mod.rs: ## @@ -215,11 +216,28 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { _ => (expr, false),

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on code in PR #10392: URL: https://github.com/apache/datafusion/pull/10392#discussion_r1617977560 ## datafusion/sql/src/expr/mod.rs: ## @@ -215,11 +216,28 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { _ => (expr, false),

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on code in PR #10392: URL: https://github.com/apache/datafusion/pull/10392#discussion_r1617967562 ## datafusion/sql/src/expr/mod.rs: ## @@ -215,11 +216,28 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { _ => (expr, false),

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on PR #10392: URL: https://github.com/apache/datafusion/pull/10392#issuecomment-2136212148 I've handled all other trivial new ASTs. But how to migrate `SQLExpr::ArrayIndex` to `SQLExpr::Subscript` still need some time to work out. Perhaps @jmhain can take a look and

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-28 Thread via GitHub
tisonkun commented on PR #10392: URL: https://github.com/apache/datafusion/pull/10392#issuecomment-2136204241 I'm testing on @jmhain's branch on the subscript fixes and encounter the following new API breaks: ``` Compiling datafusion-sql v38.0.0

Re: [PR] feat: Add "Comet Fuzz" fuzz-testing utility [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #472: URL: https://github.com/apache/datafusion-comet/pull/472#discussion_r1617944984 ## fuzz-testing/pom.xml: ## @@ -0,0 +1,106 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +

Re: [I] Substrait integration doesn't recognize typed functions [datafusion]

2024-05-28 Thread via GitHub
EpsilonPrime commented on issue #10412: URL: https://github.com/apache/datafusion/issues/10412#issuecomment-2136173407 Be wary, Gluten contains a copy of Substrait instead of depending on the main repo. As a result its Substrait is incompatible with the rest of the ecosystem. That works

Re: [I] UNION ALL and AVG returns unexpected result for floats [datafusion]

2024-05-28 Thread via GitHub
comphead commented on issue #10688: URL: https://github.com/apache/datafusion/issues/10688#issuecomment-2136159580 Thats weird. ``` > explain select -128.2::float union all select -128.2; +---+---+ |

Re: [PR] fix: Fallback to Spark for LIKE with custom escape character [datafusion-comet]

2024-05-28 Thread via GitHub
sujithjay commented on code in PR #478: URL: https://github.com/apache/datafusion-comet/pull/478#discussion_r1617923861 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -578,6 +578,24 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] fix: Fallback to Spark for LIKE with custom escape character [datafusion-comet]

2024-05-28 Thread via GitHub
andygrove commented on code in PR #478: URL: https://github.com/apache/datafusion-comet/pull/478#discussion_r1617882461 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -578,6 +578,24 @@ class CometExpressionSuite extends CometTestBase with

Re: [I] UNION ALL and AVG returns unexpected result for floats [datafusion]

2024-05-28 Thread via GitHub
viirya commented on issue #10688: URL: https://github.com/apache/datafusion/issues/10688#issuecomment-2136086622 When `-128.2::float` is treated as float64, there is some precision difference. ``` > explain select avg(a) from (select -128.2::float a union all select 32768.3 union

Re: [PR] fix: Reuse previous CometDictionary Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #489: URL: https://github.com/apache/datafusion-comet/pull/489#issuecomment-2136083084 Merged. Thanks @andygrove @kazuyukitanimura -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Reuse previous `CometDictionary` Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya closed issue #488: Reuse previous `CometDictionary` Java arrays URL: https://github.com/apache/datafusion-comet/issues/488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] fix: Reuse previous CometDictionary Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya merged PR #489: URL: https://github.com/apache/datafusion-comet/pull/489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] feat: Add HashJoin support for BuildRight [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #437: URL: https://github.com/apache/datafusion-comet/pull/437#issuecomment-2136077501 This requires https://github.com/apache/datafusion/pull/10702 to be in DataFusion new release. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Make swap_hash_join public API [datafusion]

2024-05-28 Thread via GitHub
viirya commented on code in PR #10702: URL: https://github.com/apache/datafusion/pull/10702#discussion_r1617872081 ## datafusion/core/src/physical_optimizer/join_selection.rs: ## @@ -157,7 +157,9 @@ fn swap_join_projection( } /// This function swaps the inputs of the given

[PR] Make swap_hash_join public API [datafusion]

2024-05-28 Thread via GitHub
viirya opened a new pull request, #10702: URL: https://github.com/apache/datafusion/pull/10702 ## Which issue does this PR close? Closes #9603. ## Rationale for this change ## What changes are included in this PR? ## Are these changes

Re: [PR] fix: Reuse previous CometDictionary Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #489: URL: https://github.com/apache/datafusion-comet/pull/489#issuecomment-2136063966 Thank you @kazuyukitanimura -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] fix: Reuse previous CometDictionary Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #489: URL: https://github.com/apache/datafusion-comet/pull/489#issuecomment-2136063825 cc @andygrove too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] fix: Fallback to Spark for LIKE with custom escape character [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #478: URL: https://github.com/apache/datafusion-comet/pull/478#discussion_r1617851614 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -578,6 +578,24 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] Support consuming Substrait with compound signature function names [datafusion]

2024-05-28 Thread via GitHub
richtia commented on PR #10653: URL: https://github.com/apache/datafusion/pull/10653#issuecomment-2136031347 > > though curious if you have any thoughts on how to make it simpler - given DF doesn't yet produce those compound names, I cannot use a roundtrip test, and writing the substrait

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617839444 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,371 @@ +// Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1617835890 ## core/src/execution/datafusion/expressions/negative.rs: ## @@ -0,0 +1,350 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1617820865 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1469,5 +1469,36 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1617816507 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1469,5 +1469,36 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617545329 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,296 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: Support CartesianProductExec in comet [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #442: URL: https://github.com/apache/datafusion-comet/pull/442#issuecomment-2135981042 @leoluan2009 You may need to generate plan stability results to pass the CI pipelines. -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] fix: Reuse previous CometDictionary Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya opened a new pull request, #489: URL: https://github.com/apache/datafusion-comet/pull/489 ## Which issue does this PR close? Closes #488. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

[I] Reuse previous `CometDictionary` Java arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya opened a new issue, #488: URL: https://github.com/apache/datafusion-comet/issues/488 ### Describe the bug After #474, `ColumnReader.loadVector` initiates `CometDictionary` after re-import arrays. It is reported by @andygrove

Re: [PR] fix: Fallback to Spark for LIKE with custom escape character [datafusion-comet]

2024-05-28 Thread via GitHub
sujithjay commented on code in PR #478: URL: https://github.com/apache/datafusion-comet/pull/478#discussion_r1617782025 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -981,7 +981,7 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde {

Re: [PR] fix: `ColumnReader.loadVector` should initiate `CometDictionary` after re-import arrays [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #473: URL: https://github.com/apache/datafusion-comet/pull/473#issuecomment-2135939279 I got the reason why it has regression. I will propose a fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] build: Add spark-4.0 profile and shims [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura commented on PR #407: URL: https://github.com/apache/datafusion-comet/pull/407#issuecomment-2135933132 Thank you @viirya merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat: Add HashJoin support for BuildRight [datafusion-comet]

2024-05-28 Thread via GitHub
viirya closed pull request #487: feat: Add HashJoin support for BuildRight URL: https://github.com/apache/datafusion-comet/pull/487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] build: Add spark-4.0 profile and shims [datafusion-comet]

2024-05-28 Thread via GitHub
kazuyukitanimura merged PR #407: URL: https://github.com/apache/datafusion-comet/pull/407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Alamb/advanced parquet index [datafusion]

2024-05-28 Thread via GitHub
alamb opened a new pull request, #10701: URL: https://github.com/apache/datafusion/pull/10701 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10580 ## Rationale for this change See

Re: [PR] Remove redundant upper case aliases for `median`, `first_value` and `last_value` [datafusion]

2024-05-28 Thread via GitHub
alamb commented on code in PR #10696: URL: https://github.com/apache/datafusion/pull/10696#discussion_r1617756810 ## datafusion/functions-aggregate/src/first_last.rs: ## @@ -72,7 +72,7 @@ impl Default for FirstValue { impl FirstValue { pub fn new() -> Self { Self

Re: [PR] build: Add spark-4.0 profile and shims [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on PR #407: URL: https://github.com/apache/datafusion-comet/pull/407#issuecomment-2135868331 @kazuyukitanimura I think you can try to merge this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] feat: Add HashJoin support for BuildRight [datafusion-comet]

2024-05-28 Thread via GitHub
viirya opened a new pull request, #487: URL: https://github.com/apache/datafusion-comet/pull/487 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
MohamedAbdeen21 commented on code in PR #10700: URL: https://github.com/apache/datafusion/pull/10700#discussion_r1617618779 ## datafusion/functions/src/core/mod.rs: ## @@ -41,70 +38,40 @@ make_udf_function!(r#struct::StructFunc, STRUCT, r#struct);

Re: [PR] feat: add substrait support for Interval types and literals [datafusion]

2024-05-28 Thread via GitHub
alamb commented on code in PR #10646: URL: https://github.com/apache/datafusion/pull/10646#discussion_r1617607185 ## datafusion/substrait/src/variation_const.rs: ## @@ -37,3 +38,58 @@ pub const DEFAULT_CONTAINER_TYPE_REF: u32 = 0; pub const LARGE_CONTAINER_TYPE_REF: u32 = 1;

Re: [PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
MohamedAbdeen21 commented on code in PR #10700: URL: https://github.com/apache/datafusion/pull/10700#discussion_r1617606993 ## datafusion/functions/src/macros.rs: ## @@ -59,6 +59,30 @@ macro_rules! export_functions { }; } +macro_rules! make_function { +// single

Re: [I] Suport unparsing `LogicalPlan::Distinct` to `DISTINCT` [datafusion]

2024-05-28 Thread via GitHub
alamb closed issue #10663: Suport unparsing `LogicalPlan::Distinct` to `DISTINCT` URL: https://github.com/apache/datafusion/issues/10663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Support LogicalPlan::Distinct in unparser [datafusion]

2024-05-28 Thread via GitHub
alamb merged PR #10690: URL: https://github.com/apache/datafusion/pull/10690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Reduce repetition in math and functions modules with macros [datafusion]

2024-05-28 Thread via GitHub
MohamedAbdeen21 opened a new pull request, #10700: URL: https://github.com/apache/datafusion/pull/10700 ## Which issue does this PR close? Possibly closes #10397. ## Rationale for this change Extending the `export_function!` macro to be used in math and

[I] DataFusion weekly project plan (Andrew Lamb) - May 27, 2024 [datafusion]

2024-05-28 Thread via GitHub
alamb opened a new issue, #10699: URL: https://github.com/apache/datafusion/issues/10699 Follow on to https://github.com/apache/datafusion/issues/10579 My (personal) North ⭐ : 1000 projects are built using DataFusion  **It would be great for other contributors to DataFusion

Re: [I] DataFusion weekly project plan (Andrew Lamb) - May 20, 2024 [datafusion]

2024-05-28 Thread via GitHub
alamb commented on issue #10579: URL: https://github.com/apache/datafusion/issues/10579#issuecomment-2135636751 Next week: https://github.com/apache/datafusion/issues/10699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] DataFusion weekly project plan (Andrew Lamb) - May 20, 2024 [datafusion]

2024-05-28 Thread via GitHub
alamb closed issue #10579: DataFusion weekly project plan (Andrew Lamb) - May 20, 2024 URL: https://github.com/apache/datafusion/issues/10579 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Support consuming Substrait with compound signature function names [datafusion]

2024-05-28 Thread via GitHub
waynexia commented on PR #10653: URL: https://github.com/apache/datafusion/pull/10653#issuecomment-2135630314 I've skimmed this patch and it looks good to me :+1: I'll take a detailed look into the test part tomorrow. -- This is an automated message from the Apache Git Service. To

Re: [I] Substrait integration doesn't recognize typed functions [datafusion]

2024-05-28 Thread via GitHub
waynexia commented on issue #10412: URL: https://github.com/apache/datafusion/issues/10412#issuecomment-2135626355 Sounds great!! I'm very interested in this project, looking forward to your progress! -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Support consuming Substrait with compound signature function names [datafusion]

2024-05-28 Thread via GitHub
Blizzara commented on PR #10653: URL: https://github.com/apache/datafusion/pull/10653#issuecomment-2135606718 > As a big scenario for our substrait support is to inter-cooperation with other systems, it might be a good idea to have some "integration" test, which uses another project like

Re: [I] June 2024 ASF Board Report [datafusion]

2024-05-28 Thread via GitHub
alamb commented on issue #10155: URL: https://github.com/apache/datafusion/issues/10155#issuecomment-2135606371 Also posted to mailing list https://lists.apache.org/thread/199ymolos20sr9vvz5ctv6j2nnrgrbo2 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Example for building an external index for parquet files [datafusion]

2024-05-28 Thread via GitHub
alamb commented on issue #10546: URL: https://github.com/apache/datafusion/issues/10546#issuecomment-2135602160 Update here is I have a basic example https://github.com/apache/datafusion/pull/10549 ready for review / merge -- This is an automated message from the Apache Git Service. To

Re: [I] Substrait integration doesn't recognize typed functions [datafusion]

2024-05-28 Thread via GitHub
Blizzara commented on issue #10412: URL: https://github.com/apache/datafusion/issues/10412#issuecomment-2135601093 Yes! I'm working on using DataFusion to basically execute Spark dataframes through Spark -> Substrait -> DataFusion. The Spark -> Substrait part is a (currently closed-source,

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617547871 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1038,6 +1038,20 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617547465 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1038,6 +1038,20 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617545329 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,296 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617545329 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,296 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617541513 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,296 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617538649 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,191 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: add hex scalar function [datafusion-comet]

2024-05-28 Thread via GitHub
viirya commented on code in PR #449: URL: https://github.com/apache/datafusion-comet/pull/449#discussion_r1617533349 ## core/src/execution/datafusion/expressions/scalar_funcs/hex.rs: ## @@ -0,0 +1,371 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [I] Substrait integration doesn't recognize typed functions [datafusion]

2024-05-28 Thread via GitHub
waynexia commented on issue #10412: URL: https://github.com/apache/datafusion/issues/10412#issuecomment-2135569464 Thanks for taking this @Blizzara! Just out of curiosity, are you using this datafusion-substrait somewhere and find this inconsistency? -- This is an automated message from

Re: [PR] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

2024-05-28 Thread via GitHub
andygrove commented on code in PR #471: URL: https://github.com/apache/datafusion-comet/pull/471#discussion_r1617526204 ## core/src/execution/datafusion/planner.rs: ## @@ -566,8 +567,23 @@ impl PhysicalPlanner { Ok(Arc::new(NotExpr::new(child))) }

  1   2   >