Re: [I] Performance regression on timestemp range join. [datafusion]

2024-04-22 Thread via GitHub
Omega359 commented on issue #9755: URL: https://github.com/apache/datafusion/issues/9755#issuecomment-2069623628 Note that the first statement in this ticket does not run out of the box in main: ``` CREATE OR REPLACE TABLE pricing AS SELECT t, RANDOM() as v

[PR] Move create_physical_expr to phy-expr-common #2 [datafusion]

2024-04-22 Thread via GitHub
jayzhan211 opened a new pull request, #10176: URL: https://github.com/apache/datafusion/pull/10176 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes

[I] CLI unable to execute queries with range(..) function [datafusion]

2024-04-22 Thread via GitHub
Omega359 opened a new issue, #10177: URL: https://github.com/apache/datafusion/issues/10177 ### Describe the bug Noticed when attempting to reproduce #9755 the initial query in that issue does not work when run against current main branch: ``` ❯ docker build -f

Re: [I] Consider introducing unique expression IDs in Logical/Physical plan [datafusion]

2024-04-22 Thread via GitHub
tv42 commented on issue #8379: URL: https://github.com/apache/datafusion/issues/8379#issuecomment-2070322980 This might be a duplicate of #6543? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Minor: Add `Column::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
comphead commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575124647 ## benchmarks/src/tpch/convert.rs: ## @@ -88,9 +88,8 @@ impl ConvertOpt { .schema() .iter()

(datafusion) branch main updated: Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` (#10126)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 3c3cb87198 Improve `TreeNode` and `LogicalPlan`

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] [DOC] Add test example for backtraces [datafusion]

2024-04-22 Thread via GitHub
comphead commented on code in PR #10143: URL: https://github.com/apache/datafusion/pull/10143#discussion_r1574974271 ## docs/source/user-guide/example-usage.md: ## @@ -279,4 +279,28 @@ backtrace:0: std::backtrace_rs::backtrace::libunwind::trace ``` +The

[I] Range/inequality joins are slow [datafusion]

2024-04-22 Thread via GitHub
simonvandel opened a new issue, #8393: URL: https://github.com/apache/datafusion/issues/8393 ### Describe the bug Joins where the `ON` filter are not equality, but rather inequalities like `<`, `> etc. seem slow. Atleast compared to DuckDB which seem like a direct "competitor".

Re: [I] Range/inequality joins are slow [datafusion]

2024-04-22 Thread via GitHub
Dandandan commented on issue #8393: URL: https://github.com/apache/datafusion/issues/8393#issuecomment-2070623786 I don't think this issue should be closed. #9676 seems to take care of ordering but I think it doesn't improve range/inequality joins much? -- This is an automated

Re: [PR] minor: fix installation section link [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10179: URL: https://github.com/apache/datafusion/pull/10179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(datafusion) branch main updated: fix installation section link (#10179)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 44bc16337f fix installation section link

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10088: URL: https://github.com/apache/datafusion/pull/10088#discussion_r1575265148 ## datafusion/functions-aggregate/src/first_last.rs: ## @@ -895,6 +891,31 @@ fn convert_to_sort_cols( .collect::>() } +fn

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10088: URL: https://github.com/apache/datafusion/pull/10088#discussion_r1575269612 ## datafusion/functions-aggregate/src/first_last.rs: ## @@ -895,6 +891,31 @@ fn convert_to_sort_cols( .collect::>() } +fn

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10088: URL: https://github.com/apache/datafusion/pull/10088 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(datafusion) branch main updated: Projection Expression - Input Field Inconsistencies during Projection (#10088)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 07804384cb Projection Expression - Input Field

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10088: URL: https://github.com/apache/datafusion/pull/10088#issuecomment-2070782001 Thanks @berkaysynnada -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] `DataSink` additions [datafusion]

2024-04-22 Thread via GitHub
joroKr21 commented on code in PR #7778: URL: https://github.com/apache/datafusion/pull/7778#discussion_r1574975580 ## datafusion/physical-plan/src/insert.rs: ## @@ -151,6 +162,16 @@ impl FileSinkExec { } Ok(streams) } + +/// Returns insert sink +

Re: [PR] Minor: Add `Column::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575099919 ## datafusion/common/src/column.rs: ## @@ -346,6 +348,13 @@ impl From<(Option<>, )> for Column { } } +/// Create a column, use qualifier and field name

[PR] Minor: Add `Column::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10178: URL: https://github.com/apache/datafusion/pull/10178 ## Which issue does this PR close? Follow on to https://github.com/apache/datafusion/pull/9595 from @haohuaijin ## Rationale for this change @appletreeisyellow and I noticed

Re: [PR] feat: support input reordering for `NestedLoopJoinExec` [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #9676: URL: https://github.com/apache/datafusion/pull/9676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] feat: support input reordering for `NestedLoopJoinExec` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #9676: URL: https://github.com/apache/datafusion/pull/9676#issuecomment-2070591539 Thanks agian @korowa -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

(datafusion) branch main updated: feat: support input reordering for `NestedLoopJoinExec` (#9676)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 8f8e105072 feat: support input reordering for

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [I] Range/inequality joins are slow [datafusion]

2024-04-22 Thread via GitHub
alamb closed issue #8393: Range/inequality joins are slow URL: https://github.com/apache/datafusion/issues/8393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

(datafusion) branch main updated: Add test example for backtraces (#10143)

2024-04-22 Thread comphead
This is an automated email from the ASF dual-hosted git repository. comphead pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 0b5bfe2cfe Add test example for backtraces

Re: [PR] [DOC] Add test example for backtraces [datafusion]

2024-04-22 Thread via GitHub
comphead merged PR #10143: URL: https://github.com/apache/datafusion/pull/10143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10088: URL: https://github.com/apache/datafusion/pull/10088#issuecomment-2070754771 Is this PR waiting on anything prior to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Update github repo links [datafusion]

2024-04-22 Thread via GitHub
comphead commented on PR #10167: URL: https://github.com/apache/datafusion/pull/10167#issuecomment-2069920067 > wow, epic work @lewiszlw I clicked on some links, it works, but Installation link gives me 404 But we can do that in follow up PR to avoid conflicts -- This is an

(datafusion) branch asf-site updated: Publish built docs triggered by 0b5bfe2cfeccae6df4ee09010c1532e1299eab29

2024-04-22 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/asf-site by this push: new 3f08835b0e Publish built docs

[PR] minor: fix installation section link [datafusion]

2024-04-22 Thread via GitHub
comphead opened a new pull request, #10179: URL: https://github.com/apache/datafusion/pull/10179 ## Which issue does this PR close? Closes #. ## Rationale for this change fix installation section link ## What changes are included in this PR?

Re: [PR] Minor: Add `Column::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575249802 ## benchmarks/src/tpch/convert.rs: ## @@ -88,9 +88,8 @@ impl ConvertOpt { .schema() .iter()

Re: [PR] Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10126: URL: https://github.com/apache/datafusion/pull/10126#issuecomment-2070731173 Thanks again @peter-toth -- epic work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [I] Deprecate TreeNode `transform_xx_mut` methods [datafusion]

2024-04-22 Thread via GitHub
alamb closed issue #10097: Deprecate TreeNode `transform_xx_mut` methods URL: https://github.com/apache/datafusion/issues/10097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10126: URL: https://github.com/apache/datafusion/pull/10126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Minor: Add `Column::from(Tableref, )`, `Expr::from(Column)` and `Expr::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10178: URL: https://github.com/apache/datafusion/pull/10178#issuecomment-2070804781 > lgtm thanks @alamb CIL testdoc keeps failing Thanks -- fixed in 4db07ee31 -- This is an automated message from the Apache Git Service. To respond to the message, please log

(datafusion) branch asf-site updated: Publish built docs triggered by 465c89f7f16d48b030d4a384733567b91dab88fa

2024-04-22 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/asf-site by this push: new bba0addf64 Publish built docs

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] Update github repo links [datafusion]

2024-04-22 Thread via GitHub
andygrove commented on PR #10167: URL: https://github.com/apache/datafusion/pull/10167#issuecomment-2070285103 Thanks @lewiszlw -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update github repo links [datafusion]

2024-04-22 Thread via GitHub
andygrove merged PR #10167: URL: https://github.com/apache/datafusion/pull/10167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[I] "Cannot infer common argument type for comparison operation Union..." [datafusion]

2024-04-22 Thread via GitHub
samuelcolvin opened a new issue, #10180: URL: https://github.com/apache/datafusion/issues/10180 ### Describe the bug See https://github.com/datafusion-contrib/datafusion-functions-json/pull/3 I have a union defined by ```rs DataType::Union(

Re: [PR] feat: support input reordering for `NestedLoopJoinExec` [datafusion]

2024-04-22 Thread via GitHub
Dandandan commented on PR #9676: URL: https://github.com/apache/datafusion/pull/9676#issuecomment-2070626124 Thank you @korowa  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10166: URL: https://github.com/apache/datafusion/pull/10166#discussion_r1575275630 ## datafusion/optimizer/src/filter_null_join_keys.rs: ## @@ -100,11 +105,18 @@ fn create_not_null_predicate(filters: Vec) -> Expr { .into_iter()

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2070154517 Ok, the DNS issue has been resolved. We have the website up (needs some links fixed) https://datafusion.apache.org/ The mailing lists are working as well. For

Re: [I] Stop copying `Expr`s so much during Common Subexpression Elimination [datafusion]

2024-04-22 Thread via GitHub
peter-toth commented on issue #9873: URL: https://github.com/apache/datafusion/issues/9873#issuecomment-2070324592 I've started working on this, but it will surely take some time... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat: support input reordering for `NestedLoopJoinExec` [datafusion]

2024-04-22 Thread via GitHub
korowa commented on code in PR #9676: URL: https://github.com/apache/datafusion/pull/9676#discussion_r1575151891 ## datafusion/core/src/physical_optimizer/join_selection.rs: ## @@ -785,6 +828,35 @@ mod tests_statistical { }] } +/// Create join filter for

Re: [PR] Minor: Add `Column::from(Tableref, )`, `Expr::from(Column)` and `Expr::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575252042 ## benchmarks/src/tpch/convert.rs: ## @@ -88,9 +88,7 @@ impl ConvertOpt { .schema() .iter()

(datafusion-comet) branch main updated: feat: Add extended explain info to Comet plan (#255)

2024-04-22 Thread viirya
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git The following commit(s) were added to refs/heads/main by this push: new 6d01f6a feat: Add extended explain info

[I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new issue, #10182: URL: https://github.com/apache/datafusion/issues/10182 ### Describe the bug Something is wrong with datafusion-cli and creating external tables ### To Reproduce ``` andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion$

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070989408 Here is an example of how to make Expr::struct work in 37.1.0: https://github.com/apache/datafusion/pull/10183 I think we need a better API to do this for real (in 38.0.0

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on issue #10182: URL: https://github.com/apache/datafusion/issues/10182#issuecomment-2071007541 Take this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575451507 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review Comment:

Re: [PR] Update NOTICE.txt to be relevant to DataFusion [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10185: URL: https://github.com/apache/datafusion/pull/10185#discussion_r1575486264 ## NOTICE.txt: ## @@ -1,84 +1,5 @@ -Apache Arrow -Copyright 2016-2019 The Apache Software Foundation +Apache DataFusion +Copyright 2019-2024 The Apache Software

Re: [PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
comphead commented on code in PR #10186: URL: https://github.com/apache/datafusion/pull/10186#discussion_r1575504885 ## datafusion/common/src/error.rs: ## @@ -778,6 +817,20 @@ mod test { ); } +#[test] +fn test_strip_error_name() { +let res:

[PR] Move create_physical_expr to phy-expr-common #3 [datafusion]

2024-04-22 Thread via GitHub
jayzhan211 opened a new pull request, #10188: URL: https://github.com/apache/datafusion/pull/10188 ## Which issue does this PR close? Closes #10074 All in one in this PR. ## Rationale for this change ## What changes are included in this PR?

Re: [PR] Add benchmark for DFSchema [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] commented on PR #7948: URL: https://github.com/apache/datafusion/pull/7948#issuecomment-2071259258 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Support AND operator as alias of array intersect function [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] closed pull request #8496: Support AND operator as alias of array intersect function URL: https://github.com/apache/datafusion/pull/8496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10098: URL: https://github.com/apache/datafusion/pull/10098#discussion_r1575310332 ## datafusion/expr/src/expr.rs: ## @@ -362,10 +362,6 @@ impl Between { #[derive(Debug, Clone, PartialEq, Eq, Hash)] /// Defines which implementation of a function

Re: [I] Consider introducing unique expression IDs in Logical/Physical plan [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #8379: URL: https://github.com/apache/datafusion/issues/8379#issuecomment-2070886046 @tv42 I agree it is certainly related -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] Improve documentation on `TreeNode` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10035: URL: https://github.com/apache/datafusion/pull/10035#issuecomment-2070900137   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(datafusion) branch main updated: Improve documentation on `TreeNode` (#10035)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 16369d8612 Improve documentation on `TreeNode`

Re: [PR] Improve documentation on `TreeNode` [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10035: URL: https://github.com/apache/datafusion/pull/10035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070917587 I'll work on creating an example shortly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575437882 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on PR #10125: URL: https://github.com/apache/datafusion/pull/10125#issuecomment-2071082691 @Lordworms i added one comment, i plan to review more later / tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071088353 @Lordworms did you get the chance to compare querying with a filter / pruning involved (ideally with a range) between dashmap and sequence trie? Not sure if the dataset is

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575440108 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review

[PR] Implement rewrite for EliminateOneUnion and EliminateJoin [datafusion]

2024-04-22 Thread via GitHub
Lordworms opened a new pull request, #10184: URL: https://github.com/apache/datafusion/pull/10184 ## Which issue does this PR close? part of #9637 Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10182: URL: https://github.com/apache/datafusion/issues/10182#issuecomment-2071125951 樂 I tested it locally and likewise it seems to work just fine for me in a debug build I rebuilt a release build and it works fine. Not sure what was going on on my

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb closed issue #10182: `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo URL: https://github.com/apache/datafusion/issues/10182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Move `create_physical_expr` to `phy-expr-common` #1 [datafusion]

2024-04-22 Thread via GitHub
jayzhan211 commented on PR #10144: URL: https://github.com/apache/datafusion/pull/10144#issuecomment-2071213734 I created an ALL-in-one PR in #10188 to know what it is like after moving all the create-expr function -- This is an automated message from the Apache Git Service. To respond

(datafusion) branch main updated: implement short_circuits function for ScalarUDFImpl trait (#10168)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new f5ab312165 implement short_circuits function for

Re: [I] Adding the new API `is_short_circuits()`(default to false, when need set to true) to `ScalarUDF` and `ScalarUDFImpl` might be a good way to do this because users may want to define their own s

2024-04-22 Thread via GitHub
alamb closed issue #10162: Adding the new API `is_short_circuits()`(default to false, when need set to true) to `ScalarUDF` and `ScalarUDFImpl` might be a good way to do this because users may want to define their own short-circuit functions. URL:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
ion-elgreco commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070893136 @alamb this is how we create expressions: ```rust /// Parse a string predicate into an `Expr` pub(crate) fn parse_predicate_expression( schema: , expr:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070993030 > The "Expr walking" code now has to be aware of both the GetStructField and the ScalarUDF variants of field access. I think this can be controlled by the consumer -- for

Re: [I] Any plan to support JSON or JSONB? [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #7845: URL: https://github.com/apache/datafusion/issues/7845#issuecomment-2071004118 Thanks @WenyXu -- sounds very neat. FWI I think @samuelcolvin is also thinking about the representation in

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on code in PR #10166: URL: https://github.com/apache/datafusion/pull/10166#discussion_r1575279632 ## datafusion/optimizer/src/filter_null_join_keys.rs: ## @@ -100,11 +105,18 @@ fn create_not_null_predicate(filters: Vec) -> Expr { .into_iter()

Re: [PR] implement short_circuits function for ScalarUDFImpl trait [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10168: URL: https://github.com/apache/datafusion/pull/10168 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070887996 I think I'd be happy with 2. The example you linked is how we are using datafusion. Here is an updated example that fails with the error: ``` // For example,

Re: [PR] implement rewrite for ExtractEquijoinPredicate and avoid clone in filter [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10165: URL: https://github.com/apache/datafusion/pull/10165#discussion_r1575341213 ## datafusion/optimizer/src/extract_equijoin_predicate.rs: ## @@ -67,66 +88,97 @@ impl OptimizerRule for ExtractEquijoinPredicate { }) => { Review

Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10183: URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141 ## datafusion-examples/examples/expr_api.rs: ## @@ -248,18 +296,35 @@ fn make_ts_field(name: ) -> Field { make_field(name,

[PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10183: URL: https://github.com/apache/datafusion/pull/10183 I don't intend to merge this, but I want to put it up to illustrate how to use the 37.1.0 API to do this reasonably ## Which issue does this PR close? Part of

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071008009 > I think this can be controlled by the consumer -- for example if you are walking Exprs in lancedb, you can control when you transform Expr::GetStructField into ScalarUDF

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071068360 @Lordworms apologies it took me longer than expected to get some free time. I plan to review between tonight and tomorrow. -- This is an automated message from the

[PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10187: URL: https://github.com/apache/datafusion/pull/10187 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10124 ## Rationale for this change Old links (and google) direct to cli.html, which I

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10187: URL: https://github.com/apache/datafusion/pull/10187#discussion_r1575500022 ## docs/source/index.rst: ## @@ -103,3 +103,9 @@ Please see the `developer’s guide`_ for contributing and `communication`_ for contributor-guide/roadmap

Re: [I] Old URL for CLI docs page is showing 404 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10124: URL: https://github.com/apache/datafusion/issues/10124#issuecomment-2071176791 (I hit this issue when google searching today so figured I would make a PR) This is what I came up with : https://github.com/apache/datafusion/pull/10187 -- This is

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10187: URL: https://github.com/apache/datafusion/pull/10187#discussion_r1575500022 ## docs/source/index.rst: ## @@ -103,3 +103,9 @@ Please see the `developer’s guide`_ for contributing and `communication`_ for contributor-guide/roadmap

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070864010 Example from @ion-elgreco @alamb this is the code: ``` let (table, _metrics) = DeltaOps(table) .delete()

Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #9904: URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2070863362 Filed https://github.com/apache/datafusion/issues/10181 to track the issue with `internal error: entered unreachable code: NamedStructField should be rewritten in

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10098: URL: https://github.com/apache/datafusion/pull/10098#issuecomment-2070877181 Thanks @Omega359 for filing follow on tickets https://github.com/apache/datafusion/issues/10173 https://github.com/apache/datafusion/issues/10174

Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10183: URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141 ## datafusion-examples/examples/expr_api.rs: ## @@ -248,18 +296,35 @@ fn make_ts_field(name: ) -> Field { make_field(name,

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071090146 > @Lordworms did you get the chance to compare querying with a filter / pruning involved (ideally with a range) between dashmap and sequence trie? Not sure if the dataset is

Re: [PR] implement rewrite for ExtractEquijoinPredicate and avoid clone in filter [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10165: URL: https://github.com/apache/datafusion/pull/10165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

  1   2   >