Re: [PR] perf: Improve NLJ for very small right side case [datafusion]

2025-09-15 Thread via GitHub
comphead commented on code in PR #17562: URL: https://github.com/apache/datafusion/pull/17562#discussion_r2349437381 ## datafusion/physical-plan/src/joins/nested_loop_join.rs: ## @@ -1232,11 +1235,49 @@ impl NestedLoopJoinStream { // and push the result into output_buff

Re: [PR] perf: Improve the performance of WINDOW functions with many partitions [datafusion]

2025-09-15 Thread via GitHub
nuno-faria commented on code in PR #17528: URL: https://github.com/apache/datafusion/pull/17528#discussion_r2348169771 ## datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs: ## @@ -870,9 +879,11 @@ impl SortedSearch { cur_window_expr_out_result_len

[PR] Upgrade arrow/parquet to 56.1.0 [datafusion]

2025-09-15 Thread via GitHub
AdamGS opened a new pull request, #17571: URL: https://github.com/apache/datafusion/pull/17571 ## Which issue does this PR close? - Related to https://github.com/apache/arrow-rs/issues/7837 ## Rationale for this change Aside from general maintenance and performance improv

Re: [PR] chore(deps): bump rust_decimal from 1.37.2 to 1.38.0 [datafusion]

2025-09-15 Thread via GitHub
Jefffrey merged PR #17564: URL: https://github.com/apache/datafusion/pull/17564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Enable feature specific unit tests [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra closed issue #2360: Enable feature specific unit tests URL: https://github.com/apache/datafusion-comet/issues/2360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: Add `OR REPLACE` to creating external tables [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n commented on code in PR #17580: URL: https://github.com/apache/datafusion/pull/17580#discussion_r2350461808 ## datafusion/sql/src/parser.rs: ## @@ -724,13 +728,29 @@ impl<'a> DFParser<'a> { /// Parse a SQL `CREATE` statement handling `CREATE EXTERNAL TABLE`

Re: [PR] Use `Display` formatting of `DataType`:s in error messagwe [datafusion]

2025-09-15 Thread via GitHub
alamb commented on PR #17565: URL: https://github.com/apache/datafusion/pull/17565#issuecomment-3292252841 > I recommend running cargo test locally first to catch all the test errors and fix them in one go, in case you're not already doing so My personal favorite is `cargo nextest

Re: [PR] feat: `ClassicJoin` for PWMJ [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n commented on code in PR #17482: URL: https://github.com/apache/datafusion/pull/17482#discussion_r2350515225 ## datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs: ## @@ -0,0 +1,1471 @@ +// Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Draft: Test fast gc for sort string view [datafusion]

2025-09-15 Thread via GitHub
github-actions[bot] commented on PR #16739: URL: https://github.com/apache/datafusion/pull/16739#issuecomment-3294576342 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [I] CometHashAggregate prefixed with ! in explain plan [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on issue #2214: URL: https://github.com/apache/datafusion-comet/issues/2214#issuecomment-3294265715 Spark considers `count#51L` to be as input, and thinks it is missing. This is the reason for exclamation mark -- This is an automated message from the Apache Git Service

[I] Expose physical plan optimizer [datafusion-python]

2025-09-15 Thread via GitHub
timsaucer opened a new issue, #1238: URL: https://github.com/apache/datafusion-python/issues/1238 **Is your feature request related to a problem or challenge? Please describe what you are trying to do.** I want to add custom physical plan optimizer rules, written in rust, to `datafus

Re: [PR] feat(spark): implement Spark `try_parse_url` function [datafusion]

2025-09-15 Thread via GitHub
Jefffrey commented on code in PR #17485: URL: https://github.com/apache/datafusion/pull/17485#discussion_r2350519469 ## datafusion/spark/src/function/url/parse_url.rs: ## @@ -154,7 +151,7 @@ impl ScalarUDFImpl for ParseUrl { ); } match arg_types.le

Re: [PR] Add Comet 0.10.0 blog post draft [datafusion-site]

2025-09-15 Thread via GitHub
alamb commented on code in PR #111: URL: https://github.com/apache/datafusion-site/pull/111#discussion_r2348980448 ## content/blog/2025-09-13-datafusion-comet-0.10.0.md: ## @@ -0,0 +1,132 @@ +--- +layout: post +title: Apache DataFusion Comet 0.10.0 Release +date: 2025-09-13 +aut

Re: [PR] feat: `ClassicJoin` for PWMJ [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n commented on code in PR #17482: URL: https://github.com/apache/datafusion/pull/17482#discussion_r2350515225 ## datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs: ## @@ -0,0 +1,1471 @@ +// Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] feat: feature specific tests [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on code in PR #2372: URL: https://github.com/apache/datafusion-comet/pull/2372#discussion_r2350200597 ## native/core/src/lib.rs: ## @@ -100,6 +100,35 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_init( const LOG_PATTERN: &str = "{d(%y/%m/%d %H

Re: [I] CometHashAggregate prefixed with ! in explain plan [datafusion-comet]

2025-09-15 Thread via GitHub
andygrove commented on issue #2214: URL: https://github.com/apache/datafusion-comet/issues/2214#issuecomment-3294369630 > Spark considers `count#51L` to be as input, and thinks it is missing. This is the reason for exclamation mark > > UPD: Spark `HashAggregate.producedAttributes` co

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
jonahgao commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350522060 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] feat: `ClassicJoin` for PWMJ [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n commented on PR #17482: URL: https://github.com/apache/datafusion/pull/17482#issuecomment-3294640608 I'll try to complete all the refactoring tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Introduce wildcard const for FixedSizeBinary type signature [datafusion]

2025-09-15 Thread via GitHub
Jefffrey commented on PR #17531: URL: https://github.com/apache/datafusion/pull/17531#issuecomment-3294613301 > I realized we may get away without introducing a generic type concept. The existing `TypeSignature::Coercible`'s `TypeSignatureClass` seems to be more or less it. did you try to u

Re: [PR] optimizer: Rewrite `IS NOT DISTINCT FROM` joins as Hash Joins [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n commented on PR #17319: URL: https://github.com/apache/datafusion/pull/17319#issuecomment-3294611283 @comphead @adriangb This should be an easy pull request to understand and review. PTAL when you have the time! -- This is an automated message from the Apache Git Service. To r

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
zhuqi-lucas commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350461412 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the com

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
goldmedal commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350484503 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commu

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
Jefffrey commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350493799 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] bug: Support null as argument to to_local_time [datafusion]

2025-09-15 Thread via GitHub
petern48 commented on code in PR #17491: URL: https://github.com/apache/datafusion/pull/17491#discussion_r2333987540 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -360,6 +361,7 @@ impl ScalarUDFImpl for ToLocalTimeFunc { match time_value {

Re: [I] Enable the `ListFilesCache` to be available for partitioned tables [datafusion]

2025-09-15 Thread via GitHub
alamb commented on issue #17211: URL: https://github.com/apache/datafusion/issues/17211#issuecomment-3270891819 > [@alamb](https://github.com/alamb) Would you feel comfortable gathering some initial data for this effort using the existing code in > > * [POC: datafusion-cli instrumente

Re: [PR] make `giscus` comment section opt-in to comply with ASF policy [datafusion-site]

2025-09-15 Thread via GitHub
kevinjqliu commented on PR #106: URL: https://github.com/apache/datafusion-site/pull/106#issuecomment-3267246631 one step closer! thanks for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
Ted-Jiang commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350497696 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commu

Re: [PR] feat: Support comet native log level conf [datafusion-comet]

2025-09-15 Thread via GitHub
wForget commented on code in PR #2379: URL: https://github.com/apache/datafusion-comet/pull/2379#discussion_r2350465015 ## common/src/main/java/org/apache/comet/NativeBase.java: ## @@ -153,18 +154,25 @@ private static void bundleLoadLibrary() { private static void initWithL

[PR] feat: Add `OR REPLACE` to creating external tables [datafusion]

2025-09-15 Thread via GitHub
jonathanc-n opened a new pull request, #17580: URL: https://github.com/apache/datafusion/pull/17580 ## Which issue does this PR close? - Closes #17496 . ## What changes are included in this PR? Added parsing for OR REPLACE in create external tables + functionality for

[PR] minor: fix compilation issue for extended tests due to missing parquet encryption flag [datafusion]

2025-09-15 Thread via GitHub
Jefffrey opened a new pull request, #17579: URL: https://github.com/apache/datafusion/pull/17579 `Datafusion extended tests / cargo test hash collisions (amd64) (push)` currently failing on main: ``` error[E0560]: struct `ParquetOpener` has no field named `file_decryption_propert

[I] Use taiki-e/install-action in CI [datafusion]

2025-09-15 Thread via GitHub
AdamGS opened a new issue, #17572: URL: https://github.com/apache/datafusion/issues/17572 I suggest using the [taiki-e/install-action](https://github.com/taiki-e/install-action) github action in CI (when applicable), it uses binstall and other specialized configs to install popular rust to

Re: [PR] Prevent exponential planning time for Window functions [datafusion]

2025-09-15 Thread via GitHub
findepi commented on PR #17563: URL: https://github.com/apache/datafusion/pull/17563#issuecomment-3291024985 Will post benchmark results soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] chore: Add hdfs feature test job [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on code in PR #2350: URL: https://github.com/apache/datafusion-comet/pull/2350#discussion_r2349985476 ## spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala: ## @@ -1138,4 +1138,9 @@ abstract class CometTestBase usingDataSourceExec(conf) &&

Re: [PR] feat: feature specific tests [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on PR #2372: URL: https://github.com/apache/datafusion-comet/pull/2372#issuecomment-3294318867 merged. Thank you @comphead @wForget -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] chore(deps): bump substrait from 0.58.0 to 0.59.1 [datafusion]

2025-09-15 Thread via GitHub
dependabot[bot] commented on PR #17317: URL: https://github.com/apache/datafusion/pull/17317#issuecomment-3291342135 Superseded by #17567. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
mvzink commented on code in PR #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029#discussion_r2350269012 ## src/parser/mod.rs: ## @@ -7061,19 +7061,27 @@ impl<'a> Parser<'a> { pub fn parse_create_index(&mut self, unique: bool) -> Result { let

Re: [I] CometHashAggregate prefixed with ! in explain plan [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on issue #2214: URL: https://github.com/apache/datafusion-comet/issues/2214#issuecomment-3294213466 ``` test("invalid CometHashAggregate") { sql("CREATE TABLE t(id int, value int) USING PARQUET") sql("INSERT INTO t VALUES (1, 10)") sql("INSERT INTO

Re: [I] CometHashAggregate prefixed with ! in explain plan [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on issue #2214: URL: https://github.com/apache/datafusion-comet/issues/2214#issuecomment-3294143899 I remember hitting this `!` on other operators, like `CometTakeOrderedAndProject`, checking it -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] feat: [iceberg] delete rows support using selection vectors [datafusion-comet]

2025-09-15 Thread via GitHub
andygrove commented on PR #2346: URL: https://github.com/apache/datafusion-comet/pull/2346#issuecomment-3271342810 @parthchandra Ci failures are now resolved in `main` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] CometHashAggregate prefixed with ! in explain plan [datafusion-comet]

2025-09-15 Thread via GitHub
comphead commented on issue #2214: URL: https://github.com/apache/datafusion-comet/issues/2214#issuecomment-3294163671 @rishvin does it happen with AQE only? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
MohamedAbdeen21 commented on code in PR #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029#discussion_r2350178081 ## src/parser/mod.rs: ## @@ -7061,19 +7061,27 @@ impl<'a> Parser<'a> { pub fn parse_create_index(&mut self, unique: bool) -> Result {

Re: [PR] feat: feature specific tests [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on code in PR #2372: URL: https://github.com/apache/datafusion-comet/pull/2372#discussion_r2350084785 ## native/core/src/lib.rs: ## @@ -100,6 +100,35 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_init( const LOG_PATTERN: &str = "{d(%y/%m/%

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
wjones127 commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350202157 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commu

Re: [PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
MohamedAbdeen21 commented on code in PR #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029#discussion_r2350182971 ## src/parser/mod.rs: ## @@ -8588,6 +8596,14 @@ impl<'a> Parser<'a> { } } +pub fn parse_using_index_type_clause(&mut self)

Re: [PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
mvzink commented on code in PR #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029#discussion_r2350170962 ## src/parser/mod.rs: ## @@ -7061,19 +7061,27 @@ impl<'a> Parser<'a> { pub fn parse_create_index(&mut self, unique: bool) -> Result { let

[PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
MohamedAbdeen21 opened a new pull request, #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029 MySQL allows specifying the index type `USING index_type` before the `ON` clause in `CREATE INDEX` statements. This PR allows the `CREATE INDEX` parser to accept both pos

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
ozankabak commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349992296 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commu

Re: [I] Remove on-heap memory pools [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on issue #2342: URL: https://github.com/apache/datafusion-comet/issues/2342#issuecomment-3294012149 To clarify, this is the JVM's on-heap memory we're taking about. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
andygrove commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349648258 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commu

Re: [PR] MySQL: allow USING clause before ON in CREATE INDEX [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
MohamedAbdeen21 commented on code in PR #2029: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2029#discussion_r2350157341 ## src/parser/mod.rs: ## @@ -7061,19 +7061,27 @@ impl<'a> Parser<'a> { pub fn parse_create_index(&mut self, unique: bool) -> Result {

Re: [PR] fix: ignore non-existent columns when adding filter equivalence info in `FileScanConfig` [datafusion]

2025-09-15 Thread via GitHub
rkrishn7 commented on PR #17546: URL: https://github.com/apache/datafusion/pull/17546#issuecomment-3293238849 > My understanding of the issue which I would like to clarify + document in the PR (maybe in the code) if it is correct is that this applies to a situation like: > > ``` >

Re: [PR] chore(deps): bump the arrow-parquet group with 7 updates [datafusion]

2025-09-15 Thread via GitHub
dependabot[bot] commented on PR #17434: URL: https://github.com/apache/datafusion/pull/17434#issuecomment-3293937304 Dependabot tried to update this pull request, but something went wrong. We're looking into it, but in the meantime you can retry the update by commenting `@dependabot rebase`

Re: [PR] feat: Support more date part expressions [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on code in PR #2316: URL: https://github.com/apache/datafusion-comet/pull/2316#discussion_r2350117883 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -1681,14 +1681,17 @@ class CometExpressionSuite extends CometTestBase with Ada

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
milenkovicm commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2350116417 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the com

Re: [PR] feat: feature specific tests [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on PR #2372: URL: https://github.com/apache/datafusion-comet/pull/2372#issuecomment-3293919594 > How about we replace it with `getCometFeatures` to get the features enabled by comet and caching them in the JVM? That way we can avoid multiple jni calls caused by multi

Re: [PR] Fix typo in error message in `substring.rs` [datafusion]

2025-09-15 Thread via GitHub
alamb merged PR #17570: URL: https://github.com/apache/datafusion/pull/17570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Typo in error message in substring [datafusion]

2025-09-15 Thread via GitHub
alamb closed issue #17569: Typo in error message in substring URL: https://github.com/apache/datafusion/issues/17569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Trying cargo machete to prune unused deps. [datafusion]

2025-09-15 Thread via GitHub
alamb merged PR #17545: URL: https://github.com/apache/datafusion/pull/17545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Trying cargo machete to prune unused deps. [datafusion]

2025-09-15 Thread via GitHub
alamb commented on PR #17545: URL: https://github.com/apache/datafusion/pull/17545#issuecomment-3293907144 > > Anything else you think we should do with this PR @samueleresca ? I marked it ready for review as it looked great > > Thanks 😄 I think we can proceed with this PR. I might op

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
mbutrovich commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349996536 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the comm

[PR] proto: don't include datafusion default features [datafusion]

2025-09-15 Thread via GitHub
jackkleeman opened a new pull request, #17577: URL: https://github.com/apache/datafusion/pull/17577 The default features from datafusion are pretty heavy and don't appear to be actually needed by the proto package -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] feat: rpad support column for second arg instead of just literal [datafusion-comet]

2025-09-15 Thread via GitHub
coderfender commented on PR #2099: URL: https://github.com/apache/datafusion-comet/pull/2099#issuecomment-3293775552 @comphead thank you for the approval . I did make changes to add UTF string test case and also remove unsafe unwrap method and handled error through match statement . Please

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
yjshen commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349960621 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the communit

Re: [PR] feat: Support comet native log level conf [datafusion-comet]

2025-09-15 Thread via GitHub
parthchandra commented on code in PR #2379: URL: https://github.com/apache/datafusion-comet/pull/2379#discussion_r2349832684 ## common/src/main/java/org/apache/comet/NativeBase.java: ## @@ -153,18 +154,25 @@ private static void bundleLoadLibrary() { private static void init

Re: [PR] Use taiki-e/install-action and binstall in CI [datafusion]

2025-09-15 Thread via GitHub
blaginin merged PR #17573: URL: https://github.com/apache/datafusion/pull/17573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Use taiki-e/install-action in CI [datafusion]

2025-09-15 Thread via GitHub
blaginin closed issue #17572: Use taiki-e/install-action in CI URL: https://github.com/apache/datafusion/issues/17572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] perf: Implement specialized aggregates for `COUNT(*)` and `COUNT(expr)` [datafusion-comet]

2025-09-15 Thread via GitHub
mbutrovich commented on PR #2397: URL: https://github.com/apache/datafusion-comet/pull/2397#issuecomment-3293674773 > * New `CometFuzzTestBase` extracted from `CometFuzzTestSuite` Oh I love this. > This PR fixes some old tech debt around the way we implemented `COUNT` aggregat

Re: [PR] fix: prevent UnionExec panic with empty inputs [datafusion]

2025-09-15 Thread via GitHub
findepi merged PR #17449: URL: https://github.com/apache/datafusion/pull/17449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
findepi commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349913385 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the communi

Re: [PR] Support Decimal32/64 types [datafusion]

2025-09-15 Thread via GitHub
AdamGS commented on PR #17501: URL: https://github.com/apache/datafusion/pull/17501#issuecomment-3291741873 Another issue I've just run into - casting support in arrow-rs was only released in 56.1, so this PR should probably only merge once that's available here. The [dependabot PR](https:

Re: [PR] fix: ignore non-existent columns when adding filter equivalence info in `FileScanConfig` [datafusion]

2025-09-15 Thread via GitHub
adriangb commented on PR #17546: URL: https://github.com/apache/datafusion/pull/17546#issuecomment-3292696154 > cc @adriangb I think this was inadvertently introduced in #17323 Sorry for the delay in reviewing this and sorry to the community if this doesn't make it into 50.0.0 and it

Re: [I] minor: `UnionExec` inputs validation [datafusion]

2025-09-15 Thread via GitHub
findepi closed issue #17052: minor: `UnionExec` inputs validation URL: https://github.com/apache/datafusion/issues/17052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Generalize struct-to-struct casting with CastOptions and SchemaAdapter integration [datafusion]

2025-09-15 Thread via GitHub
comphead commented on PR #17468: URL: https://github.com/apache/datafusion/pull/17468#issuecomment-3293572977 Created rollback PR https://github.com/apache/datafusion/pull/17576 in case of we cannot find a quick solution -- This is an automated message from the Apache Git Service. To resp

[PR] Revert "Generalize struct-to-struct casting with CastOptions and Sche… [datafusion]

2025-09-15 Thread via GitHub
comphead opened a new pull request, #17576: URL: https://github.com/apache/datafusion/pull/17576 …maAdapter integration (#17468)" This reverts commit e711f147e41fdb6c5899cd72bf2f4579011dac13. ## Which issue does this PR close? - Closes #. ## Rationale for t

Re: [I] Simplify `CASE WHEN true ...` [datafusion]

2025-09-15 Thread via GitHub
geoffreyclaude commented on issue #17448: URL: https://github.com/apache/datafusion/issues/17448#issuecomment-3293549174 Can the symmetric case: ```sql CASE WHEN false THEN x ELSE 1 END ``` be similarly simplified to just `1`, or is the first arm required to deduce the overall ty

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
adriangb commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349864247 ## docs/source/contributor-guide/governance.md: ## @@ -70,7 +70,7 @@ Notes: | Tim Saucer | timsaucer| [timsaucer](https://github.com/timsau

Re: [PR] Use taiki-e/install-action and binstall in CI [datafusion]

2025-09-15 Thread via GitHub
blaginin commented on PR #17573: URL: https://github.com/apache/datafusion/pull/17573#issuecomment-3293512177 Got it! I think it's working now (and actually uses less resources are not built), so going to merge if that's ok with everyone. I think we can iterate on top (as in adding `nextest

Re: [PR] Support Decimal32/64 types [datafusion]

2025-09-15 Thread via GitHub
AdamGS commented on PR #17501: URL: https://github.com/apache/datafusion/pull/17501#issuecomment-3291653091 One issue that is potentially breaking for backwards compatibility and is also an issue for slt tests, is when parsing SQL we use the precision and scale to infer the decimal size, if

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
adriangb commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349864247 ## docs/source/contributor-guide/governance.md: ## @@ -70,7 +70,7 @@ Notes: | Tim Saucer | timsaucer| [timsaucer](https://github.com/timsau

Re: [PR] Add support for ClickHouse CSE. [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
pravic commented on code in PR #2024: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2024#discussion_r2349843369 ## src/dialect/mod.rs: ## @@ -596,6 +596,20 @@ pub trait Dialect: Debug + Any { false } +/// Returns true if the dialect supports Co

Re: [PR] Add `TableProvider::scan_with_args` [datafusion]

2025-09-15 Thread via GitHub
adriangb merged PR #17336: URL: https://github.com/apache/datafusion/pull/17336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Generalize struct-to-struct casting with CastOptions and SchemaAdapter integration [datafusion]

2025-09-15 Thread via GitHub
adriangb commented on PR #17468: URL: https://github.com/apache/datafusion/pull/17468#issuecomment-3293507827 @kosiew looks like we broke main by merging this: https://github.com/apache/datafusion/actions/runs/17738016795/job/50405112093#step:7:7775 -- This is an automated message from th

[I] Potential performance regression with `parquet 56.1.0` / data ranges [datafusion]

2025-09-15 Thread via GitHub
alamb opened a new issue, #17575: URL: https://github.com/apache/datafusion/issues/17575 DataFusion 56.1.0 includes a new predicate cache - https://github.com/apache/arrow-rs/pull/7850 We tried hard to include a switch to disable the cache to prevent regressions, but apparently it

Re: [PR] WIP: Upgrade to arrow 56.1.0 [datafusion]

2025-09-15 Thread via GitHub
alamb commented on PR #17275: URL: https://github.com/apache/datafusion/pull/17275#issuecomment-3293468853 I think we should proceed to reviewing and merging this PR. Since @AdamGS was already looking at this and - https://github.com/apache/datafusion/pull/17571 - I will f

Re: [PR] Implemented the `From` method for all clear variants in Statement [datafusion-sqlparser-rs]

2025-09-15 Thread via GitHub
LucaCappelletti94 commented on code in PR #2028: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2028#discussion_r2348801228 ## src/ast/statement_from_impls.rs: ## Review Comment: I created the different module as the main one is ever more massive, so I though

Re: [PR] Upgrade arrow/parquet to 56.1.0 [datafusion]

2025-09-15 Thread via GitHub
alamb commented on PR #17571: URL: https://github.com/apache/datafusion/pull/17571#issuecomment-3293460234 > That's my bad 🤦 should probably have looked specifically at your PRs. I'll close this one. No worry -- it is actually my bad I should not have kept the other PR waiting so lon

Re: [PR] Upgrade arrow/parquet to 56.1.0 [datafusion]

2025-09-15 Thread via GitHub
AdamGS closed pull request #17571: Upgrade arrow/parquet to 56.1.0 URL: https://github.com/apache/datafusion/pull/17571 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] feat: Ensure explain format in config is valid [datafusion]

2025-09-15 Thread via GitHub
xudong963 commented on PR #17549: URL: https://github.com/apache/datafusion/pull/17549#issuecomment-3290776872 @Jefffrey Thank you for the quick improvement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Upgrade to protobuf 3.15 so that we can use `optional` keyword [datafusion]

2025-09-15 Thread via GitHub
Jefffrey commented on issue #8639: URL: https://github.com/apache/datafusion/issues/8639#issuecomment-3292497871 We already use `optional` keyword in proto definition now: https://github.com/apache/datafusion/blob/8b19ce83da1e89b1bd38626049135a0ca842694c/datafusion/proto/proto/datafus

Re: [I] Upgrade to protobuf 3.15 so that we can use `optional` keyword [datafusion]

2025-09-15 Thread via GitHub
Jefffrey closed issue #8639: Upgrade to protobuf 3.15 so that we can use `optional` keyword URL: https://github.com/apache/datafusion/issues/8639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
Weijun-H commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349752549 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
comphead commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349749758 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
comphead commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349738683 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
comphead commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349734408 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the commun

Re: [PR] Dynamic filters blog post (rev 2) [datafusion-site]

2025-09-15 Thread via GitHub
nuno-faria commented on PR #103: URL: https://github.com/apache/datafusion-site/pull/103#issuecomment-3293316385 > @nuno-faria what computer (or at least how many cores) did you run [#103 (comment)](https://github.com/apache/datafusion-site/pull/103#issuecomment-3262612288) on? @adri

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
sunchao commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349703923 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the communi

Re: [PR] Add explicit PMC/committers list to governance docs page [datafusion]

2025-09-15 Thread via GitHub
avantgardnerio commented on code in PR #17574: URL: https://github.com/apache/datafusion/pull/17574#discussion_r2349531650 ## docs/source/contributor-guide/governance.md: ## @@ -38,6 +34,84 @@ As much as practicable, we strive to make decisions by consensus, and anyone in the

Re: [PR] Relax constraint that file sort order must only reference individual columns [datafusion]

2025-09-15 Thread via GitHub
alamb commented on PR #17419: URL: https://github.com/apache/datafusion/pull/17419#issuecomment-3291502344 > Perhaps a two step approach would be better then where we try the “column only” version first and only use the more complex code path as fallback. This would have the very nice

Re: [PR] fix: ignore non-existent columns when adding filter equivalence info in `FileScanConfig` [datafusion]

2025-09-15 Thread via GitHub
rkrishn7 commented on PR #17546: URL: https://github.com/apache/datafusion/pull/17546#issuecomment-3293240717 > > cc @adriangb I think this was inadvertently introduced in #17323 > > Sorry for the delay in reviewing this and sorry to the community if this doesn't make it into 50.0.0 a

Re: [PR] fix: Check reused broadcast plan in non-AQE and make setNumPartitions thread safe [datafusion-comet]

2025-09-15 Thread via GitHub
wForget commented on PR #2398: URL: https://github.com/apache/datafusion-comet/pull/2398#issuecomment-3291734147 It seems difficult to create an appropriate test case for this change, but according to the feedback from @and124578963 https://github.com/apache/datafusion-comet/issues/2389#is

  1   2   >