[PR] Consolidate Example: simplify_udwf_expression.rs into advanced_udwf.rs [datafusion]

2024-12-22 Thread via GitHub
xarus01 opened a new pull request, #13883: URL: https://github.com/apache/datafusion/pull/13883 ## Which issue does this PR close? Closes #13843. ## Rationale for this change Part of #11172 ## What changes are included in this PR? * C

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
goldmedal commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895372613 ## datafusion/sql/tests/cases/plan_to_sql.rs: ## @@ -1406,3 +1414,150 @@ fn test_join_with_no_conditions() { "SELECT * FROM j1 CROSS JOIN j2", );

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
goldmedal commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895368410 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a cust

Re: [PR] feat: Add a `spark.comet.exec.memoryPool` configuration for experimenting with various datafusion memory pool setups. [datafusion-comet]

2024-12-22 Thread via GitHub
Kontinuation commented on PR #1021: URL: https://github.com/apache/datafusion-comet/pull/1021#issuecomment-2559075333 > Just to add some context here, we have discovered a need to support existing deployments that are not using offheap memory, so would like to add a non-offheap option back

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
goldmedal commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895366799 ## datafusion/sql/src/unparser/plan.rs: ## @@ -121,6 +124,40 @@ impl Unparser<'_> { } } +/// Try to unparse a [UserDefinedLogicalNode] to a S

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
goldmedal commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895362378 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a cust

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
zhuqi-lucas commented on PR #13874: URL: https://github.com/apache/datafusion/pull/13874#issuecomment-2558886583 Updated the code to also include the write_table testing case which is the same with insert into sql. -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Cannot create a `List` of `FixedSizedList` in SQL [datafusion]

2024-12-22 Thread via GitHub
alan910127 commented on issue #13819: URL: https://github.com/apache/datafusion/issues/13819#issuecomment-2558830043 @alamb Sorry for the late response. Based on your discussion, I think making your example work with the behavior aligning with DuckDB (i.e. don't cast back to `FixedSizedList

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
phillipleblanc commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895160201 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a

Re: [PR] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-22 Thread via GitHub
zhuqi-lucas commented on PR #13877: URL: https://github.com/apache/datafusion/pull/13877#issuecomment-2558747940 Thank you @alamb for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
zhuqi-lucas commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1895145562 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan

Re: [I] Improve efficiency of CI checks (so we can add MORE!) [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13845: URL: https://github.com/apache/datafusion/issues/13845#issuecomment-2558611440 > Maintaining an extended workflow shouldn't be too bad tbh. I think having a workflow that runs outside of PR's (iow runs nightly) could be useful as well for expensive tests tha

Re: [I] Why does `PruningPredicate` reference a `row_count` for each column? [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13836: URL: https://github.com/apache/datafusion/issues/13836#issuecomment-2558610663 > Even if the stats have a row count per column, surely we can pick one, rename it to `row_count` and discard the rest? Would you be open to that change? I agree -- this mak

Re: [PR] Merge composite and compound expr test cases [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
alamb commented on PR #1615: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1615#issuecomment-2558610944 ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Merge composite and compound expr test cases [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
alamb merged PR #1615: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Restore `DocBuilder::new()` to avoid breaking API change [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13870: URL: https://github.com/apache/datafusion/pull/13870#issuecomment-2558610452 Thank you @comphead and @Omega359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Restore `DocBuilder::new()` to avoid breaking API change [datafusion]

2024-12-22 Thread via GitHub
alamb merged PR #13870: URL: https://github.com/apache/datafusion/pull/13870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Release DataFusion `44.0.0` [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13334: URL: https://github.com/apache/datafusion/issues/13334#issuecomment-2558608736 > @alamb The miri issue does not seem to be resolved for Comet - [#13835 (comment)](https://github.com/apache/datafusion/issues/13835#issuecomment-2558543030) > > Is there a

Re: [PR] Support unparsing implicit lateral `UNNEST` plan to SQL text [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on PR #13824: URL: https://github.com/apache/datafusion/pull/13824#issuecomment-2558607823 Looks good 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895049631 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a custo

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895049631 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a custo

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on PR #13880: URL: https://github.com/apache/datafusion/pull/13880#issuecomment-2558593517 @goldmedal - looks great, two minor questions/comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895052716 ## datafusion/sql/src/unparser/plan.rs: ## @@ -121,6 +124,40 @@ impl Unparser<'_> { } } +/// Try to unparse a [UserDefinedLogicalNode] to a SQ

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
Dandandan commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1895051122 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan =

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
Dandandan commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1894960842 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan =

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
Dandandan commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1895051122 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan =

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895049631 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a custo

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
sgrebnov commented on code in PR #13880: URL: https://github.com/apache/datafusion/pull/13880#discussion_r1895049631 ## datafusion/sql/src/unparser/mod.rs: ## @@ -105,13 +109,33 @@ impl<'a> Unparser<'a> { self.pretty = pretty; self } + +/// Add a custo

Re: [PR] doc-gen: migrate builtin scalar functions documentation to attribute based [datafusion]

2024-12-22 Thread via GitHub
comphead commented on PR #13878: URL: https://github.com/apache/datafusion/pull/13878#issuecomment-2558589372 Great work 💪 thanks @Chen-Yuan-Lai -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] feat: Move shuffle block decompression and decoding to native code [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove commented on PR #1192: URL: https://github.com/apache/datafusion-comet/pull/1192#issuecomment-2558579492 The status is that this is working, but I am running into some executor OOMs when trying to run complete benchmarks. I will pick this up again after the holidays. -- This i

Re: [I] Release DataFusion `44.0.0` [datafusion]

2024-12-22 Thread via GitHub
andygrove commented on issue #13334: URL: https://github.com/apache/datafusion/issues/13334#issuecomment-2558576916 @alamb The miri issue does not seem to be resolved for Comet - https://github.com/apache/datafusion/issues/13835#issuecomment-2558543030 Is there anything I should have

Re: [I] Why does `PruningPredicate` reference a `row_count` for each column? [datafusion]

2024-12-22 Thread via GitHub
adriangb commented on issue #13836: URL: https://github.com/apache/datafusion/issues/13836#issuecomment-2558574774 Even if the stats have a row count per column, surely we can pick one, rename it to `row_count` and discard the rest? Would you be open to that change? -- This is an automate

Re: [PR] minor: refactor to move decodeBatches to broadcast exchange code as private function [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove merged PR #1195: URL: https://github.com/apache/datafusion-comet/pull/1195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] minor: refactor prepare_output so that it does not require an ExecutionContext [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove merged PR #1194: URL: https://github.com/apache/datafusion-comet/pull/1194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Test DataFusion 44.0.0 with Comet [datafusion]

2024-12-22 Thread via GitHub
andygrove commented on issue #13835: URL: https://github.com/apache/datafusion/issues/13835#issuecomment-2558542900 Reopening ... I upgraded to latest and disabled default features, enabled our miri checks again but they are still failing. -- This is an automated message from the Apache G

Re: [I] Test DataFusion 44.0.0 with Comet [datafusion]

2024-12-22 Thread via GitHub
andygrove commented on issue #13835: URL: https://github.com/apache/datafusion/issues/13835#issuecomment-2558543030 ``` error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5

Re: [I] Complete / integrate sqlite sqllogictest test scripts integrattion [datafusion]

2024-12-22 Thread via GitHub
Omega359 commented on issue #13812: URL: https://github.com/apache/datafusion/issues/13812#issuecomment-2558528827 Thank you very much @alamb. I'll work on the PR later today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Complete / integrate sqlite sqllogictest test scripts integrattion [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13812: URL: https://github.com/apache/datafusion/issues/13812#issuecomment-2558520181 I did some poking around and found on this JIRA: https://issues.apache.org/jira/browse/INFRA-26314?jql=project%20%3D%20INFRA%20AND%20summary%20~%20%22create%20repo%22%20ORDER%20BY%

Re: [PR] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-22 Thread via GitHub
alamb merged PR #13877: URL: https://github.com/apache/datafusion/pull/13877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-22 Thread via GitHub
alamb closed issue #13844: Consolidate Example: dataframe_output.rs into dataframe.rs URL: https://github.com/apache/datafusion/issues/13844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13844: URL: https://github.com/apache/datafusion/issues/13844#issuecomment-2558518458 Thank you @zhuqi-lucas 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[PR] Prepare for 44.0.0 release: version and changelog [datafusion]

2024-12-22 Thread via GitHub
alamb opened a new pull request, #13882: URL: https://github.com/apache/datafusion/pull/13882 ## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/13334 ## Rationale for this change Prepare for the release ## What changes are

Re: [I] Why does `PruningPredicate` reference a `row_count` for each column? [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13836: URL: https://github.com/apache/datafusion/issues/13836#issuecomment-2558511204 I agree I can't think of any reason for each column to have a separate row count -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Test DataFusion 44.0.0 with Comet [datafusion]

2024-12-22 Thread via GitHub
alamb closed issue #13835: Test DataFusion 44.0.0 with Comet URL: https://github.com/apache/datafusion/issues/13835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] Test DataFusion 44.0.0 with Comet [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13835: URL: https://github.com/apache/datafusion/issues/13835#issuecomment-2558510406 Testing looks good from what I can tell ✅ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Test DataFusion 44.0.0 with delta.rs [datafusion]

2024-12-22 Thread via GitHub
alamb closed issue #13834: Test DataFusion 44.0.0 with delta.rs URL: https://github.com/apache/datafusion/issues/13834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [I] Test DataFusion 44.0.0 with delta.rs [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13834: URL: https://github.com/apache/datafusion/issues/13834#issuecomment-2558510316 Testing is complete and looks good ✅ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Handle empty projection in Postgres SELECT statements [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
alamb commented on PR #1613: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1613#issuecomment-2558507966 Thanks @tobyhede and @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Handle empty projection in Postgres SELECT statements [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
alamb merged PR #1613: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1613 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [I] Datafusion binary size has been getting bigger [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13816: URL: https://github.com/apache/datafusion/issues/13816#issuecomment-2558507731 That is a very cool page 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Replace `BufferBuilder` with `Vec` [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13867: URL: https://github.com/apache/datafusion/issues/13867#issuecomment-2558507063 Thanks @jayzhan211 and @poweihuang0817 -- sounds like a good change Maybe we can do it as a few PRs (for example, start with `ArrowBytesMap`) -- that will be faster to rev

Re: [I] [Epic] A Collection of Additional UTF8View support tickets [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13504: URL: https://github.com/apache/datafusion/issues/13504#issuecomment-2558506198 Update here is that many of these remaining issues seem to be waiting on an upgrade to arrow-rs. Hopefully we'll get a fix out in the next week or two -- This is an automated me

Re: [PR] Feature scalar regexp match benchmark [datafusion]

2024-12-22 Thread via GitHub
Omega359 commented on PR #13789: URL: https://github.com/apache/datafusion/pull/13789#issuecomment-2558505140 I ran your benchmark and it looks good. I am unsure though the benefit of this benchmark over the existing 'regx' benchmark which targets the udf versions of the Postgresql symbols

Re: [I] Regression: `Invalid comparison operation: Utf8 == Utf8View` error during LEFT ANTI JOIN [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13510: URL: https://github.com/apache/datafusion/issues/13510#issuecomment-2558505287 Summary: - This issue can be worked around by setting the `datafusion.execution.parquet.schema_force_view_types` config to false The error comes from arrow-rs (sourc

Re: [I] Regression in 43.0.0: coalesce no longer works between Utf8 and Utf8View columns [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13568: URL: https://github.com/apache/datafusion/issues/13568#issuecomment-2558504029 I verified that the reproducer query now runs without error on main (will be release in DataFusion 44). Thank you again for the report @ttencate -- This is an automated

Re: [I] Regression in 43.0.0: coalesce no longer works between Utf8 and Utf8View columns [datafusion]

2024-12-22 Thread via GitHub
alamb closed issue #13568: Regression in 43.0.0: coalesce no longer works between Utf8 and Utf8View columns URL: https://github.com/apache/datafusion/issues/13568 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Add documentation for `SHOW FUNCTIONS` [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13868: URL: https://github.com/apache/datafusion/pull/13868#issuecomment-2558502767 Thank you @comphead and @goldmedal 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
alamb commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1894978630 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan = if

Re: [I] Release DataFusion `44.0.0` [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13334: URL: https://github.com/apache/datafusion/issues/13334#issuecomment-2558499462 I am looking at https://github.com/apache/datafusion/issues/13510 now, and then plan to make a PR with release notes, verison udate, etc -- This is an automated message from the

Re: [I] Release DataFusion `44.0.0` [datafusion]

2024-12-22 Thread via GitHub
alamb commented on issue #13334: URL: https://github.com/apache/datafusion/issues/13334#issuecomment-2558498557 Status update here: ## Ease upgrades: Summary: From my perspective the following issues are nice to have, rather than required for the release - [ ] https://github.com

Re: [PR] Support Snowflake Update-From-Select [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
yuval-illumex commented on PR #1604: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1604#issuecomment-2558493934 @iffyio appreciate your feedback again 🙏 (Tests that are failing are not related to my PR): https://github.com/apache/datafusion-sqlparser-rs/actions/runs/12

Re: [PR] Support Snowflake Update-From-Select [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
yuval-illumex commented on code in PR #1604: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1604#discussion_r1894970422 ## src/parser/mod.rs: ## @@ -11679,14 +11679,21 @@ impl<'a> Parser<'a> { pub fn parse_update(&mut self) -> Result { let or = self.p

Re: [PR] Merge composite and compound expr test cases [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio commented on PR #1615: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1615#issuecomment-2558493281 cc @alamb FYI @ayman-sigma @goldmedal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] Merge composite and compound expr test cases [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio opened a new pull request, #1615: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1615 Fixes test failure on main due to conflicting representations of compound expressions after landing from #1600 and #1551 -- This is an automated message from the Apache Git Service. T

Re: [PR] minor: refactor to move decodeBatches to broacast exchange code as private function [datafusion-comet]

2024-12-22 Thread via GitHub
codecov-commenter commented on PR #1195: URL: https://github.com/apache/datafusion-comet/pull/1195#issuecomment-2558488944 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1195?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]

2024-12-22 Thread via GitHub
lichuang commented on issue #13821: URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2558485724 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
Dandandan commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1894960842 ## datafusion/core/src/dataframe/parquet.rs: ## @@ -74,8 +74,16 @@ impl DataFrame { let file_type = format_as_file_type(format); +let plan =

Re: [PR] Consolidate `MapAccess`, and `Subscript` into `CompoundExpr` to handle the complex field access chain [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
goldmedal commented on PR #1551: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1551#issuecomment-2558481001 Thanks @iffyio and @alamb 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-22 Thread via GitHub
alamb commented on code in PR #13874: URL: https://github.com/apache/datafusion/pull/13874#discussion_r1894948500 ## datafusion/core/src/dataframe/mod.rs: ## @@ -4057,4 +4094,178 @@ mod tests { Ok(()) } + +// Test issue: https://github.com/apache/datafusion/i

Re: [PR] chore: Upgrade to DataFusion 44.0.0 [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove commented on PR #1154: URL: https://github.com/apache/datafusion-comet/pull/1154#issuecomment-2558475306 I have upgraded to latest DF and reinstated the miri checks. I'll keep this in draft until the RC is available later this week. -- This is an automated message from the Apac

Re: [PR] Consolidate `MapAccess`, and `Subscript` into `CompoundExpr` to handle the complex field access chain [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio commented on PR #1551: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1551#issuecomment-2558475789 Thanks for working on this @goldmedal! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] doc-gen: migrate builtin scalar functions documentation to attribute based [datafusion]

2024-12-22 Thread via GitHub
Omega359 commented on PR #13878: URL: https://github.com/apache/datafusion/pull/13878#issuecomment-2558475486 I can help with reviewing the smaller PR's once they are submitted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Consolidate `MapAccess`, and `Subscript` into `CompoundExpr` to handle the complex field access chain [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio merged PR #1551: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Support Snowflake Update-From-Select [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
yuval-illumex commented on code in PR #1604: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1604#discussion_r1894950935 ## src/parser/mod.rs: ## @@ -11679,14 +11679,21 @@ impl<'a> Parser<'a> { pub fn parse_update(&mut self) -> Result { let or = self.p

Re: [I] Complete / integrate sqlite sqllogictest test scripts integrattion [datafusion]

2024-12-22 Thread via GitHub
Omega359 commented on issue #13812: URL: https://github.com/apache/datafusion/issues/13812#issuecomment-2558473921 What are the steps required to have a `datafusion-testing`created? I went hunting quickly through the apache docs I could find but I couldn't find the steps required -- This

Re: [PR] Minor: improve error message when ARRAY literals can not be planned [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13859: URL: https://github.com/apache/datafusion/pull/13859#issuecomment-2558473941 Thank you @comphead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Support 1 or 3 arg in generate_series() UDTF [datafusion]

2024-12-22 Thread via GitHub
UBarney commented on PR #13856: URL: https://github.com/apache/datafusion/pull/13856#issuecomment-2558473205 > Thanks @UBarney. Overall looks good to me but some minor comments. cc @2010YOUY01 Thanks @goldmedal for reviewing. All comments have been addressed. Please take another look

Re: [PR] Restore `DocBuilder::new()` to avoid breaking API change [datafusion]

2024-12-22 Thread via GitHub
alamb commented on code in PR #13870: URL: https://github.com/apache/datafusion/pull/13870#discussion_r1894949740 ## datafusion/sql/src/statement.rs: ## @@ -62,8 +62,8 @@ use sqlparser::ast::{ Assignment, AssignmentTarget, ColumnDef, CreateIndex, CreateTable, CreateTab

Re: [PR] Restore `DocBuilder::new()` to avoid breaking API change [datafusion]

2024-12-22 Thread via GitHub
Omega359 commented on code in PR #13870: URL: https://github.com/apache/datafusion/pull/13870#discussion_r1894949432 ## datafusion/sql/src/statement.rs: ## @@ -62,8 +62,8 @@ use sqlparser::ast::{ Assignment, AssignmentTarget, ColumnDef, CreateIndex, CreateTable, Create

Re: [PR] Support arbitrary composite access expressions [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio merged PR #1600: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Support Snowflake Update-From-Select [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
iffyio commented on code in PR #1604: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1604#discussion_r1894948705 ## src/parser/mod.rs: ## @@ -11679,14 +11679,21 @@ impl<'a> Parser<'a> { pub fn parse_update(&mut self) -> Result { let or = self.parse_co

Re: [PR] minor: fixs typos in comments / structure names [datafusion]

2024-12-22 Thread via GitHub
zhuliquan commented on code in PR #13879: URL: https://github.com/apache/datafusion/pull/13879#discussion_r1894946948 ## datafusion-examples/examples/advanced_parquet_index.rs: ## @@ -211,7 +211,7 @@ async fn main() -> Result<()> { // // Note: in order to prune pages,

[PR] minor: refactor to move decodeBatches to broacast exchange code as private function [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove opened a new pull request, #1195: URL: https://github.com/apache/datafusion-comet/pull/1195 ## Which issue does this PR close? N/A ## Rationale for this change This is a small refactor extracted from https://github.com/apache/datafusion-comet/pu

Re: [PR] Consolidate `MapAccess`, and `Subscript` into `CompoundExpr` to handle the complex field access chain [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
goldmedal commented on PR #1551: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1551#issuecomment-2558469234 > Given the potential for non trivial downstream conflicts due to this change (look at the list of conflicts it has already collected) I would like to merge it in right

Re: [PR] doc-gen: migrate builtin scalar functions documentation to attribute based [datafusion]

2024-12-22 Thread via GitHub
Chen-Yuan-Lai commented on PR #13878: URL: https://github.com/apache/datafusion/pull/13878#issuecomment-2558468286 @alamb Sure! I'll break it down into smaller PRs. Thank you for the suggestion. -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] minor: refactor prepare_output so that it does not require an ExecutionContext [datafusion-comet]

2024-12-22 Thread via GitHub
andygrove opened a new pull request, #1194: URL: https://github.com/apache/datafusion-comet/pull/1194 ## Which issue does this PR close? N/A ## Rationale for this change This refactor is extracted from https://github.com/apache/datafusion-comet/pull/1192.

Re: [PR] Minor: remove unused async-compression `futures-io` feature [datafusion]

2024-12-22 Thread via GitHub
alamb merged PR #13875: URL: https://github.com/apache/datafusion/pull/13875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Improve error messages for incorrect zero argument signatures [datafusion]

2024-12-22 Thread via GitHub
alamb commented on code in PR #13881: URL: https://github.com/apache/datafusion/pull/13881#discussion_r1894938593 ## datafusion/expr/src/type_coercion/functions.rs: ## @@ -50,17 +50,23 @@ pub fn data_types_with_scalar_udf( func: &ScalarUDF, ) -> Result> { let signatur

Re: [PR] Require all zero argument UDFs use `Signature::Nullary`, improve error messages [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13871: URL: https://github.com/apache/datafusion/pull/13871#issuecomment-2558462047 Here is a PR with just improvements to the error messages - https://github.com/apache/datafusion/pull/13881 -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Restore `DocBuilder::new()` to avoid breaking API change [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13870: URL: https://github.com/apache/datafusion/pull/13870#issuecomment-2558461372 @Omega359 or @comphead I wonder if you have time to review this change prior to the 44 release? -- This is an automated message from the Apache Git Service. To respond to the messag

[PR] Improve error messages for incorrect zero argument signatures [datafusion]

2024-12-22 Thread via GitHub
alamb opened a new pull request, #13881: URL: https://github.com/apache/datafusion/pull/13881 ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/13763 ## Rationale for this change As described on https://github.com/apache/datafusio

Re: [PR] minor: fix typo error in datafusion [datafusion]

2024-12-22 Thread via GitHub
zhuliquan commented on code in PR #13879: URL: https://github.com/apache/datafusion/pull/13879#discussion_r1894933257 ## datafusion-cli/src/main.rs: ## @@ -209,7 +209,7 @@ async fn main_inner() -> Result<()> { if !rc.is_empty() { exec::exec_from_files(&ctx,

Re: [PR] Require all zero argument UDFs use `Signature::Nullary`, improve error messages [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13871: URL: https://github.com/apache/datafusion/pull/13871#issuecomment-2558455848 Given there are a few more built in functions with exact, I think we should postpone this PR until next release (aka postpone merge for a few days) In the intervening time I will

Re: [PR] doc-gen: migrate builtin scalar functions documentation to attribute based [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13878: URL: https://github.com/apache/datafusion/pull/13878#issuecomment-2558454069 Thank you @Chen-Yuan-Lai -- this looks epic! Given the size of this PR I suspect it will be hard to find time to review and also hard to keep up to date (it will likely accumula

Re: [PR] minor: fix typo error in datafusion [datafusion]

2024-12-22 Thread via GitHub
alamb commented on code in PR #13879: URL: https://github.com/apache/datafusion/pull/13879#discussion_r1894927649 ## datafusion/expr-common/src/accumulator.rs: ## @@ -115,7 +115,7 @@ pub trait Accumulator: Send + Sync + Debug { /// │

Re: [PR] [minor] make recursive package dependency optional [datafusion]

2024-12-22 Thread via GitHub
alamb commented on PR #13778: URL: https://github.com/apache/datafusion/pull/13778#issuecomment-2558448612 Thanks again @buraksenn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Making the `recursive` dependency an optional feature [datafusion]

2024-12-22 Thread via GitHub
alamb closed issue #13766: Making the `recursive` dependency an optional feature URL: https://github.com/apache/datafusion/issues/13766 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] [minor] make recursive package dependency optional [datafusion]

2024-12-22 Thread via GitHub
alamb merged PR #13778: URL: https://github.com/apache/datafusion/pull/13778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Introduce `UserDefinedLogicalNodeUnparser` for User-defined Logical Plan unparsing [datafusion]

2024-12-22 Thread via GitHub
goldmedal commented on PR #13880: URL: https://github.com/apache/datafusion/pull/13880#issuecomment-2558438474 Could @phillipleblanc or @sgrebnov take a look at this? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Support Snowflake Update-From-Select [datafusion-sqlparser-rs]

2024-12-22 Thread via GitHub
yuval-illumex commented on PR #1604: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1604#issuecomment-2558434174 Thank you @iffyio for your feedback, I asked your opinion in one of the comments. -- This is an automated message from the Apache Git Service. To respond to the m

  1   2   >