Re: [I] CometExecIterator uses incorrect partition index [datafusion-comet]

2024-11-24 Thread via GitHub
viirya closed issue #1113: CometExecIterator uses incorrect partition index URL: https://github.com/apache/datafusion-comet/issues/1113 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] fix: Use RDD partition index [datafusion-comet]

2024-11-24 Thread via GitHub
viirya commented on PR #1112: URL: https://github.com/apache/datafusion-comet/pull/1112#issuecomment-2497041835 Thanks @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] fix: Use RDD partition index [datafusion-comet]

2024-11-24 Thread via GitHub
viirya merged PR #1112: URL: https://github.com/apache/datafusion-comet/pull/1112 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

[PR] [minor]: Update median implementation [datafusion]

2024-11-24 Thread via GitHub
akurmustafa opened a new pull request, #13554: URL: https://github.com/apache/datafusion/pull/13554 ## Which issue does this PR close? Improves the situation in [13550](https://github.com/apache/datafusion/issues/13550) ## Rationale for this change While examining th

Re: [PR] POC: Fusing repart and partial aggr [datafusion]

2024-11-24 Thread via GitHub
Rachelint commented on PR #12526: URL: https://github.com/apache/datafusion/pull/12526#issuecomment-2496875061 Will push it forward in next few days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] POC: Remove unnecessary null check for `GroupColumn` [datafusion]

2024-11-24 Thread via GitHub
Rachelint closed pull request #12947: POC: Remove unnecessary null check for `GroupColumn` URL: https://github.com/apache/datafusion/pull/12947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] POC: Remove unnecessary null check for `GroupColumn` [datafusion]

2024-11-24 Thread via GitHub
Rachelint commented on PR #12947: URL: https://github.com/apache/datafusion/pull/12947#issuecomment-2496871517 It is poc version of #12996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-11-24 Thread via GitHub
Rachelint closed pull request #11943: Sketch for aggregation intermediate results blocked management URL: https://github.com/apache/datafusion/pull/11943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-11-24 Thread via GitHub
Rachelint commented on PR #11943: URL: https://github.com/apache/datafusion/pull/11943#issuecomment-2496856615 Codes here are stale, I will submit a new pr when starting to push it forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [bug]: Fix wrong order by removal from plan [datafusion]

2024-11-24 Thread via GitHub
akurmustafa commented on PR #13497: URL: https://github.com/apache/datafusion/pull/13497#issuecomment-2496781966 > and result of those are valid. One thing that just could be noted `VALUES ('name1'), ('name2')` and `VALUES ('name1'), ('name1')` are both regarded as not constant and not

Re: [PR] Handle alias when parsing sql(parse_sql_expr) [datafusion]

2024-11-24 Thread via GitHub
Eason0729 commented on PR #12939: URL: https://github.com/apache/datafusion/pull/12939#issuecomment-2496607441 > Thanks @Eason0729 ! > > I looked at #12518 > > Could you possible add a test showing the usecase from that issue working? > > For example a test that does some

Re: [PR] feat: Wrap `TableScan` with `Filter` in Join Unparsing [datafusion]

2024-11-24 Thread via GitHub
jonathanc-n commented on PR #13496: URL: https://github.com/apache/datafusion/pull/13496#issuecomment-2496557539 @jayzhan211 I agree, I'll try to wrap it in a subquery, i'll mark it as a draft for now. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Enhance the nested type access for Generic and DuckDB dialect [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
goldmedal commented on code in PR #1541: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1541#discussion_r1855696643 ## src/parser/mod.rs: ## @@ -2935,12 +2935,23 @@ impl<'a> Parser<'a> { }) } else if Token::LBracket == tok { if di

Re: [PR] Enhance the nested type access for Generic and DuckDB dialect [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
goldmedal commented on code in PR #1541: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1541#discussion_r1855696643 ## src/parser/mod.rs: ## @@ -2935,12 +2935,23 @@ impl<'a> Parser<'a> { }) } else if Token::LBracket == tok { if di

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-11-24 Thread via GitHub
github-actions[bot] commented on PR #11943: URL: https://github.com/apache/datafusion/pull/11943#issuecomment-2496525505 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] feat: Wrap `TableScan` with `Filter` in Join Unparsing [datafusion]

2024-11-24 Thread via GitHub
goldmedal commented on PR #13496: URL: https://github.com/apache/datafusion/pull/13496#issuecomment-2496518022 > @jonathanc-n, @goldmedal - thank you, I've reviewed this change and it seems it brings back the following issue (there is additional context of why filtering added this way produ

Re: [I] [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) [datafusion]

2024-11-24 Thread via GitHub
andygrove commented on issue #13525: URL: https://github.com/apache/datafusion/issues/13525#issuecomment-2496487538 > I have felt the pain points of the version upgrades. I mentioned this in the discord server, but one thing I think we can do that will lessen this is to push hard to get the

Re: [PR] fix: Remove dangling table references in `unparser` [datafusion]

2024-11-24 Thread via GitHub
phillipleblanc commented on PR #13405: URL: https://github.com/apache/datafusion/pull/13405#issuecomment-2496456491 > Maybe @sgrebnov or @phillipleblanc can offer some advice about how to proceed with this PR -- I feel like it adds significantly complexity and I am not sure it doesn't also

Re: [PR] fix: Remove dangling table references in `unparser` [datafusion]

2024-11-24 Thread via GitHub
peasee closed pull request #13405: fix: Remove dangling table references in `unparser` URL: https://github.com/apache/datafusion/pull/13405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Refactor regexplike signature [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on PR #13394: URL: https://github.com/apache/datafusion/pull/13394#issuecomment-2496447845 I think the CI failures are a type casting issue. There are two rows in the example that have `4000` as the values that I think are not being coerced to strings as they should be.

Re: [I] Simplify expressions swallows a cast expression [datafusion]

2024-11-24 Thread via GitHub
timsaucer closed issue #13481: Simplify expressions swallows a cast expression URL: https://github.com/apache/datafusion/issues/13481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Preserve field name when casting List [datafusion]

2024-11-24 Thread via GitHub
timsaucer merged PR #13468: URL: https://github.com/apache/datafusion/pull/13468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] Support custom field metadata in UDF [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on PR #13458: URL: https://github.com/apache/datafusion/pull/13458#issuecomment-2496436510 I think a test would be in order that would showcase why the new metadata method exists and what problem it solves. -- This is an automated message from the Apache Git Service. To

Re: [PR] Support custom field metadata in UDF [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on code in PR #13458: URL: https://github.com/apache/datafusion/pull/13458#discussion_r1855634897 ## datafusion/expr/src/udf.rs: ## @@ -477,6 +487,15 @@ pub trait ScalarUDFImpl: Debug + Send + Sync { true } +/// Returns the field metadata f

Re: [PR] chore: Create devcontainer.json [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on PR #13520: URL: https://github.com/apache/datafusion/pull/13520#issuecomment-2496343488 FYI, here is the devcontainer I personally use for all my DataFusion development: https://github.com/Omega359/arrow-datafusion/tree/feature/devcontainer/.devcontainer -- This is

Re: [I] Add `greatest(T,...)` and `least(T,...)` SQL functions [datafusion]

2024-11-24 Thread via GitHub
rluvaton commented on issue #6531: URL: https://github.com/apache/datafusion/issues/6531#issuecomment-2496321591 I'll try to add the least function in the coming week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] feat: Add ConfigOptions to ScalarFunctionArgs [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13527: URL: https://github.com/apache/datafusion/pull/13527#issuecomment-2496304460 I plan to review this carefully tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Add `greatest(T,...)` and `least(T,...)` SQL functions [datafusion]

2024-11-24 Thread via GitHub
alamb commented on issue #6531: URL: https://github.com/apache/datafusion/issues/6531#issuecomment-2496304090 Upate is that @rluvaton added support for greatest in - https://github.com/apache/datafusion/pull/12474 Perhaps we should file a separate ticket for `least` 🤔 -- T

Re: [PR] Preserve field name when casting List [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13468: URL: https://github.com/apache/datafusion/pull/13468#issuecomment-2496303274 Thank you very much @timsaucer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Preserve field name when casting List [datafusion]

2024-11-24 Thread via GitHub
alamb commented on code in PR #13468: URL: https://github.com/apache/datafusion/pull/13468#discussion_r1855566618 ## datafusion/common/src/utils/mod.rs: ## @@ -342,6 +342,20 @@ pub fn array_into_list_array(arr: ArrayRef, nullable: bool) -> ListArray { ) } +pub fn array_

Re: [PR] docs: add sql examples for window functions [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13551: URL: https://github.com/apache/datafusion/pull/13551#issuecomment-2496301097 Thank you @spencerscott917 🙏 Since the user documentation is generated, I think you will have to add these examples to get the CI to pass Here is an example of how to do

Re: [PR] chore: Remove redundant assert_starts_with test helper [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13542: URL: https://github.com/apache/datafusion/pull/13542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Support LIKE with ESCAPE `\` [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13312: URL: https://github.com/apache/datafusion/pull/13312#issuecomment-2496293704 Thanks again @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Support LIKE with ESCAPE `\` [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13312: URL: https://github.com/apache/datafusion/pull/13312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] test: allow external_access_plan run on windows [datafusion]

2024-11-24 Thread via GitHub
alamb commented on code in PR #13531: URL: https://github.com/apache/datafusion/pull/13531#discussion_r1855561110 ## datafusion/core/tests/parquet/external_access_plan.rs: ## @@ -369,45 +379,43 @@ struct TestData { file_size: u64, } -static TEST_DATA: OnceLock = OnceLock

Re: [PR] test: allow external_access_plan run on windows [datafusion]

2024-11-24 Thread via GitHub
alamb commented on code in PR #13531: URL: https://github.com/apache/datafusion/pull/13531#discussion_r1855561330 ## datafusion/core/tests/parquet/external_access_plan.rs: ## @@ -369,45 +379,43 @@ struct TestData { file_size: u64, } -static TEST_DATA: OnceLock = OnceLock

Re: [PR] Support Unnest in Subqueries [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13523: URL: https://github.com/apache/datafusion/pull/13523 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Set timezone for group column timestamp type [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13535: URL: https://github.com/apache/datafusion/pull/13535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Subquery check_internal_plan does not support LogicalPlan::Unnest [datafusion]

2024-11-24 Thread via GitHub
alamb closed issue #13498: Subquery check_internal_plan does not support LogicalPlan::Unnest URL: https://github.com/apache/datafusion/issues/13498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Set timezone for group column timestamp type [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13535: URL: https://github.com/apache/datafusion/pull/13535#issuecomment-2496289384 Thanks again @jayzhan211 and @adriangb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Simple query fails with `column types must match schema types` [datafusion]

2024-11-24 Thread via GitHub
alamb closed issue #13534: Simple query fails with `column types must match schema types` URL: https://github.com/apache/datafusion/issues/13534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] chore: add version to deprecation messages [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13543: URL: https://github.com/apache/datafusion/pull/13543 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Minor: Fix two test cases causing CI failures by adding `rowsort` [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13547: URL: https://github.com/apache/datafusion/pull/13547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Fix panic when hashing empty FixedSizeList Array [datafusion]

2024-11-24 Thread via GitHub
alamb merged PR #13533: URL: https://github.com/apache/datafusion/pull/13533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Move many udf implementations from `invoke` to `invoke_batch` [datafusion]

2024-11-24 Thread via GitHub
alamb commented on PR #13491: URL: https://github.com/apache/datafusion/pull/13491#issuecomment-2496257949 Here is what I suggest we do with this PR: 1. Remove the change for https://github.com/apache/datafusion/pull/13507 (let's do that in a separate PR) 2. Fix up the regressions intr

Re: [PR] Add support for MSSQL's `JSON_ARRAY`/`JSON_OBJECT` expr [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa commented on PR #1507: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1507#issuecomment-2496237908 I opened a new PR to enable this syntax in postgres, which supports it too: https://github.com/apache/datafusion-sqlparser-rs/pull/1546 https://www.postgresql.org/d

Re: [PR] `json_object('k' VALUE 'v')` in postgres [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa commented on PR #1547: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1547#issuecomment-2496235833 The new test will fail until https://github.com/apache/datafusion-sqlparser-rs/pull/1546 is merged -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Support snowflake double dot notation for object name [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
ayman-sigma commented on code in PR #1540: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1540#discussion_r1855531538 ## src/parser/mod.rs: ## @@ -8349,6 +8349,13 @@ impl<'a> Parser<'a> { pub fn parse_object_name(&mut self, in_table_clause: bool) -> Result {

Re: [PR] support `json_object('k':'v')` in postgres [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa commented on PR #1546: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1546#issuecomment-2496222357 I moved the corresponding tests from `tests/sqlparser_mssql.rs` to `tests/sqlparser_common.rs` -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Support snowflake double dot notation for object name [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
ayman-sigma commented on code in PR #1540: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1540#discussion_r1855531079 ## tests/sqlparser_snowflake.rs: ## @@ -2846,3 +2846,32 @@ fn test_parse_show_columns_sql() { snowflake().verified_stmt("SHOW COLUMNS IN TABLE

Re: [PR] Support snowflake double dot notation for object name [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
ayman-sigma commented on code in PR #1540: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1540#discussion_r1855531149 ## tests/sqlparser_snowflake.rs: ## @@ -2846,3 +2846,32 @@ fn test_parse_show_columns_sql() { snowflake().verified_stmt("SHOW COLUMNS IN TABLE

[PR] support `json_object('k':'v')` in postgres [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa opened a new pull request, #1546: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1546 This just updates the postgres dialect definition to add support for the json_object syntax added in https://github.com/apache/datafusion-sqlparser-rs/pull/1507 The syntax was i

Re: [PR] Include license file in sqlparser_derive crate [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
ankane commented on PR #1543: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1543#issuecomment-2496220674 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] User documentation: Generate docs from macros, make`DocumentationBuilder::build` infallable [datafusion]

2024-11-24 Thread via GitHub
comphead commented on PR #12822: URL: https://github.com/apache/datafusion/pull/12822#issuecomment-2496217932 Filed https://github.com/apache/datafusion/issues/13553 to add missing parts -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] User documentation: Generate docs from macros, make`DocumentationBuilder::build` infallable [datafusion]

2024-11-24 Thread via GitHub
comphead merged PR #12822: URL: https://github.com/apache/datafusion/pull/12822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

[I] support postgres `json_object` syntax [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa opened a new issue, #1545: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1545 Postgresql supports the following syntax for the json_object function: ```sql json_object('label' VALUE 'value') json_object('label': 'value') ``` Reference: https://ww

Re: [I] support for mssql `json_object` syntax [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa closed issue #1506: support for mssql `json_object` syntax URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] support for mssql `json_object` syntax [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
lovasoa commented on issue #1506: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1506#issuecomment-2496199367 Fixed by @gaoqiangz . Thank you ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[PR] chore: dependancy updates [datafusion-ballista]

2024-11-24 Thread via GitHub
milenkovicm opened a new pull request, #1131: URL: https://github.com/apache/datafusion-ballista/pull/1131 # Which issue does this PR close? Closes #. # Rationale for this change # What changes are included in this PR? # Are there any user-facing c

[I] [Proposal] String function data type handling requirements [datafusion]

2024-11-24 Thread via GitHub
Omega359 opened a new issue, #13552: URL: https://github.com/apache/datafusion/issues/13552 ### Is your feature request related to a problem or challenge? One of the things I've been thinking about when working on utf8view support in udfs is what exactly datafusion should support in t

Re: [I] Filters on `RANDOM()` are applied incorrectly when pushdown_filters is enabled. [datafusion]

2024-11-24 Thread via GitHub
theirix commented on issue #13268: URL: https://github.com/apache/datafusion/issues/13268#issuecomment-2496157453 This issue is fixed by avoiding pushing down volatile filters, so the sampling is achieved by a manual `random() < 0.1` expression. @findebi, regarding the sampling suppor

[PR] docs: add sql examples for window functions [datafusion]

2024-11-24 Thread via GitHub
spencerscott917 opened a new pull request, #13551: URL: https://github.com/apache/datafusion/pull/13551 ## Which issue does this PR close? Closes [#13399](https://github.com/apache/datafusion/issues/13399) ## Rationale for this change Give users examples of how to use window func

Re: [PR] User documentation: Generate docs from macros, make`DocumentationBuilder::build` infallable [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on PR #12822: URL: https://github.com/apache/datafusion/pull/12822#issuecomment-2496121234 I'm fine with that @comphead - we can file followup issues to add the alternative syntax, etc -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Minor: Exclude all DDL statements from Ray scheduling [datafusion-ray]

2024-11-24 Thread via GitHub
ccciudatu commented on PR #42: URL: https://github.com/apache/datafusion-ray/pull/42#issuecomment-2496117721 @edmondop any clue on what the bug was? Is it the "index out of bounds" caused by DDL statements other than crete/drop view? -- This is an automated message from the Apache Git Ser

Re: [PR] User documentation: Generate docs from macros, make`DocumentationBuilder::build` infallable [datafusion]

2024-11-24 Thread via GitHub
comphead commented on PR #12822: URL: https://github.com/apache/datafusion/pull/12822#issuecomment-2496113115 @Omega359 I'm planning to merge it today, getting more code conflicts, let me know if and objections -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] chore: Make list.rs non generic & simplify the code [datafusion-comet]

2024-11-24 Thread via GitHub
SemyonSinchenko commented on PR #1118: URL: https://github.com/apache/datafusion-comet/pull/1118#issuecomment-2496110565 @andygrove As you mentioned in #1073, it would be interesting to remove all logic related to `LargeList` and check if there is any regression. I did just that and all te

[PR] docs: Add more technical detail and new diagram to Comet plugin overview [datafusion-comet]

2024-11-24 Thread via GitHub
andygrove opened a new pull request, #1119: URL: https://github.com/apache/datafusion-comet/pull/1119 ## Which issue does this PR close? N/A ## Rationale for this change Now that I have a better understanding of the architecture, I am updating the documen

Re: [PR] fix: Various metrics bug fixes and improvements [datafusion-comet]

2024-11-24 Thread via GitHub
codecov-commenter commented on PR #: URL: https://github.com/apache/datafusion-comet/pull/#issuecomment-2496096131 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] [DISCUSSION] More SqlLogicTest test coverage for queries, including join queries [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on issue #13470: URL: https://github.com/apache/datafusion/issues/13470#issuecomment-2496089503 I have been looking over the sqllogictests this weekend and I think first of all that they need a good reorganization to clean them up, split them into smaller chunks testing j

Re: [I] [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) [datafusion]

2024-11-24 Thread via GitHub
Omega359 commented on issue #13525: URL: https://github.com/apache/datafusion/issues/13525#issuecomment-2496085474 On Dependency Management I have a suggestion that I think could help but would take actual and ongoing work on the DF communities' part. I would suggest that we file tick

Re: [PR] fix: Various metrics bug fixes and improvements [datafusion-comet]

2024-11-24 Thread via GitHub
andygrove commented on code in PR #: URL: https://github.com/apache/datafusion-comet/pull/#discussion_r1855475759 ## native/core/src/execution/operators/scan.rs: ## @@ -365,28 +378,23 @@ struct ScanStream<'a> { scan: ScanExec, /// Schema representing the data

Re: [I] Improve performance of `corr` function [datafusion]

2024-11-24 Thread via GitHub
2010YOUY01 commented on issue #13549: URL: https://github.com/apache/datafusion/issues/13549#issuecomment-2496052419 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Improve performance of `median` function [datafusion]

2024-11-24 Thread via GitHub
Rachelint commented on issue #13550: URL: https://github.com/apache/datafusion/issues/13550#issuecomment-2496050412 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] chore: Make list.rs non generic & simplify the code [datafusion-comet]

2024-11-24 Thread via GitHub
SemyonSinchenko opened a new pull request, #1118: URL: https://github.com/apache/datafusion-comet/pull/1118 ## Which issue does this PR close? Closes #1114 ## Rationale for this change Spark supports only i32 indixed arrays but Comet attempts to support both ListArray (

[I] Improve performance of `corr` function [datafusion]

2024-11-24 Thread via GitHub
alamb opened a new issue, #13549: URL: https://github.com/apache/datafusion/issues/13549 ### Is your feature request related to a problem or challenge? The `corr` function is used in the h20 benchmark and is quite slow. https://github.com/apache/datafusion/blob/main/datafusion

[I] [EPIC] Improved aggregate function performance [datafusion]

2024-11-24 Thread via GitHub
alamb opened a new issue, #13548: URL: https://github.com/apache/datafusion/issues/13548 ### Is your feature request related to a problem or challenge? The basic aggregate functions like `COUNT` and `SUM` in DataFusion are *very* fast (see [Apache DataFusion is now the fastest single

Re: [PR] refactor: SessionStateExt and SessionConfigExt [datafusion-ballista]

2024-11-24 Thread via GitHub
andygrove merged PR #1130: URL: https://github.com/apache/datafusion-ballista/pull/1130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] refactor: consolidate ballista tests [datafusion-ballista]

2024-11-24 Thread via GitHub
andygrove merged PR #1129: URL: https://github.com/apache/datafusion-ballista/pull/1129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] refactor: BallistaLogicalExtensionCodec refactoring and improvements [datafusion-ballista]

2024-11-24 Thread via GitHub
andygrove merged PR #1127: URL: https://github.com/apache/datafusion-ballista/pull/1127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

[PR] simplify introductory online docs [datafusion-python]

2024-11-24 Thread via GitHub
timsaucer opened a new pull request, #955: URL: https://github.com/apache/datafusion-python/pull/955 # Which issue does this PR close? None # Rationale for this change Based on user feedback linked in https://github.com/apache/datafusion/pull/13468 this is a change to t

[PR] Minor: Fix two test cases causing CI failures [datafusion]

2024-11-24 Thread via GitHub
2010YOUY01 opened a new pull request, #13547: URL: https://github.com/apache/datafusion/pull/13547 ## Which issue does this PR close? Closes #. ## Rationale for this change I encountered a CI failure in https://github.com/apache/datafusion/actions/runs/119956

[PR] POC: Update to sqlparser with span (source text) location [datafusion]

2024-11-24 Thread via GitHub
alamb opened a new pull request, #13546: URL: https://github.com/apache/datafusion/pull/13546 ## Which issue does this PR close? Related to https://github.com/apache/datafusion-sqlparser-rs/pull/1435 ## Rationale for this change This PR demonstrates what is needed to update D

Re: [PR] Add `#[recursive]` [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb commented on PR #1522: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1522#issuecomment-2495967142 I think this PR needs a bit more documentation and we shoudl figure out how to rationalize with the existing `recursion_limit` argument. https://docs.rs/sqlpars

Re: [PR] Include license file in sqlparser_derive crate [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb commented on PR #1543: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1543#issuecomment-2495966670 Thanks @ankane and @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Include license file in sqlparser_derive crate [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb merged PR #1543: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1543 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Include license file in sqlparser_derive crate [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb commented on PR #1543: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1543#issuecomment-2495966628 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add `#[recursive]` [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb commented on PR #1522: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1522#issuecomment-2495966342 > Ah I see, thanks for the context @peter-toth! > > cc @alamb for overall thoughts on adding this dependency to sqlparser? While adding new dependencies in gene

Re: [PR] Implement `Spanned` to retrieve source locations on AST nodes [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
alamb commented on PR #1435: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1435#issuecomment-2495964382 I started looking over this PR -- and it is looking great. I took the liberty of merging up from main and fixing clippy/fmt. I am going to test out the implications o

Re: [PR] [bug]: Fix wrong order by removal from plan [datafusion]

2024-11-24 Thread via GitHub
berkaysynnada merged PR #13497: URL: https://github.com/apache/datafusion/pull/13497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Order by is ignored [datafusion]

2024-11-24 Thread via GitHub
berkaysynnada closed issue #13483: Order by is ignored URL: https://github.com/apache/datafusion/issues/13483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Add example for using a separate threadpool for CPU bound work [datafusion]

2024-11-24 Thread via GitHub
berkaysynnada commented on code in PR #13424: URL: https://github.com/apache/datafusion/pull/13424#discussion_r1855421673 ## datafusion-examples/examples/thread_pools.rs: ## @@ -0,0 +1,213 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Implement `Spanned` to retrieve source locations on AST nodes [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
iffyio commented on code in PR #1435: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1435#discussion_r1855413273 ## tests/sqlparser_postgres.rs: ## @@ -2507,6 +2520,7 @@ fn parse_array_subquery_expr() { op: SetOperator::Union,

Re: [PR] Fix MySQL parsing of GRANT, REVOKE, and CREATE VIEW [datafusion-sqlparser-rs]

2024-11-24 Thread via GitHub
iffyio commented on code in PR #1538: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1538#discussion_r1855409232 ## src/parser/mod.rs: ## @@ -4693,9 +4699,59 @@ impl<'a> Parser<'a> { if_not_exists, temporary, to, +

[I] SQL/PGQ or even GQL support [datafusion]

2024-11-24 Thread via GitHub
gsvgit opened a new issue, #13545: URL: https://github.com/apache/datafusion/issues/13545 ### Is your feature request related to a problem or challenge? SQL (standard) was recently extended with property graph querying features (PGQ): [ISO standard](https://www.iso.org/standard/79473.

[PR] refactor: SessionStateExt and SessionConfigExt [datafusion-ballista]

2024-11-24 Thread via GitHub
milenkovicm opened a new pull request, #1130: URL: https://github.com/apache/datafusion-ballista/pull/1130 # Which issue does this PR close? Closes none. # Rationale for this change separation of concerns of internal modules and reducing public api removing methods not