Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on PR #11943: URL: https://github.com/apache/datafusion/pull/11943#issuecomment-2287994535 > THank you @Rachelint -- I took a look at this PR and here is some feedback: > > 1. I think it is important to spend time actually showing this approach makes some queri

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1716358223 ## datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/bool_op.rs: ## @@ -68,11 +70,21 @@ where fn update_batch( Review Comment: Yes,

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1716358223 ## datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/bool_op.rs: ## @@ -68,11 +70,21 @@ where fn update_batch( Review Comment: Yes,

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1716358223 ## datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/bool_op.rs: ## @@ -68,11 +70,21 @@ where fn update_batch( Review Comment: Yes,

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1716358223 ## datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/bool_op.rs: ## @@ -68,11 +70,21 @@ where fn update_batch( Review Comment: Yes,

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1716358223 ## datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/bool_op.rs: ## @@ -68,11 +70,21 @@ where fn update_batch( Review Comment: Yes,

[PR] Add native stringview support for LEFT [datafusion]

2024-08-13 Thread via GitHub
Kev1n8 opened a new pull request, #11976: URL: https://github.com/apache/datafusion/pull/11976 ## Which issue does this PR close? Issue similar to #11917. Instead of `RIGHT`, this PR adds string view support for `LEFT`, whose issue has not yet been filed. ## Rationale f

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
Rachelint commented on PR #11943: URL: https://github.com/apache/datafusion/pull/11943#issuecomment-2287897129 @2010YOUY01 make sense, it seems `emit_early_if_necessary` function is actually introduced in the spilling pr #7400. I am checking the related codes about memory control, too.

Re: [PR] Update SPLIT_PART scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
Lordworms commented on PR #11975: URL: https://github.com/apache/datafusion/pull/11975#issuecomment-2287824264 an unrelated ci fail -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] Update SPLIT_PART scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
Lordworms opened a new pull request, #11975: URL: https://github.com/apache/datafusion/pull/11975 ## Which issue does this PR close? Closes #11950 ## Rationale for this change ## What changes are included in this PR? ## Are these changes te

Re: [PR] perf: Pass statistics for completed query stages to native planner [datafusion-comet]

2024-08-13 Thread via GitHub
codecov-commenter commented on PR #827: URL: https://github.com/apache/datafusion-comet/pull/827#issuecomment-2287805086 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/827?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campai

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
2010YOUY01 commented on PR #11943: URL: https://github.com/apache/datafusion/pull/11943#issuecomment-2287796117 > > I think I'm not so familiar with the Emit::First and there is no block implementation done yet. Could we emit every block size of values we have? Something like Emit::First(bl

[PR] Implement native support StringView for substr_index [datafusion]

2024-08-13 Thread via GitHub
PsiACE opened a new pull request, #11974: URL: https://github.com/apache/datafusion/pull/11974 ## Which issue does this PR close? Closes #. ## Rationale for this change Part of https://github.com/apache/datafusion/issues/11752 and https://github.com/apach

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1716253312 ## datafusion/functions/src/unicode/translate.rs: ## @@ -120,7 +144,7 @@ fn translate(args: &[ArrayRef]) -> Result { } _ => None,

Re: [PR] Support `extract` on intervals [datafusion]

2024-08-13 Thread via GitHub
nrc commented on code in PR #11501: URL: https://github.com/apache/datafusion/pull/11501#discussion_r1716238810 ## datafusion/functions/src/datetime/date_part.rs: ## @@ -81,6 +82,9 @@ impl DatePartFunc { Exact(vec![Utf8, Time32(Millisecond)]),

Re: [PR] Implement Utf8View for lpad scalar function [datafusion]

2024-08-13 Thread via GitHub
Omega359 commented on PR #11941: URL: https://github.com/apache/datafusion/pull/11941#issuecomment-2287720721 > Let me know if you would like me to file a ticket for > > > I was looking at @Lordworms implementation in #11942 and I think it would make sense to align the two implementat

Re: [PR] fix: Optimize not to call getNullCount as much as possible [datafusion-comet]

2024-08-13 Thread via GitHub
kazuyukitanimura commented on PR #820: URL: https://github.com/apache/datafusion-comet/pull/820#issuecomment-2287719370 @andygrove @comphead @huaxingao @parthchandra @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Support `extract` on intervals [datafusion]

2024-08-13 Thread via GitHub
nrc commented on code in PR #11501: URL: https://github.com/apache/datafusion/pull/11501#discussion_r1716231772 ## datafusion/functions/src/datetime/date_part.rs: ## @@ -81,6 +82,9 @@ impl DatePartFunc { Exact(vec![Utf8, Time32(Millisecond)]),

Re: [PR] fix: move coercion of union from builder to `TypeCoercion` [datafusion]

2024-08-13 Thread via GitHub
jonahgao commented on code in PR #11961: URL: https://github.com/apache/datafusion/pull/11961#discussion_r1716228329 ## datafusion/sqllogictest/test_files/type_coercion.slt: ## @@ -49,3 +49,179 @@ select interval '1 month' - '2023-05-01'::date; # interval - timestamp query err

[PR] Update REVERSE scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
Omega359 opened a new pull request, #11973: URL: https://github.com/apache/datafusion/pull/11973 ## Which issue does this PR close? Closes #11915 ## Rationale for this change Update reverse udf to support utf8view ## What changes are included in this PR? cod

Re: [PR] Support Null aware anti join by HashJoin [datafusion]

2024-08-13 Thread via GitHub
github-actions[bot] commented on PR #10584: URL: https://github.com/apache/datafusion/pull/10584#issuecomment-2287682781 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Move wildcard expansions to the analyzer [datafusion]

2024-08-13 Thread via GitHub
goldmedal commented on PR #11681: URL: https://github.com/apache/datafusion/pull/11681#issuecomment-2287674491 Thanks @alamb @jayzhan211 ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Generic FlightTableFactory with a default FlightSqlDriver [datafusion]

2024-08-13 Thread via GitHub
ccciudatu commented on PR #11938: URL: https://github.com/apache/datafusion/pull/11938#issuecomment-2287605620 I added the proto marshalling and marked the PR "ready for review", once again. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Implement native support StringView for find in set [datafusion]

2024-08-13 Thread via GitHub
PsiACE commented on code in PR #11970: URL: https://github.com/apache/datafusion/pull/11970#discussion_r1716172792 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -913,7 +913,7 @@ EXPLAIN SELECT FROM test; logical_plan -01)Projection: find_in_set(CAST(test.c

Re: [PR] fix: Optimize not to call getNullCount as much as possible [datafusion-comet]

2024-08-13 Thread via GitHub
codecov-commenter commented on PR #820: URL: https://github.com/apache/datafusion-comet/pull/820#issuecomment-2287583533 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/820?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campai

Re: [PR] Implement native support StringView for overlay [datafusion]

2024-08-13 Thread via GitHub
PsiACE commented on code in PR #11968: URL: https://github.com/apache/datafusion/pull/11968#discussion_r1716166644 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -726,7 +726,7 @@ EXPLAIN SELECT FROM test; logical_plan -01)Projection: overlay(CAST(test.colum

Re: [PR] chore: Add more unsupported cases to supportedSortType [datafusion-comet]

2024-08-13 Thread via GitHub
viirya commented on PR #825: URL: https://github.com/apache/datafusion-comet/pull/825#issuecomment-2287572528 Thanks @andygrove @huaxingao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] chore: Add more unsupported cases to supportedSortType [datafusion-comet]

2024-08-13 Thread via GitHub
viirya merged PR #825: URL: https://github.com/apache/datafusion-comet/pull/825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Debug dump of MemoryManager [datafusion]

2024-08-13 Thread via GitHub
wiedld commented on issue #6934: URL: https://github.com/apache/datafusion/issues/6934#issuecomment-2287548529 Is the goal of this ticket to improve the OOM messages? Or to provide an API to get the top memory consumers (in the absence of an error)? We have updated the OOM message to

Re: [PR] Use tracked-consumers memory pool be the default. [datafusion]

2024-08-13 Thread via GitHub
wiedld commented on PR #11949: URL: https://github.com/apache/datafusion/pull/11949#issuecomment-2287510246 > lgtm thanks @wiedld I think it is good for now, perhaps in future if there are lots of memory consumers then the error message might be unreadbale Absolutely agreed. I set the

Re: [PR] Implement native support StringView for find in set [datafusion]

2024-08-13 Thread via GitHub
PsiACE commented on code in PR #11970: URL: https://github.com/apache/datafusion/pull/11970#discussion_r1716133464 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -913,7 +913,7 @@ EXPLAIN SELECT FROM test; logical_plan -01)Projection: find_in_set(CAST(test.c

Re: [PR] feat: `CreateArray` support [datafusion-comet]

2024-08-13 Thread via GitHub
Kimahriman commented on PR #793: URL: https://github.com/apache/datafusion-comet/pull/793#issuecomment-2287437461 > Thanks for the contribution @Kimahriman. I plan on reviewing this in the next day or two. Thanks, definitely interested in your thoughts on the nullability issue. --

Re: [PR] Use tracked-consumers memory pool be the default. [datafusion]

2024-08-13 Thread via GitHub
wiedld commented on code in PR #11949: URL: https://github.com/apache/datafusion/pull/11949#discussion_r1716115035 ## datafusion/core/tests/memory_limit/mod.rs: ## @@ -76,8 +76,7 @@ async fn group_by_none() { TestCase::new() .with_query("select median(request_bytes

Re: [PR] feat: Implement to_json for subset of types [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on PR #805: URL: https://github.com/apache/datafusion-comet/pull/805#issuecomment-2287399027 Thanks for the review @eejbyfeldt! It is really appreciated. Some very good feedback there. I will address the feedback over the next day or two. -- This is an automated messag

[I] Expose `between` [datafusion-python]

2024-08-13 Thread via GitHub
timsaucer opened a new issue, #809: URL: https://github.com/apache/datafusion-python/issues/809 **Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Expose `between` function. **Describe the solution you'd like** *

Re: [PR] perf: Pass statistics for completed query stages to native planner [datafusion-comet]

2024-08-13 Thread via GitHub
parthchandra commented on code in PR #827: URL: https://github.com/apache/datafusion-comet/pull/827#discussion_r1716102690 ## native/proto/src/proto/operator.proto: ## @@ -46,12 +46,19 @@ message Operator { } } +message Statistics { Review Comment: Nice! For refere

[PR] perf: Pass statistics for completed query stages to native planner [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove opened a new pull request, #827: URL: https://github.com/apache/datafusion-comet/pull/827 ## Which issue does this PR close? N/A ## Rationale for this change This PR was inspired by a random conversation with @parthchandra earlier today. I have

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1715895675 ## datafusion/functions/src/unicode/translate.rs: ## @@ -120,7 +144,7 @@ fn translate(args: &[ArrayRef]) -> Result { } _ => None,

Re: [PR] fix: Optimize not to call getNullCount as much as possible [datafusion-comet]

2024-08-13 Thread via GitHub
kazuyukitanimura commented on PR #820: URL: https://github.com/apache/datafusion-comet/pull/820#issuecomment-2287244799 ### Before ![Screenshot 2024-08-13 at 3 22 53  PM](https://github.com/user-attachments/assets/eb769a30-dcd0-4f0c-85ac-5be31b830a60) ### After ![Screenshot 2024-

Re: [PR] feat: Implement to_json for subset of types [datafusion-comet]

2024-08-13 Thread via GitHub
eejbyfeldt commented on code in PR #805: URL: https://github.com/apache/datafusion-comet/pull/805#discussion_r1715844690 ## docs/source/user-guide/expressions.md: ## @@ -182,6 +182,14 @@ The following Spark expressions are currently available. Any known compatibility | Varianc

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1715895675 ## datafusion/functions/src/unicode/translate.rs: ## @@ -120,7 +144,7 @@ fn translate(args: &[ArrayRef]) -> Result { } _ => None,

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1715896137 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -425,6 +425,19 @@ logical_plan 01)Projection: starts_with(test.column1_utf8view, Utf8View("äöüß"))

Re: [I] Test CometDriverPlugin in Kube/Yarn [datafusion-comet]

2024-08-13 Thread via GitHub
comphead commented on issue #826: URL: https://github.com/apache/datafusion-comet/issues/826#issuecomment-2287086195 Related to https://github.com/apache/datafusion-comet/issues/605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] [EPIC] Improved support for nested / structured types (`Struct` , `List`, `ListArray`, and other Composite types) [datafusion]

2024-08-13 Thread via GitHub
TheBuilderJR commented on issue #2326: URL: https://github.com/apache/datafusion/issues/2326#issuecomment-2287085675 Right now datafusion doesn't support struct evolution very well. Imagine you have a struct named `customData` with field `someOptionEnabled` in one parquet file, later down t

[I] Test CometDriverPlugin in Kube/Yarn [datafusion-comet]

2024-08-13 Thread via GitHub
comphead opened a new issue, #826: URL: https://github.com/apache/datafusion-comet/issues/826 In Comet there is a Spark driver plugin that overrides `spark.executor.memoryOverhead` value to respect native part needed for the Comet. Resource managers like YARN/Kube considers `spark.e

Re: [I] Access a Map with a non-string keys [datafusion]

2024-08-13 Thread via GitHub
dharanad commented on issue #11785: URL: https://github.com/apache/datafusion/issues/11785#issuecomment-2287082683 > I think `map_extract` function can benefit this issue #11969 I will prioritise this issue -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] doc: Add memory tuning section to user guide [datafusion-comet]

2024-08-13 Thread via GitHub
comphead commented on PR #684: URL: https://github.com/apache/datafusion-comet/pull/684#issuecomment-2287067051 > @comphead Hm, interesting, I tried launching the container with and without specifying `spark.comet.memoryOverhead` and YARN reported the same amount of memory allocated to my a

Re: [PR] doc: Add memory tuning section to user guide [datafusion-comet]

2024-08-13 Thread via GitHub
orthoxerox commented on PR #684: URL: https://github.com/apache/datafusion-comet/pull/684#issuecomment-2287042699 @comphead Hm, interesting, I tried launching the container with and without specifying `spark.comet.memoryOverhead` and YARN reported the same amount of memory allocated to my a

Re: [PR] chore: Add more unsupported cases to supportedSortType [datafusion-comet]

2024-08-13 Thread via GitHub
codecov-commenter commented on PR #825: URL: https://github.com/apache/datafusion-comet/pull/825#issuecomment-2287035187 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/825?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campai

Re: [PR] Refactor `CoalesceBatches` to use an explicit state machine [datafusion]

2024-08-13 Thread via GitHub
ozankabak commented on code in PR #11966: URL: https://github.com/apache/datafusion/pull/11966#discussion_r1715852445 ## datafusion/physical-plan/src/coalesce_batches.rs: ## @@ -364,90 +419,73 @@ impl BatchCoalescer { Arc::clone(&self.schema) } -/// Add a bat

Re: [PR] Refactor `CoalesceBatches` to use an explicit state machine [datafusion]

2024-08-13 Thread via GitHub
ozankabak commented on code in PR #11966: URL: https://github.com/apache/datafusion/pull/11966#discussion_r1715846903 ## datafusion/physical-plan/src/coalesce_batches.rs: ## @@ -364,90 +419,73 @@ impl BatchCoalescer { Arc::clone(&self.schema) } -/// Add a bat

Re: [PR] Sketch for aggregation intermediate results blocked management [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11943: URL: https://github.com/apache/datafusion/pull/11943#discussion_r1715835534 ## datafusion/expr-common/src/groups_accumulator.rs: ## @@ -123,7 +151,7 @@ pub trait GroupsAccumulator: Send { /// future use. The group_indices on subsequent

Re: [PR] Update RPAD scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
Lordworms commented on code in PR #11942: URL: https://github.com/apache/datafusion/pull/11942#discussion_r1715830931 ## datafusion/functions/src/unicode/rpad.rs: ## @@ -76,97 +84,168 @@ impl ScalarUDFImpl for RPadFunc { } fn invoke(&self, args: &[ColumnarValue]) ->

Re: [I] Update the `CONCAT` scalar function to support `Utf8View` [datafusion]

2024-08-13 Thread via GitHub
alamb commented on issue #11836: URL: https://github.com/apache/datafusion/issues/11836#issuecomment-2286990432 There are a bunch of PRs now that port functions to Utf8View -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Update RPAD scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11942: URL: https://github.com/apache/datafusion/pull/11942#issuecomment-2286989094 Thank you @Lordworms and @Omega359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Update RPAD scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11942: URL: https://github.com/apache/datafusion/pull/11942#discussion_r1715826438 ## datafusion/functions/src/unicode/rpad.rs: ## @@ -76,97 +84,168 @@ impl ScalarUDFImpl for RPadFunc { } fn invoke(&self, args: &[ColumnarValue]) -> Resu

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on PR #11967: URL: https://github.com/apache/datafusion/pull/11967#issuecomment-2286988535 @alamb I still need to implement the test case but the cast error should be gone now. I tried it out in the CLI (oops should have done that before! 😅) thanks for taking a look :)

Re: [PR] Add native stringview support for RIGHT [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11955: URL: https://github.com/apache/datafusion/pull/11955#discussion_r1715825177 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -809,16 +809,28 @@ logical_plan 03)TableScan: test projection=[column1_utf8view] ## Ensure no ca

Re: [PR] Add native stringview support for RIGHT [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11955: URL: https://github.com/apache/datafusion/pull/11955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Update `RIGHT` scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11917: Update `RIGHT` scalar function to support Utf8View URL: https://github.com/apache/datafusion/issues/11917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] feat: `CreateArray` support [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on PR #793: URL: https://github.com/apache/datafusion-comet/pull/793#issuecomment-2286985032 Thanks for the contribution @Kimahriman. I plan on reviewing this in the next day or two. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] [EPIC] Fully support ANSI mode [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on issue #313: URL: https://github.com/apache/datafusion-comet/issues/313#issuecomment-2286983899 Hi @dharanad. All of the issues here are important for Spark 4.0 support. Feel free to pick up any issues that are not already being worked on -- This is an automated mes

Re: [PR] fix: move coercion of union from builder to `TypeCoercion` [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11961: URL: https://github.com/apache/datafusion/pull/11961#discussion_r1715818843 ## datafusion/sqllogictest/test_files/type_coercion.slt: ## @@ -49,3 +49,179 @@ select interval '1 month' - '2023-05-01'::date; # interval - timestamp query error

Re: [PR] experimental: Add physical optimizer [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on code in PR #791: URL: https://github.com/apache/datafusion-comet/pull/791#discussion_r1715817136 ## native/core/src/execution/datafusion/planner.rs: ## @@ -137,18 +138,6 @@ pub struct PhysicalPlanner { session_ctx: Arc, } -impl Default for Physical

Re: [PR] perf: Remove some redundant copying of batches [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove merged PR #816: URL: https://github.com/apache/datafusion-comet/pull/816 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [PR] perf: Remove some redundant copying of batches [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on PR #816: URL: https://github.com/apache/datafusion-comet/pull/816#issuecomment-2286968279 Thanks for the review @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Implement native support StringView for overlay [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11968: URL: https://github.com/apache/datafusion/pull/11968#discussion_r1715810357 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -726,7 +726,7 @@ EXPLAIN SELECT FROM test; logical_plan -01)Projection: overlay(CAST(test.column

Re: [PR] Register get_field by default [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11959: URL: https://github.com/apache/datafusion/pull/11959#discussion_r1715808722 ## datafusion/functions/src/core/mod.rs: ## @@ -94,6 +94,7 @@ pub fn functions() -> Vec> { nvl2(), arrow_typeof(), named_struct(), +

Re: [PR] Register get_field by default [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11959: URL: https://github.com/apache/datafusion/pull/11959#issuecomment-2286958889 This came up in the ASF slack channel too: https://the-asf.slack.com/archives/C04RJ0C85UZ/p1723558402071579 Thus that is two users so I think this is a good idea to add -- Thi

Re: [PR] Use `filtered_null_mask` in `CountGroupsAccumulator ` and `PrimitiveGroupsAccumulator` [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11825: URL: https://github.com/apache/datafusion/pull/11825#issuecomment-2286949621 I ran some benchmarks which suggested this could be slowing queries down. I don't understand how that could be possible but I need to review it more carefully -- This is an automate

Re: [PR] Support Arrays for the Map scalar functions [datafusion]

2024-08-13 Thread via GitHub
dharanad commented on PR #11712: URL: https://github.com/apache/datafusion/pull/11712#issuecomment-2286939947 Thank You @alamb @goldmedal @jayzhan211 @Weijun-H . I deepened my understanding on Arrow and had fun working on this issue. -- This is an automated message from the Apache Git

[PR] chore: Add more unsupported cases to supportedSortType [datafusion-comet]

2024-08-13 Thread via GitHub
viirya opened a new pull request, #825: URL: https://github.com/apache/datafusion-comet/pull/825 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes test

Re: [PR] Implement native support StringView for `REPEAT` [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11962: URL: https://github.com/apache/datafusion/pull/11962#discussion_r1715778889 ## datafusion/functions/src/string/repeat.rs: ## @@ -87,18 +95,35 @@ fn repeat(args: &[ArrayRef]) -> Result { let result = string_array .iter()

Re: [PR] perf: Remove some redundant copying of batches [datafusion-comet]

2024-08-13 Thread via GitHub
viirya commented on code in PR #816: URL: https://github.com/apache/datafusion-comet/pull/816#discussion_r1715771045 ## native/core/src/execution/datafusion/planner.rs: ## @@ -1775,10 +1780,14 @@ impl From for DataFusionError { /// modification. This is used to determine if we

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
devanbenz commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1715761804 ## datafusion/functions/src/unicode/translate.rs: ## @@ -72,6 +75,7 @@ impl ScalarUDFImpl for TranslateFunc { fn invoke(&self, args: &[ColumnarValue]) ->

Re: [PR] perf: Remove some redundant copying of batches [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on code in PR #816: URL: https://github.com/apache/datafusion-comet/pull/816#discussion_r1715757338 ## native/core/src/execution/datafusion/planner.rs: ## @@ -1775,10 +1780,14 @@ impl From for DataFusionError { /// modification. This is used to determine if

Re: [PR] feat/11953: Support StringView for TRANSLATE() fn [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11967: URL: https://github.com/apache/datafusion/pull/11967#discussion_r1715752731 ## datafusion/functions/src/unicode/translate.rs: ## @@ -72,6 +75,7 @@ impl ScalarUDFImpl for TranslateFunc { fn invoke(&self, args: &[ColumnarValue]) -> Resu

Re: [PR] Implement native support StringView for find in set [datafusion]

2024-08-13 Thread via GitHub
alamb commented on code in PR #11970: URL: https://github.com/apache/datafusion/pull/11970#discussion_r1715747722 ## datafusion/sqllogictest/test_files/string_view.slt: ## @@ -913,7 +913,7 @@ EXPLAIN SELECT FROM test; logical_plan -01)Projection: find_in_set(CAST(test.co

Re: [PR] feat: Implement to_json for subset of types [datafusion-comet]

2024-08-13 Thread via GitHub
Kimahriman commented on PR #805: URL: https://github.com/apache/datafusion-comet/pull/805#issuecomment-2286872085 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Refactor `CoalesceBatches` to use an explicit state machine [datafusion]

2024-08-13 Thread via GitHub
berkaysynnada commented on code in PR #11966: URL: https://github.com/apache/datafusion/pull/11966#discussion_r1715744641 ## datafusion/physical-plan/src/coalesce_batches.rs: ## @@ -364,90 +393,60 @@ impl BatchCoalescer { Arc::clone(&self.schema) } -/// Add a

Re: [I] Allow custom planning behavior for selecting wildcard expression [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11639: Allow custom planning behavior for selecting wildcard expression URL: https://github.com/apache/datafusion/issues/11639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Move wildcard expansions to the analyzer [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11681: URL: https://github.com/apache/datafusion/pull/11681#issuecomment-2286867303 🚀 Thanks again @goldmedal and @jayzhan211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] doc: Add memory tuning section to user guide [datafusion-comet]

2024-08-13 Thread via GitHub
viirya commented on PR #684: URL: https://github.com/apache/datafusion-comet/pull/684#issuecomment-2286868125 If Comet `CometPlugin` is configured to be used, it will update Spark executor overhead based on the config value `spark.comet.memoryOverhead` etc. But I don't test it on YARN, not

Re: [PR] Move wildcard expansions to the analyzer [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11681: URL: https://github.com/apache/datafusion/pull/11681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Update `LTRIM` scalar function to support `Utf8View` [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11856: Update `LTRIM` scalar function to support `Utf8View` URL: https://github.com/apache/datafusion/issues/11856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Add native stringview support for LTRIM & RTRIM [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11948: URL: https://github.com/apache/datafusion/pull/11948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Update `RTRIM` scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11916: Update `RTRIM` scalar function to support Utf8View URL: https://github.com/apache/datafusion/issues/11916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Add native stringview support for LTRIM & RTRIM [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11948: URL: https://github.com/apache/datafusion/pull/11948#issuecomment-2286865761 Thanks again @Kev1n8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Implement Utf8View for lpad scalar function [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11941: URL: https://github.com/apache/datafusion/pull/11941#issuecomment-2286864817 Thanks @Omega359 -- let's merge this to get the initial implementation and tests in and then we can work on improvements as a follow on PR. Let me know if you would like me to f

Re: [I] Update `LPAD` scalar function to support `Utf8View` [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11857: Update `LPAD` scalar function to support `Utf8View` URL: https://github.com/apache/datafusion/issues/11857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Implement Utf8View for lpad scalar function [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11941: URL: https://github.com/apache/datafusion/pull/11941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Support Arrays for the Map scalar functions [datafusion]

2024-08-13 Thread via GitHub
alamb commented on PR #11712: URL: https://github.com/apache/datafusion/pull/11712#issuecomment-2286863065 Thanks @dharanad and @jayzhan211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Support Arrays for the Map scalar functions [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11712: URL: https://github.com/apache/datafusion/pull/11712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Support Arrays for the Map scalar functions [datafusion]

2024-08-13 Thread via GitHub
alamb closed issue #11436: Support Arrays for the Map scalar functions URL: https://github.com/apache/datafusion/issues/11436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Minor: simplify SQL number parsing and add a comment about unused [datafusion]

2024-08-13 Thread via GitHub
alamb merged PR #11965: URL: https://github.com/apache/datafusion/pull/11965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] perf: Remove some redundant copying of batches [datafusion-comet]

2024-08-13 Thread via GitHub
viirya commented on code in PR #816: URL: https://github.com/apache/datafusion-comet/pull/816#discussion_r1715737049 ## native/core/src/execution/datafusion/planner.rs: ## @@ -1775,10 +1780,14 @@ impl From for DataFusionError { /// modification. This is used to determine if we

Re: [PR] feat: Implement to_json for subset of types [datafusion-comet]

2024-08-13 Thread via GitHub
andygrove commented on PR #805: URL: https://github.com/apache/datafusion-comet/pull/805#issuecomment-2286855262 @eejbyfeldt @Kimahriman you may also be interested in reviewing this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Update RPAD scalar function to support Utf8View [datafusion]

2024-08-13 Thread via GitHub
Omega359 commented on code in PR #11942: URL: https://github.com/apache/datafusion/pull/11942#discussion_r1715698875 ## datafusion/functions/src/unicode/rpad.rs: ## @@ -76,97 +84,168 @@ impl ScalarUDFImpl for RPadFunc { } fn invoke(&self, args: &[ColumnarValue]) -> R

Re: [PR] feat: Implement to_json for subset of types [datafusion-comet]

2024-08-13 Thread via GitHub
dharanad commented on PR #805: URL: https://github.com/apache/datafusion-comet/pull/805#issuecomment-2286828065 i would also love to review this. will plan it for tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Implement withField and dropField for struct types [datafusion-comet]

2024-08-13 Thread via GitHub
dharanad commented on issue #813: URL: https://github.com/apache/datafusion-comet/issues/813#issuecomment-2286826355 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

  1   2   >