Re: [I] Remove `From` for `Column` [datafusion]

2025-09-02 Thread via GitHub
zuston commented on issue #17375: URL: https://github.com/apache/datafusion/issues/17375#issuecomment-3247819239 This is indeed an implicit conversion. May I help fix this issue? @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] minor: use debug level log for physical optimizer [datafusion]

2025-09-02 Thread via GitHub
2010YOUY01 merged PR #17383: URL: https://github.com/apache/datafusion/pull/17383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

[PR] chore(deps): bump aws-config from 1.8.5 to 1.8.6 [datafusion]

2025-09-02 Thread via GitHub
dependabot[bot] opened a new pull request, #17386: URL: https://github.com/apache/datafusion/pull/17386 Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.5 to 1.8.6. Commits See full diff in https://github.com/smithy-lang/smithy-rs/commits";>compare view

[PR] fix: set distinct_count to Absent when merging statistics [datafusion]

2025-09-02 Thread via GitHub
adriangb opened a new pull request, #17385: URL: https://github.com/apache/datafusion/pull/17385 ## Summary - Fixed issue where distinct_count was not properly handled when merging statistics - Set distinct_count to Precision::Absent during merge operations as the actual distinct coun

Re: [PR] chore: test DataFusion 50 [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2286: URL: https://github.com/apache/datafusion-comet/pull/2286#issuecomment-3246715675 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2286?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Support string column identifiers for sort/aggregate/window and stricter Expr validation [datafusion-python]

2025-09-02 Thread via GitHub
HeWhoHeWho commented on PR #1221: URL: https://github.com/apache/datafusion-python/pull/1221#issuecomment-3247686742 Hi, thanks for the PR. One thing to check, `lit()` or `literal()` will remain _flexible_ as below right? `df.filter(col('A') > 123)` or `df.filter(col('B') == 'Jack

[PR] chore(deps): bump uuid from 1.18.0 to 1.18.1 [datafusion]

2025-09-02 Thread via GitHub
dependabot[bot] opened a new pull request, #17384: URL: https://github.com/apache/datafusion/pull/17384 Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.18.0 to 1.18.1. Release notes Sourced from https://github.com/uuid-rs/uuid/releases";>uuid's releases. v1.18.1 What'

Re: [I] Unparsing of CROSS JOINs with filters is generating incorrect queries [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on issue #17359: URL: https://github.com/apache/datafusion/issues/17359#issuecomment-3243492118 @chenkovsky Is it possible for me to do this one? I had forgotten to 'take', but I was about to get around to it. It is fine if that is not possible, sorry about that. --

Re: [PR] fix: Remove check for custom S3 endpoints [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove commented on PR #2288: URL: https://github.com/apache/datafusion-comet/pull/2288#issuecomment-3247389146 Thanks @parthchandra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] (refac) `SortMergeJoinMetrics` to reuse `BaselineMetrics` [datafusion]

2025-09-02 Thread via GitHub
github-actions[bot] closed pull request #16576: (refac) `SortMergeJoinMetrics` to reuse `BaselineMetrics` URL: https://github.com/apache/datafusion/pull/16576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] chore(deps): bump mimalloc from 0.1.47 to 0.1.48 in /native [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich merged PR #2276: URL: https://github.com/apache/datafusion-comet/pull/2276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

[PR] fix: Remove check for custom S3 endpoints [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove opened a new pull request, #2288: URL: https://github.com/apache/datafusion-comet/pull/2288 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Set fail on warning for documentation generation [datafusion-python]

2025-09-02 Thread via GitHub
timsaucer merged PR #1218: URL: https://github.com/apache/datafusion-python/pull/1218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

[PR] Fixing a few Typos [datafusion-python]

2025-09-02 Thread via GitHub
ntjohnson1 opened a new pull request, #1220: URL: https://github.com/apache/datafusion-python/pull/1220 # Which issue does this PR close? This fixes some typos and optionally adds typo checker to the repo. Can create an issue if needed. # Rationale for this change Saw a typo in

Re: [PR] docs: [branch-0.8] Update Comet version in branch-0.8 user guide [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich merged PR #2282: URL: https://github.com/apache/datafusion-comet/pull/2282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

[PR] Push limits past windows [datafusion]

2025-09-02 Thread via GitHub
avantgardnerio opened a new pull request, #17377: URL: https://github.com/apache/datafusion/pull/17377 ## Which issue does this PR close? - Closes #VTX-9279. ## Rationale for this change Make window functions faster ## What changes are included in this PR?

Re: [PR] chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [datafusion-comet]

2025-09-02 Thread via GitHub
rishvin commented on PR #2287: URL: https://github.com/apache/datafusion-comet/pull/2287#issuecomment-3247488933 LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Comet fails to run clickbench query [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra closed issue #2038: Comet fails to run clickbench query URL: https://github.com/apache/datafusion-comet/issues/2038 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] minor: Reduce misleading fallback warnings [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove opened a new pull request, #2283: URL: https://github.com/apache/datafusion-comet/pull/2283 ## Which issue does this PR close? Closes #. ## Rationale for this change With these changes, these are all the fallback reasons that I see when running

Re: [PR] refactor: simplify json_shredding example by using ListingTable [datafusion]

2025-09-02 Thread via GitHub
waynexia commented on PR #17369: URL: https://github.com/apache/datafusion/pull/17369#issuecomment-3243580147 > > Yes, I've run this example after those changes > > Is the output the same? Yes, I've pair compared the analyze output (as well as others -- This is an automat

Re: [PR] docs: Fix issues with publishing user guide for older Comet versions [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove merged PR #2284: URL: https://github.com/apache/datafusion-comet/pull/2284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[PR] refactor: simplify json_shredding example by using ListingTable [datafusion]

2025-09-02 Thread via GitHub
waynexia opened a new pull request, #17369: URL: https://github.com/apache/datafusion/pull/17369 ## Which issue does this PR close? - Closes #. ## Rationale for this change Trying to prove "why `DataSourceExec` is not a good abstraction for end users" whe

Re: [PR] fix: repartition for grouping set [datafusion]

2025-09-02 Thread via GitHub
alamb commented on PR #16983: URL: https://github.com/apache/datafusion/pull/16983#issuecomment-3245247355 @thinkharderdev / @avantgardnerio -- do you have some time to help review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] feat: Make supported hadoop filesystem schemes configurable [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra commented on code in PR #2272: URL: https://github.com/apache/datafusion-comet/pull/2272#discussion_r2316609761 ## native/core/src/parquet/parquet_support.rs: ## @@ -382,14 +393,15 @@ pub(crate) fn prepare_object_store_with_configs( &url[url::Position::Befo

Re: [PR] chore: docker publish and docs build only for apache repo [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2289: URL: https://github.com/apache/datafusion-comet/pull/2289#issuecomment-3247591419 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2289?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] [native_iceberg_compat] Add support for custom S3 endpoints [datafusion-comet]

2025-09-02 Thread via GitHub
Kontinuation commented on issue #2261: URL: https://github.com/apache/datafusion-comet/issues/2261#issuecomment-3241606168 Now we support configuring S3 endpoint by setting `spark.hadoop.fs.s3a.endpoint`. There's also a test case for it: https://github.com/apache/datafusion-comet/blob/62b3

[PR] better preserve statistics when applying limits [datafusion]

2025-09-02 Thread via GitHub
adriangb opened a new pull request, #17381: URL: https://github.com/apache/datafusion/pull/17381 Closes #17380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] add support for filter clause in window functions [datafusion]

2025-09-02 Thread via GitHub
geoffreyclaude commented on issue #674: URL: https://github.com/apache/datafusion/issues/674#issuecomment-3242915685 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat: Add support for `COUNT(DISTINCT expr)` [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2273: URL: https://github.com/apache/datafusion-comet/pull/2273#issuecomment-3243008961 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2273?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] feat: Add support for `COUNT(DISTINCT expr)` [datafusion-comet]

2025-09-02 Thread via GitHub
comphead commented on PR #2273: URL: https://github.com/apache/datafusion-comet/pull/2273#issuecomment-3243701132 That looks incredibly useful! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] minor: Remove useless ENABLE_COMET_SHUFFLE env [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove merged PR #2280: URL: https://github.com/apache/datafusion-comet/pull/2280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Unparsing of CROSS JOINs with filters is generating incorrect queries [datafusion]

2025-09-02 Thread via GitHub
chenkovsky commented on issue #17359: URL: https://github.com/apache/datafusion/issues/17359#issuecomment-3243467835 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] minor: make dict_from_values public [datafusion]

2025-09-02 Thread via GitHub
comphead commented on code in PR #17376: URL: https://github.com/apache/datafusion/pull/17376#discussion_r2317039606 ## datafusion/common/src/scalar/mod.rs: ## @@ -882,7 +882,7 @@ fn dict_from_scalar( } /// Create a dictionary array representing all the values in values Rev

Re: [PR] Add PhysicalExpr::is_volatile [datafusion]

2025-09-02 Thread via GitHub
adriangb commented on code in PR #17351: URL: https://github.com/apache/datafusion/pull/17351#discussion_r2317444574 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -377,6 +377,19 @@ pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash {

[PR] minor: use debug level log for physical optimizer [datafusion]

2025-09-02 Thread via GitHub
waynexia opened a new pull request, #17383: URL: https://github.com/apache/datafusion/pull/17383 ## Which issue does this PR close? - Closes #. ## Rationale for this change use the same level as the logical optimizer https://github.com/apache/datafusio

Re: [I] Integrate Apache OpenDAL to support more file serivces [datafusion-comet]

2025-09-02 Thread via GitHub
wForget commented on issue #2243: URL: https://github.com/apache/datafusion-comet/issues/2243#issuecomment-3247426083 > [@wForget](https://github.com/wForget) this looks very promising. I have submitted a pr to support hdfs with OpenDAL #2272. Could you please take a look at it?

Re: [PR] doc: Document caveats of `swap_inputs()` interface in join executors [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on code in PR #17373: URL: https://github.com/apache/datafusion/pull/17373#discussion_r2317327468 ## datafusion/physical-plan/src/joins/hash_join/exec.rs: ## @@ -638,6 +638,21 @@ impl HashJoinExec { /// /// This function is public so other downstr

[PR] chore: docker publish and docs build only for apache repo [datafusion-comet]

2025-09-02 Thread via GitHub
wForget opened a new pull request, #2289: URL: https://github.com/apache/datafusion-comet/pull/2289 ## Which issue does this PR close? Closes #. ## Rationale for this change docker publish and docs build will fail in personal repo ## What changes are included in th

Re: [PR] Add PhysicalExpr::is_volatile [datafusion]

2025-09-02 Thread via GitHub
adriangb commented on PR #17351: URL: https://github.com/apache/datafusion/pull/17351#issuecomment-3247183043 > We can add this to existing volatile expressions in this PR + add tests, or we could do in follow up if not possible. How many cases are there? If it's reasonable to do here

Re: [PR] chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [datafusion-comet]

2025-09-02 Thread via GitHub
rishvin commented on code in PR #2287: URL: https://github.com/apache/datafusion-comet/pull/2287#discussion_r2317615018 ## spark/src/main/scala/org/apache/comet/serde/math.scala: ## @@ -112,6 +112,40 @@ object CometLog2 extends CometExpressionSerde[Log2] with MathExprBase {

Re: [PR] feat: Make supported hadoop filesystem schemes configurable [datafusion-comet]

2025-09-02 Thread via GitHub
wForget commented on code in PR #2272: URL: https://github.com/apache/datafusion-comet/pull/2272#discussion_r2317613385 ## common/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala: ## @@ -40,6 +41,8 @@ object NativeConfig { // Azure Data Lake Storage Gen2 secur

Re: [PR] chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2287: URL: https://github.com/apache/datafusion-comet/pull/2287#issuecomment-3247420099 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2287?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] fix: implement lazy evaluation in Coalesce function [datafusion-comet]

2025-09-02 Thread via GitHub
coderfender commented on PR #2270: URL: https://github.com/apache/datafusion-comet/pull/2270#issuecomment-3247410744 Thank you very much @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] minor: make dict_from_values public [datafusion]

2025-09-02 Thread via GitHub
parthchandra commented on code in PR #17376: URL: https://github.com/apache/datafusion/pull/17376#discussion_r2317464667 ## datafusion/common/src/scalar/mod.rs: ## @@ -882,7 +882,7 @@ fn dict_from_scalar( } /// Create a dictionary array representing all the values in values

[PR] docs: fix CaseBuilder documentation example [datafusion-python]

2025-09-02 Thread via GitHub
IndexSeek opened a new pull request, #1225: URL: https://github.com/apache/datafusion-python/pull/1225 # Which issue does this PR close? Closes #1176 # Rationale for this change # What changes are included in this PR? Resolves the `df.select( ^ Syntax

Re: [PR] Add PhysicalExpr::is_volatile [datafusion]

2025-09-02 Thread via GitHub
adriangb commented on PR #17351: URL: https://github.com/apache/datafusion/pull/17351#issuecomment-3247256961 Functions are the only ones that I know of -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] fix: implement lazy evaluation in Coalesce function [datafusion-comet]

2025-09-02 Thread via GitHub
coderfender commented on PR #2270: URL: https://github.com/apache/datafusion-comet/pull/2270#issuecomment-3247300578 @andygrove please take a look whenever you get a chance . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [I] Integrate Apache OpenDAL to support more file serivces [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra commented on issue #2243: URL: https://github.com/apache/datafusion-comet/issues/2243#issuecomment-3247243245 @wForget this looks very promising. Are you planning to work on this? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] minor: make dict_from_values public [datafusion]

2025-09-02 Thread via GitHub
comphead commented on code in PR #17376: URL: https://github.com/apache/datafusion/pull/17376#discussion_r2317471018 ## datafusion/common/src/scalar/mod.rs: ## @@ -882,7 +882,7 @@ fn dict_from_scalar( } /// Create a dictionary array representing all the values in values Rev

Re: [PR] fix: implement lazy evaluation in Coalesce function [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2270: URL: https://github.com/apache/datafusion-comet/pull/2270#issuecomment-3247154784 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2270?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [datafusion-comet]

2025-09-02 Thread via GitHub
hsiang-c commented on code in PR #2287: URL: https://github.com/apache/datafusion-comet/pull/2287#discussion_r2317396416 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -1015,23 +1017,6 @@ object QueryPlanSerde extends Logging with CometExprShim { //

[PR] chore: Refactor `hex`/`unhex` SerDe to avoid code duplication [datafusion-comet]

2025-09-02 Thread via GitHub
hsiang-c opened a new pull request, #2287: URL: https://github.com/apache/datafusion-comet/pull/2287 ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/2019 ## Rationale for this change - Reduce the complexity of `Que

Re: [PR] fix: Remove duplicate filter from `CrossJoin` unparsing [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on code in PR #17382: URL: https://github.com/apache/datafusion/pull/17382#discussion_r2317313953 ## datafusion/sql/Cargo.toml: ## @@ -61,6 +61,7 @@ sqlparser = { workspace = true } [dev-dependencies] ctor = { workspace = true } # please do not move thes

Re: [PR] Add PhysicalExpr::is_volatile [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on code in PR #17351: URL: https://github.com/apache/datafusion/pull/17351#discussion_r2317357983 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -377,6 +377,19 @@ pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash

Re: [PR] Add PhysicalExpr::is_volatile [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on code in PR #17351: URL: https://github.com/apache/datafusion/pull/17351#discussion_r2317357983 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -377,6 +377,19 @@ pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash

Re: [PR] chore: Align sort constraints w/ `arrow-rs` [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2279: URL: https://github.com/apache/datafusion-comet/pull/2279#issuecomment-3247011595 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2279?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] fix: Remove duplicate filter from `CrossJoin` unparsing [datafusion]

2025-09-02 Thread via GitHub
jonathanc-n commented on code in PR #17382: URL: https://github.com/apache/datafusion/pull/17382#discussion_r2317313953 ## datafusion/sql/Cargo.toml: ## @@ -61,6 +61,7 @@ sqlparser = { workspace = true } [dev-dependencies] ctor = { workspace = true } # please do not move thes

Re: [PR] Feat : Bringing in support for map_filter expression. [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra commented on code in PR #2236: URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2317232622 ## native/spark-expr/src/map_funcs/map_filter.rs: ## @@ -0,0 +1,539 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribu

[I] Smarter `Statistics::with_fetch` [datafusion]

2025-09-02 Thread via GitHub
adriangb opened a new issue, #17380: URL: https://github.com/apache/datafusion/issues/17380 ### Is your feature request related to a problem or challenge? It currently discards all information for column stats as well as total byte size. I think instead we should mark both as `Prec

Re: [PR] fix: potential native broadcast failure in scenarios with ReusedExhange [datafusion-comet]

2025-09-02 Thread via GitHub
viirya commented on PR #2167: URL: https://github.com/apache/datafusion-comet/pull/2167#issuecomment-3246353347 > @viirya would you be able to look at this? I will take a look but maybe not very soon. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] [native_iceberg_compat] Intermittent failure to resolve bucket region [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove closed issue #2285: [native_iceberg_compat] Intermittent failure to resolve bucket region URL: https://github.com/apache/datafusion-comet/issues/2285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] `FileScanConfig::output_ordering` must be vector of optionals [datafusion]

2025-09-02 Thread via GitHub
destrex271 commented on issue #17354: URL: https://github.com/apache/datafusion/issues/17354#issuecomment-3246806624 Hi, picking this up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] feat: Add nested Array literal support [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2181: URL: https://github.com/apache/datafusion-comet/pull/2181#issuecomment-3246781767 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2181?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] fix: handle cast to dictionary vector introduced by case when [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra commented on PR #2044: URL: https://github.com/apache/datafusion-comet/pull/2044#issuecomment-3245976334 Issue: https://github.com/apache/datafusion/issues/17366 PR: https://github.com/apache/datafusion/pull/17376 -- This is an automated message from the Apache Git Service

[I] Add named notation for table function arguments [datafusion]

2025-09-02 Thread via GitHub
timsaucer opened a new issue, #17379: URL: https://github.com/apache/datafusion/issues/17379 ### Is your feature request related to a problem or challenge? As a user I would like to pass a series of arguments to a table function. I would like to name these arguments so that I can use

Re: [PR] perf: Avoid FFI copy in `ScanExec` when reading data from exchanges [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich merged PR #2268: URL: https://github.com/apache/datafusion-comet/pull/2268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

[I] [native_iceberg_compat] Intermittent failure to resolve bucket region [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove opened a new issue, #2285: URL: https://github.com/apache/datafusion-comet/issues/2285 ### Describe the bug I sometimes see queries fail due to errors resolving the bucket region. In Comet, we have this native code: ```rust let region =

Re: [PR] fix: set IPC alignment based on schema [datafusion]

2025-09-02 Thread via GitHub
2010YOUY01 commented on code in PR #17363: URL: https://github.com/apache/datafusion/pull/17363#discussion_r2315708974 ## datafusion/physical-plan/src/spill/mod.rs: ## @@ -308,7 +308,7 @@ impl IPCStreamWriter { })?; let metadata_version = MetadataVersion::V5;

Re: [PR] feat_enable_version_specific_docs [datafusion-comet]

2025-09-02 Thread via GitHub
coderfender closed pull request #2170: feat_enable_version_specific_docs URL: https://github.com/apache/datafusion-comet/pull/2170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] docs: [branch-0.9] Update Comet version in branch-0.9 user guide [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich merged PR #2281: URL: https://github.com/apache/datafusion-comet/pull/2281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [I] Potential revamp of broadcast compression policy [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra commented on issue #2216: URL: https://github.com/apache/datafusion-comet/issues/2216#issuecomment-3246056888 I see. Thanks for clarifying! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Version documentation [datafusion]

2025-09-02 Thread via GitHub
andygrove commented on issue #17071: URL: https://github.com/apache/datafusion/issues/17071#issuecomment-3246040664 We are now publishing documentation for multiple versions of Comet. In the script that builds the documentation site, we basically just run a `git clone --depth 1` of th

Re: [PR] Push limits past windows [datafusion]

2025-09-02 Thread via GitHub
avantgardnerio closed pull request #17377: Push limits past windows URL: https://github.com/apache/datafusion/pull/17377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[PR] minor: make dict_from_values public [datafusion]

2025-09-02 Thread via GitHub
parthchandra opened a new pull request, #17376: URL: https://github.com/apache/datafusion/pull/17376 ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/17366. ## Rationale for this change reduce code duplication in Datafusion Comet.

Re: [PR] fix: handle cast to dictionary vector introduced by case when [datafusion-comet]

2025-09-02 Thread via GitHub
parthchandra merged PR #2044: URL: https://github.com/apache/datafusion-comet/pull/2044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] chore: Refactor serde for conditional expressions [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove merged PR #2266: URL: https://github.com/apache/datafusion-comet/pull/2266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[PR] docs: Fix issues with publishing user guide for older Comet versions [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove opened a new pull request, #2284: URL: https://github.com/apache/datafusion-comet/pull/2284 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Expose Arrow C stream and DataFrame iterator (zero‑copy streaming to PyArrow) [datafusion-python]

2025-09-02 Thread via GitHub
kylebarron commented on PR #1222: URL: https://github.com/apache/datafusion-python/pull/1222#issuecomment-3245924248 I'm invested in this and plan to review this this afternoon! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] docs: Publish version-specific user guides [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove commented on code in PR #2269: URL: https://github.com/apache/datafusion-comet/pull/2269#discussion_r2316490732 ## docs/generate-versions.py: ## @@ -0,0 +1,6 @@ +import os Review Comment: Thanks. I added some comments -- This is an automated message from the Ap

Re: [PR] fix: set IPC alignment based on schema [datafusion]

2025-09-02 Thread via GitHub
ding-young commented on PR #17363: URL: https://github.com/apache/datafusion/pull/17363#issuecomment-3245248823 > BTW I can't predict how bad this round trip memory blowup issue gonna be, do you know any other? If it's common we might give up, and when this validation fail we can simply pri

Re: [PR] docs: Publish version-specific user guides [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove merged PR #2269: URL: https://github.com/apache/datafusion-comet/pull/2269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] Fixing a few Typos [datafusion-python]

2025-09-02 Thread via GitHub
timsaucer merged PR #1220: URL: https://github.com/apache/datafusion-python/pull/1220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] docs: Publish version-specific user guides [datafusion-comet]

2025-09-02 Thread via GitHub
comphead commented on code in PR #2269: URL: https://github.com/apache/datafusion-comet/pull/2269#discussion_r2316431463 ## docs/generate-versions.py: ## @@ -0,0 +1,6 @@ +import os Review Comment: @andygrove please help to understand if this is a temporary helper file? If i

Re: [PR] Add PostgreSQL `CREATE USER` and `ALTER USER` support [datafusion-sqlparser-rs]

2025-09-02 Thread via GitHub
ramnes commented on code in PR #2015: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2015#discussion_r2316395025 ## src/ast/mod.rs: ## @@ -3314,6 +3314,8 @@ pub enum Statement { CreateRole { names: Vec, if_not_exists: bool, +/// Wheth

[PR] docs: update link to user example for custom table provider [datafusion-python]

2025-09-02 Thread via GitHub
IndexSeek opened a new pull request, #1224: URL: https://github.com/apache/datafusion-python/pull/1224 # Which issue does this PR close? Closes #1223 # Rationale for this change # What changes are included in this PR? Repoints the URL to the example proje

Re: [PR] Add PostgreSQL `CREATE USER` and `ALTER USER` support [datafusion-sqlparser-rs]

2025-09-02 Thread via GitHub
ramnes commented on code in PR #2015: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2015#discussion_r2316372357 ## src/ast/mod.rs: ## @@ -3314,6 +3314,8 @@ pub enum Statement { CreateRole { names: Vec, if_not_exists: bool, +/// Wheth

Re: [PR] chore(deps): bump mimalloc from 0.1.47 to 0.1.48 in /native [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich commented on PR #2276: URL: https://github.com/apache/datafusion-comet/pull/2276#issuecomment-3245692334 I don't know what is hidden behind the `v3` feature flag. That might be worth exploring at some point. For now this LGTM. -- This is an automated message from the Apache Gi

[PR] minor: Remove useless ENABLE_COMET_SHUFFLE env [datafusion-comet]

2025-09-02 Thread via GitHub
wForget opened a new pull request, #2280: URL: https://github.com/apache/datafusion-comet/pull/2280 ## Which issue does this PR close? Closes #. ## Rationale for this change `ENABLE_COMET_SHUFFLE` has been removed since #1240 ## What changes are included in

Re: [PR] perf: Avoid FFI copy in `ScanExec` when reading data from exchanges [datafusion-comet]

2025-09-02 Thread via GitHub
mbutrovich commented on code in PR #2268: URL: https://github.com/apache/datafusion-comet/pull/2268#discussion_r2316249628 ## native/core/src/execution/operators/scan.rs: ## @@ -89,6 +91,7 @@ impl ScanExec { input_source: Option>, input_source_description: &str

Re: [PR] minor: Remove useless ENABLE_COMET_SHUFFLE env [datafusion-comet]

2025-09-02 Thread via GitHub
codecov-commenter commented on PR #2280: URL: https://github.com/apache/datafusion-comet/pull/2280#issuecomment-3245516907 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2280?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] docs: Publish version-specific user guides [datafusion-comet]

2025-09-02 Thread via GitHub
andygrove commented on code in PR #2269: URL: https://github.com/apache/datafusion-comet/pull/2269#discussion_r2316188570 ## docs/generate-versions.py: ## @@ -0,0 +1,6 @@ +import os + +for version in ["0.7", "0.8", "0.9"]: Review Comment: For now, I'd prefer to keep this a m

Re: [PR] Redshift: UNLOAD [datafusion-sqlparser-rs]

2025-09-02 Thread via GitHub
iffyio merged PR #2013: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

[I] Stabilize FFI Boundary [datafusion]

2025-09-02 Thread via GitHub
timsaucer opened a new issue, #17374: URL: https://github.com/apache/datafusion/issues/17374 ### Is your feature request related to a problem or challenge? Between DataFusion 48 and 49 there was a [breaking change](https://github.com/apache/datafusion/pull/17190) in the FFI boundary

Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]

2025-09-02 Thread via GitHub
arpity22 commented on issue #14535: URL: https://github.com/apache/datafusion/issues/14535#issuecomment-3245371095 Cool! I’ll stick around and keep reading up on things. Feel free to ping me if you think I can help, or create an issue on your repo, I’ll check upon them once in a while.

Re: [PR] chore(deps): bump aws-credential-types from 1.2.5 to 1.2.6 [datafusion]

2025-09-02 Thread via GitHub
alamb merged PR #17368: URL: https://github.com/apache/datafusion/pull/17368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] chore(deps): bump the arrow-parquet group with 7 updates [datafusion]

2025-09-02 Thread via GitHub
alamb closed pull request #17335: chore(deps): bump the arrow-parquet group with 7 updates URL: https://github.com/apache/datafusion/pull/17335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] chore(deps): bump the arrow-parquet group with 7 updates [datafusion]

2025-09-02 Thread via GitHub
alamb commented on PR #17335: URL: https://github.com/apache/datafusion/pull/17335#issuecomment-3245238053 Superceded by - https://github.com/apache/datafusion/pull/17275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] fix: set IPC alignment based on schema [datafusion]

2025-09-02 Thread via GitHub
ding-young commented on code in PR #17363: URL: https://github.com/apache/datafusion/pull/17363#discussion_r2316010183 ## datafusion/physical-plan/src/spill/mod.rs: ## @@ -308,7 +308,7 @@ impl IPCStreamWriter { })?; let metadata_version = MetadataVersion::V5;

[PR] doc: Document caveats of `swap_inputs()` interface in join executors [datafusion]

2025-09-02 Thread via GitHub
2010YOUY01 opened a new pull request, #17373: URL: https://github.com/apache/datafusion/pull/17373 ## Which issue does this PR close? related to https://github.com/apache/datafusion/issues/17358 ## Rationale for this change It's not clear to me how to prop

Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]

2025-09-02 Thread via GitHub
2010YOUY01 commented on issue #14535: URL: https://github.com/apache/datafusion/issues/14535#issuecomment-3245015061 > Hi! This looks really interesting. I’d like to work on it (just as a side project for fun). Is anyone else working on it at the moment who I could sync up with? I'm

  1   2   >