MrGranday commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2303040045
##
datafusion/core/src/physical_planner.rs:
##
@@ -946,6 +947,18 @@ impl DefaultPhysicalPlanner {
))
}
Review Comment:
Makes
MrGranday commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2303038099
##
datafusion/core/src/dataframe/mod.rs:
##
@@ -232,19 +242,31 @@ pub struct DataFrame {
projection_requires_validation: bool,
}
-impl DataFrame {
-/
MrGranday commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2303036694
##
datafusion/core/src/dataframe/mod.rs:
##
@@ -232,19 +242,31 @@ pub struct DataFrame {
projection_requires_validation: bool,
}
-impl DataFrame {
-/
MrGranday commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2303035284
##
datafusion/core/src/dataframe/mod.rs:
##
@@ -232,19 +242,31 @@ pub struct DataFrame {
projection_requires_validation: bool,
}
-impl DataFrame {
-/
kosiew commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2303007475
##
datafusion/common/src/nested_struct.rs:
##
@@ -159,14 +200,23 @@ pub fn cast_column(source_col: &ArrayRef, target_field:
&Field) -> Result Result {
+) -> Result
kosiew commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2303006583
##
datafusion/expr-common/src/columnar_value.rs:
##
@@ -210,9 +210,22 @@ impl ColumnarValue {
) -> Result {
let cast_options =
cast_options.cloned().
duongcongtoai commented on PR #17110:
URL: https://github.com/apache/datafusion/pull/17110#issuecomment-3226655728
duckdb has a notion of `PropagatesNullValues`, i wonder if we have some
thing similar
```
impl ExprSchemable for Expr {
fn nullable(&self, input_schema: &dyn ExprSc
rishvin commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2302726421
##
native/spark-expr/src/map_funcs/map_filter.rs:
##
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
rishvin commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2302719310
##
spark/src/main/scala/org/apache/comet/serde/maps.scala:
##
@@ -89,3 +89,21 @@ object CometMapFromArrays extends
CometExpressionSerde[MapFromArrays] {
alamb commented on PR #16066:
URL: https://github.com/apache/datafusion/pull/16066#issuecomment-3226515095
🤖: Benchmark completed
Details
```
group
fix_common_subexpression_eliminate main
-
coderfender commented on code in PR #2099:
URL: https://github.com/apache/datafusion-comet/pull/2099#discussion_r2302668961
##
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##
@@ -71,44 +100,78 @@ fn spark_read_side_padding2(
}
}
+enum RPad
wForget commented on code in PR #2235:
URL: https://github.com/apache/datafusion-comet/pull/2235#discussion_r2302562911
##
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/NativeBatchDecoderIterator.scala:
##
@@ -192,4 +193,8 @@ object NativeBatchDecoderIterator
rishvin commented on issue #1994:
URL:
https://github.com/apache/datafusion-comet/issues/1994#issuecomment-3226301493
I would like to explore this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
andygrove merged PR #2168:
URL: https://github.com/apache/datafusion-comet/pull/2168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@
andygrove merged PR #2178:
URL: https://github.com/apache/datafusion-comet/pull/2178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@
andygrove commented on code in PR #2211:
URL: https://github.com/apache/datafusion-comet/pull/2211#discussion_r2302398942
##
dev/diffs/3.4.3.diff:
##
@@ -894,6 +894,19 @@ index 525d97e4998..8a3e7457618 100644
AccumulatorSuite.verifyPeakExecutionMemorySet(sparkContext, "ext
andygrove commented on code in PR #2211:
URL: https://github.com/apache/datafusion-comet/pull/2211#discussion_r2302398587
##
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##
@@ -2368,13 +2368,13 @@ object QueryPlanSerde extends Logging with
CometExprShim {
shehabgamin commented on code in PR #16946:
URL: https://github.com/apache/datafusion/pull/16946#discussion_r2302373800
##
datafusion/spark/src/function/conditional/if.rs:
##
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor
BlakeOrth commented on PR #17232:
URL: https://github.com/apache/datafusion/pull/17232#issuecomment-3226017592
@alamb @nuno-faria I believe I touched all the asks from the previous round
of feedback with the latest commit. I referenced a recent PR by @nuno-faria
where the parquet metadata c
waynexia merged PR #17333:
URL: https://github.com/apache/datafusion/pull/17333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@dataf
andygrove commented on code in PR #2211:
URL: https://github.com/apache/datafusion-comet/pull/2211#discussion_r2302355630
##
dev/diffs/3.4.3.diff:
##
@@ -894,6 +894,19 @@ index 525d97e4998..8a3e7457618 100644
AccumulatorSuite.verifyPeakExecutionMemorySet(sparkContext, "ext
parthchandra commented on issue #2209:
URL:
https://github.com/apache/datafusion-comet/issues/2209#issuecomment-3225999521
> For native_iceberg_compat the Comet doesnt use the SparkSchemaAdapter
where most of cases already handled
Au contraire
https://github.com/apache/datafusio
waynexia commented on PR #17333:
URL: https://github.com/apache/datafusion/pull/17333#issuecomment-3226013554
> Thanks! Would you like to add
[`main`/datafusion-examples/examples/default_column_values.rs](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/default_col
kazuyukitanimura commented on code in PR #2211:
URL: https://github.com/apache/datafusion-comet/pull/2211#discussion_r2302340455
##
dev/diffs/3.4.3.diff:
##
@@ -894,6 +894,19 @@ index 525d97e4998..8a3e7457618 100644
AccumulatorSuite.verifyPeakExecutionMemorySet(sparkContex
parthchandra commented on issue #2209:
URL:
https://github.com/apache/datafusion-comet/issues/2209#issuecomment-3226003226
It might be a while till we get to it. arrow-rs has a bunch of issues
outstanding for variant. Not sure where we are with the support in parquet.
--
This is an auto
kazuyukitanimura commented on issue #2215:
URL:
https://github.com/apache/datafusion-comet/issues/2215#issuecomment-3225998425
We should revert the diffs in
https://github.com/apache/datafusion-comet/pull/2211 once this gets resovled
--
This is an automated message from the Apache Git Se
adriangb commented on PR #17333:
URL: https://github.com/apache/datafusion/pull/17333#issuecomment-3225945382
Thanks! Would you like to add
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/default_column_values.rs
while we're at it in this or a followup PR?
--
parthchandra commented on code in PR #2168:
URL: https://github.com/apache/datafusion-comet/pull/2168#discussion_r2302291779
##
spark/src/main/java/org/apache/comet/CometBatchIterator.java:
##
@@ -26,13 +26,45 @@
import org.apache.comet.vector.NativeUtil;
/**
- * An iterator
parthchandra commented on code in PR #2168:
URL: https://github.com/apache/datafusion-comet/pull/2168#discussion_r2302290257
##
spark/src/main/java/org/apache/comet/CometBatchIterator.java:
##
@@ -26,13 +26,45 @@
import org.apache.comet.vector.NativeUtil;
/**
- * An iterator
waynexia commented on PR #17333:
URL: https://github.com/apache/datafusion/pull/17333#issuecomment-3225903005
cc @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
waynexia opened a new pull request, #17333:
URL: https://github.com/apache/datafusion/pull/17333
## Which issue does this PR close?
- Closes #.
## Rationale for this change
Come across that example is not listed in readme
## What changes are include
rishvin commented on PR #2221:
URL:
https://github.com/apache/datafusion-comet/pull/2221#issuecomment-3225869863
[PR Build (macOS) / macos-14/Spark 3.4, JDK 11, Scala 2.12 [exec]
(pull_request)](https://github.com/apache/datafusion-comet/actions/runs/17228255188/job/48948193988?pr=2221)
andygrove commented on code in PR #2235:
URL: https://github.com/apache/datafusion-comet/pull/2235#discussion_r2302221827
##
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/NativeBatchDecoderIterator.scala:
##
@@ -192,4 +193,8 @@ object NativeBatchDecoderIterat
milenkovicm opened a new pull request, #1299:
URL: https://github.com/apache/datafusion-ballista/pull/1299
# Which issue does this PR close?
Closes #.
# Rationale for this change
Make `DisplayAs` consistent across Shuffle*Exec
# What changes are included in this P
comphead commented on issue #2174:
URL:
https://github.com/apache/datafusion-comet/issues/2174#issuecomment-3225711921
[test.zip](https://github.com/user-attachments/files/21995757/test.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
comphead commented on issue #2174:
URL:
https://github.com/apache/datafusion-comet/issues/2174#issuecomment-3225710788
the issue is likely connected to parquet file reader, I can reproduce with
specific parquet created by `makeParquetFileAllPrimitiveTypes`. The same data
and data types wri
mbutrovich commented on PR #17195:
URL: https://github.com/apache/datafusion/pull/17195#issuecomment-3225676497
I haven't forgotten about this. I need to generate a new .slt file for the
relevant tests.
--
This is an automated message from the Apache Git Service.
To respond to the message
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2302046403
##
datafusion/datasource-json/src/file_format.rs:
##
@@ -251,16 +251,16 @@ impl FileFormat for JsonFormat {
async fn create_physical_plan(
&s
milenkovicm closed pull request #1272: feat: consistent hash scheduling
implemented as `DistributionPolicy`
URL: https://github.com/apache/datafusion-ballista/pull/1272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
codecov-commenter commented on PR #2242:
URL:
https://github.com/apache/datafusion-comet/pull/2242#issuecomment-3225622361
##
[Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2242?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2302051640
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -580,12 +616,12 @@ impl FileSource for ParquetSource {
self
}
-fn metrics(&self)
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2302043230
##
datafusion-examples/examples/parquet_exec_visitor.rs:
##
@@ -98,9 +98,11 @@ impl ExecutionPlanVisitor for ParquetExecVisitor {
fn pre_visit(&mut se
codecov-commenter commented on PR #2241:
URL:
https://github.com/apache/datafusion-comet/pull/2241#issuecomment-3225582066
##
[Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2241?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
andygrove opened a new pull request, #2242:
URL: https://github.com/apache/datafusion-comet/pull/2242
## Which issue does this PR close?
Closes https://github.com/apache/datafusion-comet/issues/2239
Builds on https://github.com/apache/datafusion-comet/pull/2211
##
ramnes opened a new pull request, #2015:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2015
Currently, the library only supports `CREATE ROLE` and `ALTER ROLE` for
PostgreSQL. `CREATE USER` and `ALTER USER` fail to parse with errors
like `"Expected: an object type after CR
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301993969
##
datafusion/datasource/src/file.rs:
##
@@ -270,8 +293,29 @@ impl DataSource for T {
SchedulingType::Cooperative
}
-fn data_source_sta
andygrove commented on code in PR #2241:
URL: https://github.com/apache/datafusion-comet/pull/2241#discussion_r2301931560
##
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##
@@ -849,6 +891,41 @@ case class CometExecRule(session: SparkSession) extends
Rule[Spa
iffyio merged PR #2014:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr
andygrove commented on code in PR #2241:
URL: https://github.com/apache/datafusion-comet/pull/2241#discussion_r2301927516
##
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##
@@ -849,6 +891,41 @@ case class CometExecRule(session: SparkSession) extends
Rule[Spa
iffyio merged PR #2001:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr
iffyio closed issue #2000: Support Postgres' `JSON_OBJECT` function `RETURNING`
clause
URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
iffyio merged PR #2004:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr
mbutrovich opened a new pull request, #2241:
URL: https://github.com/apache/datafusion-comet/pull/2241
## Which issue does this PR close?
Partially addresses #2199.
## Rationale for this change
CometExchange (native) and CometColumnarExchange support diffe
andygrove merged PR #2240:
URL: https://github.com/apache/datafusion-comet/pull/2240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@
andygrove commented on PR #2240:
URL:
https://github.com/apache/datafusion-comet/pull/2240#issuecomment-3225401745
Thanks for the review @mbutrovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
milenkovicm commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2301349185
##
datafusion/core/src/dataframe/mod.rs:
##
@@ -232,19 +242,31 @@ pub struct DataFrame {
projection_requires_validation: bool,
}
-impl DataFrame {
-
timsaucer opened a new pull request, #17332:
URL: https://github.com/apache/datafusion/pull/17332
## Which issue does this PR close?
- Closes https://github.com/apache/datafusion/issues/15907
## Rationale for this change
For some users, adding in sqlparser greatly increas
comphead closed issue #2193: Improve SHA2 user-error message in serdes code
URL: https://github.com/apache/datafusion-comet/issues/2193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
comphead merged PR #2213:
URL: https://github.com/apache/datafusion-comet/pull/2213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@d
andygrove commented on PR #2240:
URL:
https://github.com/apache/datafusion-comet/pull/2240#issuecomment-3225303195
> * Should we prefer "Comet" over "we" in comments?
Yes, I can update this in the next PR
(https://github.com/apache/datafusion-comet/pull/2211)
> * Do we
vegarsti commented on code in PR #16985:
URL: https://github.com/apache/datafusion/pull/16985#discussion_r2301809113
##
datafusion/sqllogictest/test_files/unnest.slt:
##
@@ -941,3 +941,33 @@ where min_height * width1 = (
)
4 7 4 28
+
+## Unnest with ordering on unrelated
mbutrovich merged PR #2237:
URL: https://github.com/apache/datafusion-comet/pull/2237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...
andygrove commented on code in PR #2240:
URL: https://github.com/apache/datafusion-comet/pull/2240#discussion_r2301765516
##
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##
@@ -21,16 +21,7 @@ package org.apache.comet.expressions
import org.apache.spark.s
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301560549
##
datafusion/datasource/src/file.rs:
##
@@ -50,31 +65,31 @@ pub fn as_file_source(source: T)
-> Arc
/// *
[`ParquetSource`](https://docs.rs/datafusion/
codecov-commenter commented on PR #2240:
URL:
https://github.com/apache/datafusion-comet/pull/2240#issuecomment-3225142264
##
[Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2240?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
xanderbailey commented on code in PR #17299:
URL: https://github.com/apache/datafusion/pull/17299#discussion_r2301668484
##
datafusion/substrait/src/logical_plan/consumer/rel/project_rel.rs:
##
@@ -62,7 +62,18 @@ pub async fn from_project_rel(
// to transform it
andygrove opened a new pull request, #2240:
URL: https://github.com/apache/datafusion-comet/pull/2240
## Which issue does this PR close?
Part of https://github.com/apache/datafusion-comet/issues/2239
## Rationale for this change
We currently have two diffe
adriangb commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301621010
##
datafusion/datasource/src/source.rs:
##
@@ -179,6 +179,10 @@ pub trait DataSource: Send + Sync + Debug {
vec![PushedDown::No; filters.len()],
codecov-commenter commented on PR #2238:
URL:
https://github.com/apache/datafusion-comet/pull/2238#issuecomment-3225031407
##
[Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2238?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca
adriangb commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301619788
##
datafusion/datasource/src/file.rs:
##
@@ -50,31 +65,31 @@ pub fn as_file_source(source: T)
-> Arc
/// *
[`ParquetSource`](https://docs.rs/datafusion/latest/
adriangb commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301621710
##
datafusion/datasource/src/source.rs:
##
@@ -465,18 +468,9 @@ impl DataSourceExec {
///
/// Returns `None` if
/// 1. the datasource is not scanni
andygrove opened a new pull request, #2238:
URL: https://github.com/apache/datafusion-comet/pull/2238
## Which issue does this PR close?
N/A
## Rationale for this change
This is a step towards making `Cast` serde more consistent with other
expressions.
andygrove closed pull request #2238: chore: Add type parameter to IncompatExpr
URL: https://github.com/apache/datafusion-comet/pull/2238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
friendlymatthew commented on code in PR #17242:
URL: https://github.com/apache/datafusion/pull/17242#discussion_r2301593271
##
datafusion/datasource/src/source.rs:
##
@@ -179,6 +179,10 @@ pub trait DataSource: Send + Sync + Debug {
vec)
}
Review Comment:
I do
codetyri0n commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2301332228
##
native/spark-expr/src/map_funcs/map_filter.rs:
##
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributo
milenkovicm commented on code in PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#discussion_r2301340322
##
datafusion/core/src/dataframe/mod.rs:
##
@@ -232,19 +242,31 @@ pub struct DataFrame {
projection_requires_validation: bool,
}
-impl DataFrame {
-
codetyri0n commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2301332228
##
native/spark-expr/src/map_funcs/map_filter.rs:
##
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributo
milenkovicm commented on PR #17314:
URL: https://github.com/apache/datafusion/pull/17314#issuecomment-3224607188
is this PR work in progress @MrGranday ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
caicancai commented on code in PR #17307:
URL: https://github.com/apache/datafusion/pull/17307#discussion_r2301114549
##
datafusion/core/tests/catalog/memory.rs:
##
@@ -47,6 +47,20 @@ fn memory_catalog_dereg_nonempty_schema() {
assert!(cat.deregister_schema("foo", true).unw
codetyri0n commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2301322614
##
native/spark-expr/src/map_funcs/map_filter.rs:
##
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributo
caicancai commented on code in PR #17307:
URL: https://github.com/apache/datafusion/pull/17307#discussion_r2301285539
##
datafusion/core/tests/catalog/memory.rs:
##
@@ -47,6 +47,20 @@ fn memory_catalog_dereg_nonempty_schema() {
assert!(cat.deregister_schema("foo", true).unw
mbutrovich commented on code in PR #2236:
URL: https://github.com/apache/datafusion-comet/pull/2236#discussion_r2301183180
##
native/spark-expr/src/map_funcs/map_filter.rs:
##
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributo
caicancai commented on code in PR #17307:
URL: https://github.com/apache/datafusion/pull/17307#discussion_r2301114549
##
datafusion/core/tests/catalog/memory.rs:
##
@@ -47,6 +47,20 @@ fn memory_catalog_dereg_nonempty_schema() {
assert!(cat.deregister_schema("foo", true).unw
1 - 100 of 153 matches
Mail list logo
