richardc-db opened a new pull request, #47252:
URL: https://github.com/apache/spark/pull/47252
### What changes were proposed in this pull request?
adds support for variant type in `InMemoryTableScan`, or `df.cache()` by
supporting writing the type to an inmemory buffer.
wayneguow commented on PR #47243:
URL: https://github.com/apache/spark/pull/47243#issuecomment-2213150614
Since https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/1.2.0 has
no significant improvements for Spark, I will close this PR. @LuciferYang
--
This is an automated message
wayneguow closed pull request #47243: [SPARK-48827] Upgrade `RoaringBitmap` to
1.2.0
URL: https://github.com/apache/spark/pull/47243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific com
HeartSaVioR closed pull request #47205: [SPARK-48800][CONNECT][SS] Deflake
ClientStreamingQuerySuite
URL: https://github.com/apache/spark/pull/47205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
HeartSaVioR commented on PR #47205:
URL: https://github.com/apache/spark/pull/47205#issuecomment-2213109564
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
HeartSaVioR commented on code in PR #47188:
URL: https://github.com/apache/spark/pull/47188#discussion_r1667893863
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:
##
@@ -94,10 +94,21 @@ class StateDataSource extends TableProv
wayneguow closed pull request #47042: [SPARK-48671][SQL][TESTS] Add test cases
for `Hex.hex`
URL: https://github.com/apache/spark/pull/47042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
LuciferYang opened a new pull request, #47251:
URL: https://github.com/apache/spark/pull/47251
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
srielau commented on PR #47186:
URL: https://github.com/apache/spark/pull/47186#issuecomment-2213069069
> I'm also reluctant like @yaooqinn 's comment.
>
> Could you summarize the status of other DBMS in the PR description first
for further discussion?
As noted above SparkSQL w
wayneguow commented on code in PR #47221:
URL: https://github.com/apache/spark/pull/47221#discussion_r1668016491
##
pom.xml:
##
@@ -2774,6 +2774,10 @@
com.google.http-client
google-http-client
+
Review Comment:
Thanks for the s
wayneguow closed pull request #47221: [SPARK-48814][BUILD] Upgrade `tink` to
1.14.0
URL: https://github.com/apache/spark/pull/47221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
itholic opened a new pull request, #47250:
URL: https://github.com/apache/spark/pull/47250
### What changes were proposed in this pull request?
This PR proposes to add dedicated error tests for Spark Connect
### Why are the changes needed?
To improve the test coverage
huaxingao commented on PR #44277:
URL: https://github.com/apache/spark/pull/44277#issuecomment-2213049751
@cloud-fan Thanks for the reply! I got around the problem by creating a
subclass `IcebergAnalysisException`, which takes a String errorMessage. I have
replaced all instances of `throw n
itholic commented on code in PR #47145:
URL: https://github.com/apache/spark/pull/47145#discussion_r1668000825
##
python/docs/source/development/logger.rst:
##
@@ -0,0 +1,149 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agre
zhengruifeng opened a new pull request, #47249:
URL: https://github.com/apache/spark/pull/47249
### What changes were proposed in this pull request?
Fix the default column name of `cast`, by comparing with classic
implementation
https://github.com/apache/spark/blob/9cf6dc873ff34412df6
LuciferYang commented on code in PR #47221:
URL: https://github.com/apache/spark/pull/47221#discussion_r1667988918
##
pom.xml:
##
@@ -2774,6 +2774,10 @@
com.google.http-client
google-http-client
+
Review Comment:
If that's the
itholic commented on code in PR #47145:
URL: https://github.com/apache/spark/pull/47145#discussion_r1667846742
##
python/docs/source/development/logger.rst:
##
@@ -0,0 +1,149 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agre
cloud-fan commented on PR #44277:
URL: https://github.com/apache/spark/pull/44277#issuecomment-2212965099
As a workaround, we can use reflection to invoke the main constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub a
cloud-fan commented on PR #44277:
URL: https://github.com/apache/spark/pull/44277#issuecomment-2212964123
@huaxingao let's add a new constructor to take plain error message and a
cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on t
LuciferYang closed pull request #47097: [SPARK-48720][SQL] Align the command
`ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2
URL: https://github.com/apache/spark/pull/47097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub a
wayneguow closed pull request #47220: [SPARK-48813][SQL][DOCS] Add a notice
that `mariadb` protocol does not apply when the database is `MariaDB`
URL: https://github.com/apache/spark/pull/47220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
wayneguow closed pull request #47217: [SPARK-48812][SQL][TESTS] Add some test
suites for `mariadb` jdbc connector
URL: https://github.com/apache/spark/pull/47217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
wayneguow commented on PR #47220:
URL: https://github.com/apache/spark/pull/47220#issuecomment-2212921061
@yaooqinn Thank you for reviewing `mariadb` related PRs, I think what you
said makes sense, so I will close these.
--
This is an automated message from the Apache Git Service.
To re
wayneguow commented on PR #47241:
URL: https://github.com/apache/spark/pull/47241#issuecomment-2212917146
@LuciferYang Thank you for the guidance, let me do some in-depth inspection
and update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, ple
wayneguow commented on code in PR #47221:
URL: https://github.com/apache/spark/pull/47221#discussion_r1667898111
##
pom.xml:
##
@@ -2774,6 +2774,10 @@
com.google.http-client
google-http-client
+
Review Comment:
Cause that `tink
panbingkun commented on PR #47247:
URL: https://github.com/apache/spark/pull/47247#issuecomment-2212913449
org.apache.spark.MapStatusesConvertBenchmark
JDK 17: https://github.com/panbingkun/spark/actions/runs/9832998545
JDK 21: https://github.com/panbingkun/spark/actions/runs/9833002646
wayneguow commented on code in PR #47221:
URL: https://github.com/apache/spark/pull/47221#discussion_r1667898111
##
pom.xml:
##
@@ -2774,6 +2774,10 @@
com.google.http-client
google-http-client
+
Review Comment:
Cause that `tink
Masykus commented on code in PR #47246:
URL: https://github.com/apache/spark/pull/47246#discussion_r1667894249
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala:
##
@@ -998,10 +998,27 @@ abstract class BinaryComparison extends BinaryOperat
Masykus commented on code in PR #47246:
URL: https://github.com/apache/spark/pull/47246#discussion_r1667894249
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala:
##
@@ -998,10 +998,27 @@ abstract class BinaryComparison extends BinaryOperat
LuciferYang commented on PR #47241:
URL: https://github.com/apache/spark/pull/47241#issuecomment-2212903055
What modifications are related to Spark? Can you add some details in the PR
description?
--
This is an automated message from the Apache Git Service.
To respond to the message,
uros-db commented on code in PR #46762:
URL: https://github.com/apache/spark/pull/46762#discussion_r1667892969
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##
@@ -657,57 +659,64 @@ public static Map
getCollationAwareDict(UTF8S
uros-db commented on code in PR #46762:
URL: https://github.com/apache/spark/pull/46762#discussion_r1667892372
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##
@@ -805,6 +805,18 @@ public static String[] getICULocaleNames() {
retur
uros-db commented on code in PR #46762:
URL: https://github.com/apache/spark/pull/46762#discussion_r1667891877
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##
@@ -657,57 +659,64 @@ public static Map
getCollationAwareDict(UTF8S
uros-db commented on code in PR #46762:
URL: https://github.com/apache/spark/pull/46762#discussion_r1667889626
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##
@@ -657,57 +659,64 @@ public static Map
getCollationAwareDict(UTF8S
uros-db commented on code in PR #46762:
URL: https://github.com/apache/spark/pull/46762#discussion_r1667889474
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##
@@ -657,57 +659,64 @@ public static Map
getCollationAwareDict(UTF8S
wayneguow commented on PR #47243:
URL: https://github.com/apache/spark/pull/47243#issuecomment-2212891122
Actually, related improvements are about `org.roaringbitmap.RoaringBitSet
`(A BitSet implementation based on `RoaringBitmap`) introduced in 1.1.0, but
it's not used in Spark currently.
panbingkun commented on PR #47222:
URL: https://github.com/apache/spark/pull/47222#issuecomment-2212889121
> Looks good.
>
> btw, is this the only feature we should silent the warning from pandas 2.x?
This is just what I found during the building of the `PySpark API docs`, and
beliefer closed pull request #47226: [SPARK-48820][SQL][DOC] Correct the
examples for Collate functions
URL: https://github.com/apache/spark/pull/47226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
LuciferYang commented on PR #47243:
URL: https://github.com/apache/spark/pull/47243#issuecomment-2212885179
Will this upgrade bring any benefits to Spark? Can you add relevant
information in the PR description?
--
This is an automated message from the Apache Git Service.
To respon
LuciferYang commented on code in PR #47221:
URL: https://github.com/apache/spark/pull/47221#discussion_r1667883208
##
pom.xml:
##
@@ -2774,6 +2774,10 @@
com.google.http-client
google-http-client
+
Review Comment:
Why does upgra
LuciferYang commented on PR #47242:
URL: https://github.com/apache/spark/pull/47242#issuecomment-2212877937
Merged into master for Spark 4.0. Thanks @wayneguow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
LuciferYang closed pull request #47242: [SPARK-48177][BUILD][FOLLOWUP] Update
parquet version in `sql-data-sources-parquet.md` doc
URL: https://github.com/apache/spark/pull/47242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
ulysses-you commented on code in PR #47224:
URL: https://github.com/apache/spark/pull/47224#discussion_r1667870238
##
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##
@@ -129,7 +129,15 @@ class QueryExecution(
qe.commandExecuted,
wayneguow commented on PR #47242:
URL: https://github.com/apache/spark/pull/47242#issuecomment-2212858228
cc @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
itholic commented on code in PR #47145:
URL: https://github.com/apache/spark/pull/47145#discussion_r1667846742
##
python/docs/source/development/logger.rst:
##
@@ -0,0 +1,149 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agre
panbingkun opened a new pull request, #47248:
URL: https://github.com/apache/spark/pull/47248
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
itholic commented on code in PR #47145:
URL: https://github.com/apache/spark/pull/47145#discussion_r1667840590
##
python/pyspark/logger/logger.py:
##
@@ -0,0 +1,150 @@
+# -*- encoding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contribu
beliefer commented on code in PR #47226:
URL: https://github.com/apache/spark/pull/47226#discussion_r1667832980
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala:
##
@@ -38,8 +38,8 @@ import org.apache.spark.sql.types._
Exam
beliefer commented on code in PR #47226:
URL: https://github.com/apache/spark/pull/47226#discussion_r1667832520
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala:
##
@@ -38,8 +38,8 @@ import org.apache.spark.sql.types._
Exam
panbingkun opened a new pull request, #47247:
URL: https://github.com/apache/spark/pull/47247
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
itholic commented on PR #47145:
URL: https://github.com/apache/spark/pull/47145#issuecomment-2212707736
> Should we consider making the Python side consistent with this naming
convention?
Yeah, the suggested short names looks reasonable to me. Let me address them.
Thanks!
--
This
HeartSaVioR commented on code in PR #47107:
URL: https://github.com/apache/spark/pull/47107#discussion_r1667805849
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -194,43 +207,35 @@ class PrefixKeyScanStateEncoder(
github-actions[bot] commented on PR #45349:
URL: https://github.com/apache/spark/pull/45349#issuecomment-2212655123
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #45552: [SPARK-47217][SQL] Fix
deduplicated expression resolution
URL: https://github.com/apache/spark/pull/45552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
Zawa-ll opened a new pull request, #47246:
URL: https://github.com/apache/spark/pull/47246
**What changes were proposed in this pull request?**
This PR addresses a type casting issue in Spark SQL where comparing a string
column to an integer value results in an empty result set. The chang
jerryzhou196 closed pull request #47245: Documentation
URL: https://github.com/apache/spark/pull/47245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: r
jerryzhou196 opened a new pull request, #47245:
URL: https://github.com/apache/spark/pull/47245
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### Ho
huaxingao commented on code in PR #46707:
URL: https://github.com/apache/spark/pull/46707#discussion_r1667739136
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -2970,9 +2971,14 @@ class AstBuilder extends DataTypeAstBuilder with
SQLC
huaxingao commented on code in PR #46707:
URL: https://github.com/apache/spark/pull/46707#discussion_r1667738141
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -2970,9 +2971,14 @@ class AstBuilder extends DataTypeAstBuilder with
SQLC
huaxingao commented on code in PR #47233:
URL: https://github.com/apache/spark/pull/47233#discussion_r1667732599
##
sql/core/src/main/scala/org/apache/spark/sql/UpdateWriter.scala:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
huaxingao commented on code in PR #47233:
URL: https://github.com/apache/spark/pull/47233#discussion_r1667732553
##
sql/core/src/main/scala/org/apache/spark/sql/UpdateWriter.scala:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
huaxingao commented on code in PR #47233:
URL: https://github.com/apache/spark/pull/47233#discussion_r1667732474
##
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -4136,6 +4136,26 @@ class Dataset[T] private[sql](
new MergeIntoWriter[T](table, this, condi
huaxingao commented on code in PR #47233:
URL: https://github.com/apache/spark/pull/47233#discussion_r1667732253
##
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -4136,6 +4136,26 @@ class Dataset[T] private[sql](
new MergeIntoWriter[T](table, this, condi
huaxingao commented on code in PR #47233:
URL: https://github.com/apache/spark/pull/47233#discussion_r1667731520
##
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -4136,6 +4136,26 @@ class Dataset[T] private[sql](
new MergeIntoWriter[T](table, this, condi
sauletawil commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-2212457710
+1
@pkotikalapudi
What is the status of this PR and will it make it into Spark 4.0 GA release?
Great initiative - I am very surprised Spark does not have an adaptation of
DRA
65 matches
Mail list logo
