cloud-fan commented on code in PR #43390:
URL: https://github.com/apache/spark/pull/43390#discussion_r1368202786
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala:
##
@@ -43,7 +43,8 @@ class MySQLIntegrationSuite extends
HeartSaVioR opened a new pull request, #43483:
URL: https://github.com/apache/spark/pull/43483
### What changes were proposed in this pull request?
Credit to @anish-db for the initial investigation and the fix.
This PR proposes to remove @abstractmethod annotation from `onQueryI
HeartSaVioR commented on PR #43483:
URL: https://github.com/apache/spark/pull/43483#issuecomment-1774517494
cc. @viirya @HyukjinKwon Please take a look. We would like to fix this in
master/3.5.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, pleas
MaxGekk commented on code in PR #43479:
URL: https://github.com/apache/spark/pull/43479#discussion_r1368181666
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -1553,6 +1553,11 @@
},
"sqlState" : "HY109"
},
+ "INVALID_DATABASE_NAME" : {
Review Co
LuciferYang opened a new pull request, #43482:
URL: https://github.com/apache/spark/pull/43482
### What changes were proposed in this pull request?
This pr replace `s.c.mutable.MapOps#retain` with
`s.c.mutable.MapOps#filterInPlace` due to `retain` has been marked as
deprecated since Scal
zhengruifeng commented on PR #43432:
URL: https://github.com/apache/spark/pull/43432#issuecomment-1774463708
some ml tests are flaky, so I think it is reasonable to move them to a
separate group. (I already moved `pyspark-ml-*` to a separate one).
I did some investigation (>1 year) be
zhengruifeng commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368143899
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -31,6 +31,15 @@
],
"sqlState" : "42702"
},
+ "AMBIGUOUS_COLUMN_REFERENCE" : {
+
zhengruifeng commented on PR #43380:
URL: https://github.com/apache/spark/pull/43380#issuecomment-1774455364
@srowen would you mind taking another look? I think all vector validation
related udfs are covered in this PR.
when we check the values, the performances are similar;
when w
cloud-fan commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368141330
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -31,6 +31,15 @@
],
"sqlState" : "42702"
},
+ "AMBIGUOUS_COLUMN_REFERENCE" : {
+"m
cloud-fan commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368142151
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:
##
@@ -539,4 +533,28 @@ trait ColumnResolutionHelper extends Loggi
LuciferYang commented on PR #43432:
URL: https://github.com/apache/spark/pull/43432#issuecomment-1774423837
@HyukjinKwon @zhengruifeng @dongjoon-hyun Do we need this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
LuciferYang commented on PR #43478:
URL: https://github.com/apache/spark/pull/43478#issuecomment-1774420428
Merged into master for Spark 4.0. Thanks @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
mridulm commented on code in PR #43387:
URL: https://github.com/apache/spark/pull/43387#discussion_r1368097008
##
core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferSecuritySuite.scala:
##
@@ -38,11 +38,20 @@ import org.apache.spark.internal.config.Network
imp
LuciferYang closed pull request #43478: [SPARK-45625][BUILD] Upgrade log4j to
2.21.0
URL: https://github.com/apache/spark/pull/43478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
mridulm commented on PR #43387:
URL: https://github.com/apache/spark/pull/43387#issuecomment-1774418178
Can you also fix the conflicts please ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
mridulm commented on code in PR #43387:
URL: https://github.com/apache/spark/pull/43387#discussion_r1368088945
##
core/src/test/scala/org/apache/spark/SslExternalShuffleServiceSuite.scala:
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or
mridulm commented on code in PR #43387:
URL: https://github.com/apache/spark/pull/43387#discussion_r1368097008
##
core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferSecuritySuite.scala:
##
@@ -38,11 +38,20 @@ import org.apache.spark.internal.config.Network
imp
LuciferYang commented on PR #43477:
URL: https://github.com/apache/spark/pull/43477#issuecomment-1774417049
Thanks @HyukjinKwon @mridulm and @beliefer ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
cloud-fan commented on code in PR #43403:
URL: https://github.com/apache/spark/pull/43403#discussion_r1368095915
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -2207,6 +2213,11 @@
"Must be deterministic."
]
},
+ "OPTION" : {
+
Hisoka-X commented on code in PR #42398:
URL: https://github.com/apache/spark/pull/42398#discussion_r1368090659
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ListAgg.scala:
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundati
panbingkun commented on code in PR #43479:
URL: https://github.com/apache/spark/pull/43479#discussion_r1368087736
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -1553,6 +1553,11 @@
},
"sqlState" : "HY109"
},
+ "INVALID_DATABASE_NAME" : {
Review
panbingkun commented on code in PR #43479:
URL: https://github.com/apache/spark/pull/43479#discussion_r1368087736
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -1553,6 +1553,11 @@
},
"sqlState" : "HY109"
},
+ "INVALID_DATABASE_NAME" : {
Review
ulysses-you commented on code in PR #43435:
URL: https://github.com/apache/spark/pull/43435#discussion_r1368087491
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##
@@ -700,14 +723,38 @@ case class HashShuffleSpec(
}
}
-
ulysses-you commented on code in PR #43435:
URL: https://github.com/apache/spark/pull/43435#discussion_r1368087357
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##
@@ -295,19 +284,53 @@ case class HashPartitioning(expressions: Se
ulysses-you commented on code in PR #43435:
URL: https://github.com/apache/spark/pull/43435#discussion_r1368085777
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##
@@ -700,14 +723,38 @@ case class HashShuffleSpec(
}
}
-
ulysses-you commented on code in PR #43435:
URL: https://github.com/apache/spark/pull/43435#discussion_r1368085648
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##
@@ -700,14 +723,38 @@ case class HashShuffleSpec(
}
}
-
beliefer commented on PR #43464:
URL: https://github.com/apache/spark/pull/43464#issuecomment-1774353533
@srowen Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
cloud-fan commented on PR #36564:
URL: https://github.com/apache/spark/pull/36564#issuecomment-1774351335
I'm surprised that iceberg does not overwrite
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/BatchWrite.java#L63
cc @h
cloud-fan closed pull request #43466: [SPARK-45616][CORE] Avoid ParVector,
which does not propagate ThreadLocals or SparkSession
URL: https://github.com/apache/spark/pull/43466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
cloud-fan commented on PR #43466:
URL: https://github.com/apache/spark/pull/43466#issuecomment-1774348902
The streaming test failure is unrelated, merging to master/3.5, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
beliefer commented on code in PR #43308:
URL: https://github.com/apache/spark/pull/43308#discussion_r1368071970
##
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecMapper.java:
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Sof
cloud-fan commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368069577
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:
##
@@ -490,39 +477,46 @@ trait ColumnResolutionHelper extends Logg
zhengruifeng commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368069560
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -31,6 +31,15 @@
],
"sqlState" : "42702"
},
+ "AMBIGUOUS_COLUMN_REFERENCE" : {
+
dengziming opened a new pull request, #43481:
URL: https://github.com/apache/spark/pull/43481
### What changes were proposed in this pull request?
Assign the name `UNSUPPORTED_DATA_TYPE_FOR_SIZE_FUNCTION` to the legacy
error class `_LEGACY_ERROR_TEMP_2156`.
Assign the name `UNEXPECTED_
HyukjinKwon commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368058154
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:
##
@@ -490,39 +477,46 @@ trait ColumnResolutionHelper extends Lo
HyukjinKwon commented on code in PR #43465:
URL: https://github.com/apache/spark/pull/43465#discussion_r1368057545
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:
##
@@ -539,4 +533,28 @@ trait ColumnResolutionHelper extends Log
zhengruifeng opened a new pull request, #43480:
URL: https://github.com/apache/spark/pull/43480
### What changes were proposed in this pull request?
Improve `vector_to_array` for `.mllib.linalg.SparseVector`
### Why are the changes needed?
`vector_to_array` supports both `.ml.lin
HyukjinKwon commented on PR #43456:
URL: https://github.com/apache/spark/pull/43456#issuecomment-1774309585
cc @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
HyukjinKwon commented on PR #43466:
URL: https://github.com/apache/spark/pull/43466#issuecomment-1774303618
cc @MaxGekk FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #43470: [SPARK-45620][PYTHON] Fix user-facing
APIs related to Python UDTF to use camelCase
URL: https://github.com/apache/spark/pull/43470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
HyukjinKwon commented on PR #43470:
URL: https://github.com/apache/spark/pull/43470#issuecomment-1774298126
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #43462: [SPARK-44753][PYTHON][CONNECT] XML:
pyspark sql xml reader writer
URL: https://github.com/apache/spark/pull/43462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
HyukjinKwon commented on PR #43462:
URL: https://github.com/apache/spark/pull/43462#issuecomment-1774289707
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368037503
##
python/pyspark/sql/tests/test_types.py:
##
@@ -1579,6 +1579,12 @@ def test_row_without_field_sorting(self):
self.assertEqual(r, expected)
self.
HyukjinKwon commented on code in PR #43474:
URL: https://github.com/apache/spark/pull/43474#discussion_r1368037255
##
sql/core/src/test/scala/org/apache/spark/sql/types/DataTypeTest.scala:
##
@@ -0,0 +1,68 @@
+/*
Review Comment:
Let's move the tests to `DataTypeSuite.scala`
HyukjinKwon closed pull request #43477: [SPARK-45624][CORE][TESTS] Use
`AccessibleObject#canAccess` instead of `AccessibleObject#isAccessible`
URL: https://github.com/apache/spark/pull/43477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
HyukjinKwon commented on PR #43477:
URL: https://github.com/apache/spark/pull/43477#issuecomment-1774281301
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
zhengruifeng commented on PR #43380:
URL: https://github.com/apache/spark/pull/43380#issuecomment-1774280453
on second thought, let's let it alone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
zhengruifeng closed pull request #43380: [SPARK-45547][ML] Validate Vectors
with built-in function
URL: https://github.com/apache/spark/pull/43380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
Hisoka-X commented on PR #42004:
URL: https://github.com/apache/spark/pull/42004#issuecomment-1774251565
@cloud-fan @viirya Could you help to review this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
github-actions[bot] commented on PR #41986:
URL: https://github.com/apache/spark/pull/41986#issuecomment-1774245092
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #42004:
URL: https://github.com/apache/spark/pull/42004#issuecomment-1774245078
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
mstebelev commented on PR #36564:
URL: https://github.com/apache/spark/pull/36564#issuecomment-1774214202
Hi @AngersZh. I came across problems with this changes after updating to
spark 3.4
I write data to iceberg table with S3 backend and the data upload happens in
dataWriter.commit(
MaxGekk commented on code in PR #43479:
URL: https://github.com/apache/spark/pull/43479#discussion_r1367943424
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -1553,6 +1553,11 @@
},
"sqlState" : "HY109"
},
+ "INVALID_DATABASE_NAME" : {
Review Co
LuciferYang closed pull request #41982: [SPARK-44407][BUILD] Add a new Scala
checkstyle rule to prohibit using `enum` as a variable or function name
URL: https://github.com/apache/spark/pull/41982
--
This is an automated message from the Apache Git Service.
To respond to the message, please l
LuciferYang commented on PR #43467:
URL: https://github.com/apache/spark/pull/43467#issuecomment-1774150903
I discovered a reentrancy issue during the locally testing, I am trying to
further fix it ...
https://github.com/apache/commons-crypto/pull/259
--
This is an automated messag
viirya commented on PR #43467:
URL: https://github.com/apache/spark/pull/43467#issuecomment-1774150288
Thanks. We can wait for 1.2.1 release then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
srowen closed pull request #43472:
[SPARK-45610][BUILD][CORE][SQL][SS][CONNECT][GRAPHX][DSTREAM][ML][MLLIB][K8S][YARN][SHELL][PYTHON][R][AVRO][UI][EXAMPLES]
Fix the compilation warning "Auto-application to `()` is deprecated" and turn
it into a compilation error
URL: https://github.com/apache/
srowen commented on PR #43472:
URL: https://github.com/apache/spark/pull/43472#issuecomment-1774140984
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1367926881
##
versions.json:
##
@@ -1,9 +1,38 @@
{
"versions": [
+{
+ "path": "3.5.0/scala2.12-java17-python3-ubuntu",
+ "tags": [
+"3.5.0-scala2.12-j
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1367926824
##
testing/testing.sh:
##
@@ -61,7 +61,8 @@ function remove_network() {
# Find and kill any remaining containers attached to the network
function cleanup() {
-
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1367926834
##
versions.json:
##
@@ -1,9 +1,38 @@
{
"versions": [
+{
+ "path": "3.5.0/scala2.12-java17-python3-ubuntu",
+ "tags": [
+"3.5.0-scala2.12-j
anthonywainer commented on PR #43474:
URL: https://github.com/apache/spark/pull/43474#issuecomment-1774127262
@HyukjinKwon I have re-opened the PR, could you check please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and us
vakarisbk commented on PR #56:
URL: https://github.com/apache/spark-docker/pull/56#issuecomment-1774127165
> Please also make sure:
>
> 1. All dockerfiles and entrypoint.sh should be generated by
`add-dockerfiles.sh`
All dockerfiles and entrypoints were generated using the add-
Yikun commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1358358604
##
add-dockerfiles.sh:
##
@@ -44,12 +48,20 @@ for TAG in $TAGS; do
if echo $TAG | grep -q "r-"; then
OPTS+=" --sparkr"
fi
+
+if echo $TAG | gr
zhaomin1423 commented on PR #43439:
URL: https://github.com/apache/spark/pull/43439#issuecomment-1774103666
> > What I am confused about is, how should we check the effect after gc is
triggered
>
> Construct a case where `SparkResult` won't be manually closed, and check
if after regi
panbingkun commented on PR #43479:
URL: https://github.com/apache/spark/pull/43479#issuecomment-1774101171
cc @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
panbingkun commented on PR #43479:
URL: https://github.com/apache/spark/pull/43479#issuecomment-1774100838
Before:
https://github.com/apache/spark/assets/15246973/e7a59837-4f14-4f09-872a-913d78006ede";>
After:
https://github.com/apache/spark/assets/15246973/fa141a37-53b5-42eb-9ec
panbingkun opened a new pull request, #43479:
URL: https://github.com/apache/spark/pull/43479
### What changes were proposed in this pull request?
The pr aims to
1.fix variable name of error-class:
2.assign names to the error class _LEGACY_ERROR_TEMP_1055
### Why are the chan
LuciferYang commented on PR #43467:
URL: https://github.com/apache/spark/pull/43467#issuecomment-1774081676
> Interesting. According to the replies on
[CRYPTO-169](https://issues.apache.org/jira/browse/CRYPTO-169), Commons Crypto
has never supported Apple silicon.
Fixed: https://gith
panbingkun commented on code in PR #37588:
URL: https://github.com/apache/spark/pull/37588#discussion_r1367896387
##
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowTablesSuite.scala:
##
@@ -18,8 +18,9 @@
package org.apache.spark.sql.execution.command.v2
panbingkun commented on code in PR #37588:
URL: https://github.com/apache/spark/pull/37588#discussion_r1367896333
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala:
##
@@ -53,4 +82,95 @@ case class ShowTablesExec(
case _ => fals
panbingkun commented on code in PR #37588:
URL: https://github.com/apache/spark/pull/37588#discussion_r1367896135
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala:
##
@@ -53,4 +82,95 @@ case class ShowTablesExec(
case _ => fals
panbingkun commented on code in PR #37588:
URL: https://github.com/apache/spark/pull/37588#discussion_r1367896086
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala:
##
@@ -33,15 +41,36 @@ case class ShowTablesExec(
output: Seq[Att
LuciferYang opened a new pull request, #43478:
URL: https://github.com/apache/spark/pull/43478
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
LuciferYang commented on PR #43477:
URL: https://github.com/apache/spark/pull/43477#issuecomment-1774063380
> Surprised this is the only place :-)
Yes, I'm quite surprised as well :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log o
Yikun commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1367866420
##
versions.json:
##
@@ -1,9 +1,38 @@
{
"versions": [
+{
+ "path": "3.5.0/scala2.12-java17-python3-ubuntu",
+ "tags": [
+"3.5.0-scala2.12-java1
mridulm commented on PR #43386:
URL: https://github.com/apache/spark/pull/43386#issuecomment-1774029570
The test failure is unrelated to this PR.
Merging to master.
Thanks for fixing this @hasnain-db !
--
This is an automated message from the Apache Git Service.
To respond to the
mridulm closed pull request #43386: [SPARK-45541][CORE] Add SSLFactory
URL: https://github.com/apache/spark/pull/43386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubs
LuciferYang commented on PR #43439:
URL: https://github.com/apache/spark/pull/43439#issuecomment-1774019077
@zhaomin1423 If adding new tests is indeed difficult, do you have a manual
way to confirm it is effective?
--
This is an automated message from the Apache Git Service.
To re
80 matches
Mail list logo
