LuciferYang commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763308899
If it can't be fixed quickly, can we revert this patch or ignore
`KafkaSourceStressSuite` first? The recent CI is almost impossible to succeed.
--
This is an automated messa
LuciferYang commented on PR #43194:
URL: https://github.com/apache/spark/pull/43194#issuecomment-1763312763
@xiongbo-sjtu Would it be convenient to submit a similar patch for
branch-3.5? I found that this case would also likely fail in branch-3.5.
https://github.com/apache/spark/actio
LuciferYang opened a new pull request, #43377:
URL: https://github.com/apache/spark/pull/43377
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1359852798
##
add-dockerfiles.sh:
##
@@ -44,12 +48,20 @@ for TAG in $TAGS; do
if echo $TAG | grep -q "r-"; then
OPTS+=" --sparkr"
fi
+
+if echo $TAG
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1359852744
##
3.5.0/scala2.12-java11-python3-r-ubuntu/Dockerfile:
##
@@ -20,7 +20,10 @@ USER root
RUN set -ex; \
apt-get update; \
-apt-get install -y python3 pytho
LuciferYang commented on PR #43362:
URL: https://github.com/apache/spark/pull/43362#issuecomment-1763351199
`KafkaSourceOffsetSuite` still failed with this fix.
https://github.com/dengziming/spark/actions/runs/6506157421/job/17671193820
```[info] KafkaSourceOffsetSuite:
[inf
vakarisbk commented on code in PR #56:
URL: https://github.com/apache/spark-docker/pull/56#discussion_r1359853632
##
.github/workflows/test.yml:
##
@@ -37,12 +37,15 @@ on:
- 3.3.0
java:
description: 'The Java version of Spark image.'
-default: 11
zhaomin1423 commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1359901730
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1756,6 +1759,8 @@ public static class AppShufflePartit
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1359905254
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1756,6 +1759,8 @@ public static class AppShufflePartit
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1359905254
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1756,6 +1759,8 @@ public static class AppShufflePartit
zhaomin1423 commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1359908418
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1756,6 +1759,8 @@ public static class AppShufflePartit
LuciferYang closed pull request #42198: [SPARK-44594][SS] Remove redundant
method parameter in kafka connector
URL: https://github.com/apache/spark/pull/42198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
LuciferYang commented on PR #42198:
URL: https://github.com/apache/spark/pull/42198#issuecomment-1763541871
Merged into master for Spark 4.0. Thanks @zhaomin1423 @HeartSaVioR and
@HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please l
HeartSaVioR commented on code in PR #43339:
URL: https://github.com/apache/spark/pull/43339#discussion_r1359983633
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##
@@ -109,6 +109,8 @@ class RocksDB(
dbOptions.setCreateIfMissing(true)
LuciferYang opened a new pull request, #43378:
URL: https://github.com/apache/spark/pull/43378
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
HyukjinKwon commented on code in PR #52:
URL: https://github.com/apache/spark-docker/pull/52#discussion_r1359986873
##
3.4.1/scala2.12-java11-ubuntu/entrypoint.sh:
##
@@ -77,6 +77,9 @@ elif ! [ -z "${SPARK_HOME+x}" ]; then
SPARK_CLASSPATH="$SPARK_HOME/conf:$SPARK_CLASSPATH";
github-actions[bot] commented on PR #41888:
URL: https://github.com/apache/spark/pull/41888#issuecomment-1763550979
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #41340: [SPARK-44318][BUILD] Remove
useless dependencies - javax.ws.rs-api
URL: https://github.com/apache/spark/pull/41340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
github-actions[bot] commented on PR #41162:
URL: https://github.com/apache/spark/pull/41162#issuecomment-1763550996
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #37588:
URL: https://github.com/apache/spark/pull/37588#issuecomment-1763551022
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
HyukjinKwon commented on PR #43314:
URL: https://github.com/apache/spark/pull/43314#issuecomment-1763552121
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #43314: [SPARK-45486][CONNECT] Make
add_artifact request idempotent
URL: https://github.com/apache/spark/pull/43314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
HyukjinKwon commented on PR #43367:
URL: https://github.com/apache/spark/pull/43367#issuecomment-1763553295
@xieshuaihu mind taking a look at
https://github.com/apache/spark/pull/43367/checks?check_run_id=17671719708?
--
This is an automated message from the Apache Git Service.
To respond
HyukjinKwon commented on code in PR #43378:
URL: https://github.com/apache/spark/pull/43378#discussion_r1359989990
##
dev/create-release/release-build.sh:
##
@@ -432,14 +432,14 @@ if [[ "$1" == "publish-snapshot" ]]; then
echo "" >> $tmp_settings
if [[ $PUBLISH_SCALA_2_1
HyukjinKwon commented on code in PR #43369:
URL: https://github.com/apache/spark/pull/43369#discussion_r1359990662
##
python/docs/source/user_guide/sql/type_conversions.rst:
##
@@ -0,0 +1,106 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contrib
HyukjinKwon commented on code in PR #43369:
URL: https://github.com/apache/spark/pull/43369#discussion_r1359990922
##
python/docs/source/user_guide/sql/type_conversions.rst:
##
@@ -0,0 +1,106 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contrib
HyukjinKwon commented on code in PR #43369:
URL: https://github.com/apache/spark/pull/43369#discussion_r1359991057
##
python/docs/source/user_guide/sql/type_conversions.rst:
##
@@ -0,0 +1,106 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+or more contrib
HyukjinKwon commented on code in PR #43368:
URL: https://github.com/apache/spark/pull/43368#discussion_r1359991579
##
sql/api/src/main/scala/org/apache/spark/sql/catalyst/analysis/alreadyExistException.scala:
##
@@ -31,7 +31,7 @@ class DatabaseAlreadyExistsException(db: String)
HyukjinKwon commented on PR #42762:
URL: https://github.com/apache/spark/pull/42762#issuecomment-1763558493
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #42762: [SPARK-45476][SQL][FOLLOWUP] Raise
exception directly instead of calling `resolveColumnsByPosition`
URL: https://github.com/apache/spark/pull/42762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
LuciferYang commented on code in PR #43378:
URL: https://github.com/apache/spark/pull/43378#discussion_r1359994898
##
dev/create-release/release-build.sh:
##
@@ -432,14 +432,14 @@ if [[ "$1" == "publish-snapshot" ]]; then
echo "" >> $tmp_settings
if [[ $PUBLISH_SCALA_2_1
LuciferYang opened a new pull request, #43379:
URL: https://github.com/apache/spark/pull/43379
This reverts commit d1bd21a2a219ebe6c5ac3fcb1e17db75af3c670c.
### What changes were proposed in this pull request?
This pr aims to revert SPARK-45502 to make the test case
`KafkaSourceStr
LuciferYang commented on PR #43379:
URL: https://github.com/apache/spark/pull/43379#issuecomment-1763581146
https://github.com/apache/spark/pull/43362
I see that the author is trying to fix it, we can also wait a little longer.
But since the sbt test is in a fail-fast mode, once `Kafk
HeartSaVioR commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763584923
> this failed test KafkaSourceStressSuite is also flaky in my local env, I'm
investigating this.
Let's be clear about this. Did you mean KafkaSourceStressSuite is also flaky
wi
beliefer commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360011335
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1933,6 +1946,29 @@ public RoaringBitmap getMapTracker() {
pan3793 commented on code in PR #43338:
URL: https://github.com/apache/spark/pull/43338#discussion_r1360011693
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -2001,8 +2001,10 @@ object SQLConf {
buildConf("spark.sql.streaming.stateStore.co
zhaomin1423 commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360016315
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1933,6 +1946,29 @@ public RoaringBitmap getMapTracker(
beliefer closed pull request #43363: [SPARK-45514][SQL][MLLIB] Replace
`scala.runtime.Tuple3Zipped` to `scala.collection.LazyZip3`
URL: https://github.com/apache/spark/pull/43363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
zhaomin1423 commented on PR #42198:
URL: https://github.com/apache/spark/pull/42198#issuecomment-1763616257
> Merged into master for Spark 4.0. Thanks @zhaomin1423 @HeartSaVioR and
@HyukjinKwon
Thanks for your help
--
This is an automated message from the Apache Git Service.
To res
LuciferYang commented on PR #43330:
URL: https://github.com/apache/spark/pull/43330#issuecomment-1763633049
> Details
cc @zhengruifeng Could we possibly backport `free_disk_space_container` to
branc-3.5?
--
This is an automated message from the Apache Git Service.
To respond to the
zhengruifeng opened a new pull request, #43380:
URL: https://github.com/apache/spark/pull/43380
### What changes were proposed in this pull request?
Validate Vectors with built-in function
### Why are the changes needed?
with built-in function, the logic might be optimized fu
cloud-fan commented on PR #43236:
URL: https://github.com/apache/spark/pull/43236#issuecomment-1763663144
I think the pyspark test failure is unrelated and we can merge.
@vitaliili-db how far shall we backport? is it a day-1 bug?
--
This is an automated message from the Apache Git Service
zhengruifeng commented on code in PR #43381:
URL: https://github.com/apache/spark/pull/43381#discussion_r1360065756
##
.github/workflows/build_and_test.yml:
##
@@ -407,6 +407,8 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
rest
LuciferYang commented on PR #43381:
URL: https://github.com/apache/spark/pull/43381#issuecomment-1763672247
https://github.com/apache/spark/actions/runs/6510937376
found `No space left on device` log on branch-3.5 ci:
https://github.com/apache/spark/assets/1475305/5777d152-68cd-4995
zhengruifeng commented on code in PR #43381:
URL: https://github.com/apache/spark/pull/43381#discussion_r1360066932
##
.github/workflows/build_and_test.yml:
##
@@ -407,6 +407,8 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
rest
LuciferYang commented on code in PR #43381:
URL: https://github.com/apache/spark/pull/43381#discussion_r1360066949
##
.github/workflows/build_and_test.yml:
##
@@ -407,6 +407,8 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
resto
cloud-fan commented on code in PR #43359:
URL: https://github.com/apache/spark/pull/43359#discussion_r1360067208
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##
@@ -29,48 +29,50 @@ import org.apache.spark.sql.internal.SQLConf
cloud-fan commented on PR #42725:
URL: https://github.com/apache/spark/pull/42725#issuecomment-1763675871
yea they are unrelated, thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
cloud-fan closed pull request #42725: [SPARK-45009][SQL] Decorrelate predicate
subqueries in join condition
URL: https://github.com/apache/spark/pull/42725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
LuciferYang commented on PR #43330:
URL: https://github.com/apache/spark/pull/43330#issuecomment-1763682065
> > Details
>
> cc @zhengruifeng Could we possibly backport `free_disk_space_container` to
branc-3.5?
backport [SPARK-44619](https://issues.apache.org/jira/browse/SPARK-4
HyukjinKwon commented on code in PR #43354:
URL: https://github.com/apache/spark/pull/43354#discussion_r1360080331
##
common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala:
##
@@ -330,58 +341,78 @@ private[deploy] object IvyTestUtils {
/**
* Creates a sui
HyukjinKwon commented on code in PR #43381:
URL: https://github.com/apache/spark/pull/43381#discussion_r1360082517
##
.github/workflows/build_and_test.yml:
##
@@ -407,6 +407,8 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
resto
beliefer commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360085452
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1933,6 +1946,29 @@ public RoaringBitmap getMapTracker() {
anishshri-db commented on PR #43370:
URL: https://github.com/apache/spark/pull/43370#issuecomment-1763741620
Kafka issue seems be intermittent and not related to this change - was able
to ran locally
```
[info] No tests to run for mllib / Test / testOnly
[info] - stress test wit
anishshri-db commented on PR #43370:
URL: https://github.com/apache/spark/pull/43370#issuecomment-1763744409
Seems to be failing for other PRs too -
https://github.com/apache/spark/actions/runs/6527209920/job/17723040668
--
This is an automated message from the Apache Git Service.
To resp
mridulm commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360129540
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -481,7 +482,9 @@ void closeAndDeletePartitionsIfNeeded(
mridulm commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360130919
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1864,7 +1875,14 @@ private void finalizePartition() throws
HeartSaVioR commented on PR #43370:
URL: https://github.com/apache/spark/pull/43370#issuecomment-1763759789
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
HeartSaVioR commented on PR #43370:
URL: https://github.com/apache/spark/pull/43370#issuecomment-1763759671
I agree the failure is unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
HeartSaVioR closed pull request #43370: [SPARK-45539][SS] Add assert and log to
indicate watermark definition is required for streaming aggregation queries in
append mode
URL: https://github.com/apache/spark/pull/43370
--
This is an automated message from the Apache Git Service.
To respond t
dengziming commented on PR #43348:
URL: https://github.com/apache/spark/pull/43348#issuecomment-1763761255
> is also flaky with Kafka 3.4.1 or Kafka 3.6.0 in your local env
After some attempt, I find that there is a weird situation: deleted topic
will com back again. this is the root
beliefer commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360148902
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1933,6 +1946,29 @@ public RoaringBitmap getMapTracker() {
beliefer commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360150033
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -481,7 +482,9 @@ void closeAndDeletePartitionsIfNeeded(
Yikun commented on code in PR #52:
URL: https://github.com/apache/spark-docker/pull/52#discussion_r1358343497
##
3.4.1/scala2.12-java11-ubuntu/entrypoint.sh:
##
@@ -77,6 +77,9 @@ elif ! [ -z "${SPARK_HOME+x}" ]; then
SPARK_CLASSPATH="$SPARK_HOME/conf:$SPARK_CLASSPATH";
fi
cloud-fan commented on PR #43376:
URL: https://github.com/apache/spark/pull/43376#issuecomment-1763795534
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
cloud-fan closed pull request #43376: [SPARK-45491] Add missing SQLSTATES 2/2
URL: https://github.com/apache/spark/pull/43376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360165594
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -537,7 +540,9 @@ void closeAndDeleteOutdatedPartitions(
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360165920
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -1734,6 +1735,8 @@ public static class AppShufflePartit
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360165594
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -537,7 +540,9 @@ void closeAndDeleteOutdatedPartitions(
LuciferYang commented on code in PR #43371:
URL: https://github.com/apache/spark/pull/43371#discussion_r1360177836
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##
@@ -537,7 +540,9 @@ void closeAndDeleteOutdatedPartitions(
70 matches
Mail list logo
