[GitHub] [spark] Ngone51 commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used
Ngone51 commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871147518 I see. Make sense to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions
SparkQA removed a comment on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871031005 **[Test build #140414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140414/testReport)** for PR 32883 at commit [`fd44620`](https://github.com/apache/spark/commit/fd44620c381f96029872dea62880ccb349151bb8). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1
SparkQA commented on pull request #33105: URL: https://github.com/apache/spark/pull/33105#issuecomment-871147339 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44944/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33136: [SPARK-35932][SQL] Support extracting hour/minute/second from timestamp without time zone
SparkQA commented on pull request #33136: URL: https://github.com/apache/spark/pull/33136#issuecomment-871146874 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44941/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions
SparkQA commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871146611 **[Test build #140414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140414/testReport)** for PR 32883 at commit [`fd44620`](https://github.com/apache/spark/commit/fd44620c381f96029872dea62880ccb349151bb8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33150: [SPARK-35948][INFRA] Simplify release scripts by removing Spark 2.4/Java7 parts
dongjoon-hyun commented on a change in pull request #33150:
URL: https://github.com/apache/spark/pull/33150#discussion_r661181810
##
File path: dev/create-release/release-util.sh
##
@@ -226,11 +226,5 @@ function init_maven_sbt {
MVN="build/mvn -B"
MVN_EXTRA_OPTS=
SBT_OPTS=
- if [[ $JAVA_VERSION < "1.8." ]]; then
-# Needed for maven central when using Java 7.
-SBT_OPTS="-Dhttps.protocols=TLSv1.1,TLSv1.2"
-MVN_EXTRA_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g
-Dhttps.protocols=TLSv1.1,TLSv1.2"
-MVN="$MVN $MVN_EXTRA_OPTS"
- fi
Review comment:
It's okay to fail for Java 7. So, we don't need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions
SparkQA commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871145839 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44942/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing
SparkQA commented on pull request #33137: URL: https://github.com/apache/spark/pull/33137#issuecomment-871145684 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44943/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33150: [SPARK-35948][INFRA] Simplify release scripts by removing Spark 2.4/Java7 parts
dongjoon-hyun edited a comment on pull request #33150: URL: https://github.com/apache/spark/pull/33150#issuecomment-871143724 cc @gengliangwang , @srowen , @cloud-fan , @HyukjinKwon , @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you closed pull request #33015: [SPARK-35853][SQL] Remark the shuffle origin to ENSURE_REQUIREMENTS as far as possible
ulysses-you closed pull request #33015: URL: https://github.com/apache/spark/pull/33015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33150: [SPARK-35948][INFRA] Simplify release scripts by removing Spark 2.4/Java7 parts
SparkQA commented on pull request #33150: URL: https://github.com/apache/spark/pull/33150#issuecomment-871143765 **[Test build #140432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140432/testReport)** for PR 33150 at commit [`538c86a`](https://github.com/apache/spark/commit/538c86a01e89e1fbf4bb2f1c1371e136da106369). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33150: [SPARK-35948][INFRA] Simplify release scripts by removing Spark 2.4/Java7 parts
dongjoon-hyun commented on pull request #33150: URL: https://github.com/apache/spark/pull/33150#issuecomment-871143724 cc @gengliangwang , @srowen , @cloud-fan , @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33151: fixes bug for sparkContext stopped on client mode
AmplabJenkins commented on pull request #33151: URL: https://github.com/apache/spark/pull/33151#issuecomment-871143586 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle
mridulm commented on a change in pull request #33034: URL: https://github.com/apache/spark/pull/33034#discussion_r661178449 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java ## @@ -222,7 +223,7 @@ public void sendMergedBlockMetaReq( handler.addRpcRequest(requestId, callback); RpcChannelListener listener = new RpcChannelListener(requestId, callback); channel.writeAndFlush( - new MergedBlockMetaRequest(requestId, appId, shuffleId, reduceId)).addListener(listener); + new MergedBlockMetaRequest(requestId, appId, shuffleId, shuffleSequenceId, reduceId)).addListener(listener); Review comment: I did have a theoretical concern regarding pushing blocks actually - not reading (that, as discussed above, should be fine) : and I had discussed it with @otterc and @zhouyejoe The scenario is as follows: Suppose a stage is cancelled/failed and a reattempt starts - with one or more tasks of the failed stage still running and pushing data. If ESS "knows" of a new shuffle sequence id (due to some other task from new stage attempt pushing data), it can reject all pushes for the 'older' sequence ids for the shuffle id. The current code flow does not prevent the push of old stage task from completing - but also does not cause any incorrect shuffle reads in child stages (as earlier stage attempt has failed, it will never 'finalize' that shuffle output - and so never added to MapOutputTracker as merged location). But we were debating on whether the added validation while processing push blocks is worth it (this will require propagating the shuffle seq id to those requests as well). Thoughts @Ngone51 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle
mridulm commented on a change in pull request #33034: URL: https://github.com/apache/spark/pull/33034#discussion_r661178449 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java ## @@ -222,7 +223,7 @@ public void sendMergedBlockMetaReq( handler.addRpcRequest(requestId, callback); RpcChannelListener listener = new RpcChannelListener(requestId, callback); channel.writeAndFlush( - new MergedBlockMetaRequest(requestId, appId, shuffleId, reduceId)).addListener(listener); + new MergedBlockMetaRequest(requestId, appId, shuffleId, shuffleSequenceId, reduceId)).addListener(listener); Review comment: I did have a theoretical concern regarding pushing blocks actually - not reading (that, as discussed above, should be fine) : and I had discussed it with @otterc and @zhouyejoe The scenario is as follows: Suppose a stage is cancelled/failed and a reattempt starts - with one or more tasks of the failed stage still running and pushing data. If ESS "knows" of a new shuffle sequence id (due to some other task from new stage attempt pushing data), it can reject all pushes for the 'older' sequence ids. The current code flow does not prevent the push of old stage task from completing - but also does not cause any incorrect shuffle reads in child stages (as earlier stage attempt has failed, it will never 'finalize' that shuffle output - and so never added to MapOutputTracker as merged location). But we were debating on whether the added validation while processing push blocks is worth it (this will require propagating the shuffle seq id to those requests as well). Thoughts @Ngone51 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunpe opened a new pull request #33151: fixes bug for sparkContext stopped on client mode
sunpe opened a new pull request #33151: URL: https://github.com/apache/spark/pull/33151 ### What changes were proposed in this pull request? Fixes a bug that the spark context was stopped while application is started on client mode. ### Why are the changes needed? Due to pr https://github.com/apache/spark/commit/c625eb4f9f970108d93bf3342c7ccb7ec873dc27#diff-f8564df81d845c0cd2f621bc2ed22761cbf9731f28cb2828d9cbd0491f4e7584. In client mode, the spark context will be stopped on application start. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually on the production environment in my company. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #33150: [SPARK-35948][INFRA] Simplify release scripts by removing Spark 2.4/Java7 parts
dongjoon-hyun opened a new pull request #33150: URL: https://github.com/apache/spark/pull/33150 ### What changes were proposed in this pull request? This PR aims to clean up Spark 2.4 and Java7 code path from the release scripts. ### Why are the changes needed? To simplify the logic. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? N/A -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing
MaxGekk closed pull request #33137: URL: https://github.com/apache/spark/pull/33137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing
MaxGekk commented on pull request #33137: URL: https://github.com/apache/spark/pull/33137#issuecomment-871139935 GA passed. Merging to master. Thank you, @cloud-fan for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
AmplabJenkins commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871138114 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44945/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
SparkQA commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871138101 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44945/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33149: [SPARK-35947][INFRA] Increase JVM stack size in release-build.sh
SparkQA commented on pull request #33149: URL: https://github.com/apache/spark/pull/33149#issuecomment-871136536 **[Test build #140431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140431/testReport)** for PR 33149 at commit [`f1fa0e2`](https://github.com/apache/spark/commit/f1fa0e21741f76b4ec14891cd3f606ed109c0578). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
mridulm commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871136009 Based on my limited understanding of standalone, this looks good to me. Would like others to comment as well @Ngone51 :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
AmplabJenkins removed a comment on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-871134686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33148: [SPARK-33298][CORE][FOLLOWUP] Add Unstable annotation to `FileCommitProtocol`
AmplabJenkins removed a comment on pull request #33148: URL: https://github.com/apache/spark/pull/33148#issuecomment-871134690 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
AmplabJenkins removed a comment on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871134692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32552: [SPARK-34819][SQL] MapType supports comparable semantics
AmplabJenkins removed a comment on pull request #32552: URL: https://github.com/apache/spark/pull/32552#issuecomment-871134689 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140411/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
AmplabJenkins removed a comment on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871134687 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used
mridulm commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871135005 @Ngone51 The problem is that fixing the two correctness pending issues requires protocol changes - which means serde issues if 3.3 (where this is fixed) tries to work with 3.2 (if it is released without the fixes) - client/ESS combinations (3.3 client with 3.2 ESS and 3.2 client with 3.3 ESS in future). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
AmplabJenkins commented on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-871134686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33148: [SPARK-33298][CORE][FOLLOWUP] Add Unstable annotation to `FileCommitProtocol`
AmplabJenkins commented on pull request #33148: URL: https://github.com/apache/spark/pull/33148#issuecomment-871134690 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
AmplabJenkins commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871134696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32552: [SPARK-34819][SQL] MapType supports comparable semantics
AmplabJenkins commented on pull request #32552: URL: https://github.com/apache/spark/pull/32552#issuecomment-871134689 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140411/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
AmplabJenkins commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871134691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
dongjoon-hyun commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871133359 Thanks. Ya, let's try with the latest one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"
MaxGekk closed pull request #33143: URL: https://github.com/apache/spark/pull/33143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33149: [SPARK-35947][INFRA] Increase JVM stack size in release-build.sh
dongjoon-hyun commented on pull request #33149: URL: https://github.com/apache/spark/pull/33149#issuecomment-871131578 How do you think about this, @gengliangwang and @HyukjinKwon ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"
MaxGekk commented on pull request #33143: URL: https://github.com/apache/spark/pull/33143#issuecomment-871131271 Thanks all for reviews. Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33136: [SPARK-35932][SQL] Support extracting hour/minute/second from timestamp without time zone
SparkQA commented on pull request #33136: URL: https://github.com/apache/spark/pull/33136#issuecomment-871131157 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44941/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing
SparkQA commented on pull request #33137: URL: https://github.com/apache/spark/pull/33137#issuecomment-871130009 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44943/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1
SparkQA commented on pull request #33105: URL: https://github.com/apache/spark/pull/33105#issuecomment-871129881 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44944/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #33149: [SPARK-35947][INFRA] Increase JVM stack size in release-build.sh
dongjoon-hyun opened a new pull request #33149: URL: https://github.com/apache/spark/pull/33149 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions
SparkQA commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871128957 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44942/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA removed a comment on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871070332 **[Test build #140422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140422/testReport)** for PR 32850 at commit [`6d8e915`](https://github.com/apache/spark/commit/6d8e915ca62806f511588134a57bbeca99361d86). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871127360 **[Test build #140422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140422/testReport)** for PR 32850 at commit [`6d8e915`](https://github.com/apache/spark/commit/6d8e915ca62806f511588134a57bbeca99361d86). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
wangyum commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871125876 Last question. How do other common modules use these errors? For example: spark-unsafe, spark-network-common. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
SparkQA removed a comment on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871035782 **[Test build #140415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140415/testReport)** for PR 33093 at commit [`673008d`](https://github.com/apache/spark/commit/673008d0132a50f10e4b5909bc4cda10b192af4f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871125623 **[Test build #140415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140415/testReport)** for PR 33093 at commit [`673008d`](https://github.com/apache/spark/commit/673008d0132a50f10e4b5909bc4cda10b192af4f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
viirya commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871122684 I'm not against to this point. I can change to latest guava and see what CI tells. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
SparkQA commented on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-871120244 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33148: [SPARK-33298][CORE][FOLLOWUP] Add Unstable annotation to `FileCommitProtocol`
SparkQA commented on pull request #33148: URL: https://github.com/apache/spark/pull/33148#issuecomment-871119876 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871119589 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44937/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32552: [SPARK-34819][SQL] MapType supports comparable semantics
SparkQA removed a comment on pull request #32552: URL: https://github.com/apache/spark/pull/32552#issuecomment-871011016 **[Test build #140411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140411/testReport)** for PR 32552 at commit [`29dd475`](https://github.com/apache/spark/commit/29dd475457d1285257478ca866cff600e3f34a26). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
SparkQA removed a comment on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871030899 **[Test build #140413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140413/testReport)** for PR 33093 at commit [`f221358`](https://github.com/apache/spark/commit/f221358daab3c8fcc3fb178b8c76ee0861bfbdae). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32552: [SPARK-34819][SQL] MapType supports comparable semantics
SparkQA commented on pull request #32552: URL: https://github.com/apache/spark/pull/32552#issuecomment-871118314 **[Test build #140411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140411/testReport)** for PR 32552 at commit [`29dd475`](https://github.com/apache/spark/commit/29dd475457d1285257478ca866cff600e3f34a26). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SortMapKeys(child: Expression) extends UnaryExpression with ExpectsInputTypes ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming
SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871117789 **[Test build #140413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140413/testReport)** for PR 33093 at commit [`f221358`](https://github.com/apache/spark/commit/f221358daab3c8fcc3fb178b8c76ee0861bfbdae). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33130: [SPARK-35928][BUILD] Upgrade ASM to 9.1
dongjoon-hyun commented on pull request #33130: URL: https://github.com/apache/spark/pull/33130#issuecomment-871116923 Here is some update. Although the Jenkins status is super noisy due to the timeout, we got the green light for the following at least. - Maven with Hadoop 2.7/Java11: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-jdk-11/ - SBT with Hadoop 3.2: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/ I'll update here when I collect more info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
gengliangwang closed pull request #33138: URL: https://github.com/apache/spark/pull/33138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
gengliangwang commented on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-871116358 Merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cfmcgrady commented on a change in pull request #33146: [WIP][SPARK-35912][SQL] Fix cast struct contains null value to string
cfmcgrady commented on a change in pull request #33146:
URL: https://github.com/apache/spark/pull/33146#discussion_r661147901
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -1098,29 +1100,37 @@ abstract class CastBase extends UnaryExpression with
TimeZoneAwareExpression wit
}
private def writeStructToStringBuilder(
- st: Seq[DataType],
+ st: Seq[StructField],
row: ExprValue,
buffer: ExprValue,
ctx: CodegenContext): Block = {
-val structToStringCode = st.zipWithIndex.map { case (ft, i) =>
- val fieldToStringCode = castToStringCode(ft, ctx)
- val field = ctx.freshVariable("field", ft)
- val fieldStr = ctx.freshVariable("fieldStr", StringType)
- val javaType = JavaCode.javaType(ft)
- code"""
- |${if (i != 0) code"""$buffer.append(",");""" else EmptyBlock}
- |if ($row.isNullAt($i)) {
Review comment:
When the actual value is null, for primitive type field,
`row.isNullAt(i)` return ture, but `row.getXXX` return a default value.
For exmaple:
```scala
val r = new
org.apache.spark.sql.catalyst.expressions.GenericInternalRow(Array(1, null))
println(r.getInt(0)) // 1
println(r.getInt(1)) // 0
println(r.isNullAt(1)) // true
```
so we cann't only check `row.isNullAt(i)` here, we need to do the same
logical like `BoundReference.doGenCode()`, add nullable check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
SparkQA commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871110977 **[Test build #140430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140430/testReport)** for PR 29326 at commit [`4e6da9c`](https://github.com/apache/spark/commit/4e6da9c6e730e5564c33012c2a1d72ac7c383cda). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1
AmplabJenkins removed a comment on pull request #33105: URL: https://github.com/apache/spark/pull/33105#issuecomment-870942164 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140363/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1
SparkQA commented on pull request #33105: URL: https://github.com/apache/spark/pull/33105#issuecomment-871110491 **[Test build #140429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140429/testReport)** for PR 33105 at commit [`ae4531b`](https://github.com/apache/spark/commit/ae4531bdfd4cff5a9bcb87c7cb7cd649b7c21986). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing
SparkQA commented on pull request #33137: URL: https://github.com/apache/spark/pull/33137#issuecomment-871110470 **[Test build #140428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140428/testReport)** for PR 33137 at commit [`85954ae`](https://github.com/apache/spark/commit/85954aef15ec30a14c4b4ec762d32b85d69bd133). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33133: [SPARK-35930][BUILD] Upgrade kinesis-client to 1.14.4
dongjoon-hyun commented on pull request #33133: URL: https://github.com/apache/spark/pull/33133#issuecomment-871110316 Since SPARK-34549 is reverted, I'll close this PR, @sarutak . Thank you for checking. After Guava issue is resolved, feel free to reopen this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33133: [SPARK-35930][BUILD] Upgrade kinesis-client to 1.14.4
dongjoon-hyun closed pull request #33133: URL: https://github.com/apache/spark/pull/33133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
AmplabJenkins removed a comment on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871109578 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44940/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
AmplabJenkins commented on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871109578 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44940/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
AmplabJenkins removed a comment on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871108990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec
AmplabJenkins removed a comment on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-871071885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
AmplabJenkins removed a comment on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871108995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
AmplabJenkins removed a comment on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871108992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140425/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress
AmplabJenkins removed a comment on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-871108991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
AmplabJenkins commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871108990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec
AmplabJenkins commented on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-871108994 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44931/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
AmplabJenkins commented on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871108992 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140425/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
AmplabJenkins commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871108997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress
AmplabJenkins commented on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-871109000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress
SparkQA commented on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-871107428 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44936/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33138: [SPARK-35937][SQL] Extracting date field from timestamp should work in ANSI mode
SparkQA commented on pull request #33138: URL: https://github.com/apache/spark/pull/33138#issuecomment-871106259 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33148: [SPARK-33298][CORE][FOLLOWUP] Add Unstable annotation to `FileCommitProtocol`
SparkQA commented on pull request #33148: URL: https://github.com/apache/spark/pull/33148#issuecomment-871106119 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871105704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44937/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #33105: [SPARK-35908][SQL] Remove repartition if the child maximum number of rows less than or equal to 1
wangyum commented on a change in pull request #33105:
URL: https://github.com/apache/spark/pull/33105#discussion_r661139555
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##
@@ -24,7 +24,7 @@ import
org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.plans._
-import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.plans.logical.{RepartitionOperation, _}
Review comment:
Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"
HyukjinKwon commented on a change in pull request #33143:
URL: https://github.com/apache/spark/pull/33143#discussion_r661139243
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2928,31 +2928,6 @@ object functions {
// DateTime functions
//
- /**
- * (Scala-specific) Creates a datetime interval
- *
- * @param years Number of years
- * @param months Number of months
- * @param weeks Number of weeks
- * @param days Number of days
- * @param hours Number of hours
- * @param mins Number of mins
- * @param secs Number of secs
- * @return A datetime interval
- * @group datetime_funcs
- * @since 3.2.0
- */
- def make_interval(
Review comment:
okie then i am good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33142: [SPARK-35940][SQL] Refactor EquivalentExpressions to make it more efficient
viirya commented on pull request #33142: URL: https://github.com/apache/spark/pull/33142#issuecomment-871104576 > Can you briefly introduce your idea? Sorting by height is stable and fast now. Basically, the steps are: 1. Propagate the `SubExprEliminationState` map for all subexprs (no needed to be sorted). Only create the value and isNull variables, don't do codegen yet. 2. Iterate all subexprs to do codegen. Because expression codegen will look at the map to replace subexprs, any subexpr in children will be replaced and chained. So we don't need to sort subexprs in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
HyukjinKwon edited a comment on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871104226 Thank you @dongjoon-hyun !! > Since this is only at utill.py, we are still able to turn off the pined mode by PYSPARK_PIN_THREAD=false via java_gateway and Py4JServer.scala, right? Yes. This fix is in case other projects (e.g., Zeppelin) create the Java Gateway by themselves (and set it to `SparkContext`). In this case, `PYSPARK_PIN_THREAD` env won't be respected. (server side) I fixed `util.py` to respect the created the Java Gateway (instead of `PYSPARK_PIN_THREAD` environment variable) so it won't cause breakage in the case. (client side) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
HyukjinKwon commented on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871104226 Thank you @dongjoon-hyun !! > Since this is only at utill.py, we are still able to turn off the pined mode by PYSPARK_PIN_THREAD=false via java_gateway and Py4JServer.scala, right? Yes. This fix is in case other projects (e.g., Zeppelin) create the Java Gateway by themselves (and set it to `SparkContext`). In this case, `PYSPARK_PIN_THREAD` env won't be respected. (server side) I fixed `util.py` to respect the created the Java Gateway so it won't cause breakage in the case. (client side) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
dongjoon-hyun commented on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871104019 Thank you, @HyukjinKwon and @WeichenXu123 . Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
dongjoon-hyun closed pull request #33147: URL: https://github.com/apache/spark/pull/33147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya closed pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin
viirya closed pull request #32980: URL: https://github.com/apache/spark/pull/32980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin
viirya commented on pull request #32980: URL: https://github.com/apache/spark/pull/32980#issuecomment-871102030 Thanks for review! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre
viirya commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871101530 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress
SparkQA commented on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-871100812 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44932/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
SparkQA commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871100625 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44933/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
SparkQA removed a comment on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871088117 **[Test build #140425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140425/testReport)** for PR 33147 at commit [`27199ac`](https://github.com/apache/spark/commit/27199acb240d16311a44cc953dbe1098fd9340bf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33147: [SPARK-35946][PYTHON] Respect Py4J server in InheritableThread API
SparkQA commented on pull request #33147: URL: https://github.com/apache/spark/pull/33147#issuecomment-871099454 **[Test build #140425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140425/testReport)** for PR 33147 at commit [`27199ac`](https://github.com/apache/spark/commit/27199acb240d16311a44cc953dbe1098fd9340bf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
SparkQA removed a comment on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871051176 **[Test build #140418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140418/testReport)** for PR 33028 at commit [`54ed99e`](https://github.com/apache/spark/commit/54ed99e5780c36c834d2a76b34adb80e01ce7d4a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA removed a comment on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871051277 **[Test build #140419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140419/testReport)** for PR 32850 at commit [`d73bb83`](https://github.com/apache/spark/commit/d73bb83fa61ad551f634018f6e8cbf3d2b45842c). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE
SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871098737 **[Test build #140419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140419/testReport)** for PR 32850 at commit [`d73bb83`](https://github.com/apache/spark/commit/d73bb83fa61ad551f634018f6e8cbf3d2b45842c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit
SparkQA commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871098634 **[Test build #140418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140418/testReport)** for PR 33028 at commit [`54ed99e`](https://github.com/apache/spark/commit/54ed99e5780c36c834d2a76b34adb80e01ce7d4a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
