[GitHub] [spark] AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536249061 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536249061 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536249063 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111550/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536249063 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111550/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
SparkQA removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242073 **[Test build #111550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111550/testReport)** for PR 25933 at commit [`ce4d184`](https://github.com/apache/spark/commit/ce4d184e1cf42bea35f08de8ade8666bf217c48c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
SparkQA commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536248972 **[Test build #111550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111550/testReport)** for PR 25933 at commit [`ce4d184`](https://github.com/apache/spark/commit/ce4d184e1cf42bea35f08de8ade8666bf217c48c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248918 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111554/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248915 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248918 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111554/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248915 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248910 **[Test build #111554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111554/testReport)** for PR 25729 at commit [`fcfd7af`](https://github.com/apache/spark/commit/fcfd7afc1a60b94e82bca9c668acf63b9b2bbbee). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536247594 **[Test build #111554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111554/testReport)** for PR 25729 at commit [`fcfd7af`](https://github.com/apache/spark/commit/fcfd7afc1a60b94e82bca9c668acf63b9b2bbbee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111553/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536247272 **[Test build #111553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111553/testReport)** for PR 25729 at commit [`d070059`](https://github.com/apache/spark/commit/d070059f08dba815d0a7be271e85044e09a62cfc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248552 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248552 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111553/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536248548 **[Test build #111553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111553/testReport)** for PR 25729 at commit [`d070059`](https://github.com/apache/spark/commit/d070059f08dba815d0a7be271e85044e09a62cfc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tony-huangweiyi commented on a change in pull request #25949: [SPARK-29273] set the right peakExecutionMemory value when the task end
Tony-huangweiyi commented on a change in pull request #25949: [SPARK-29273] set
the right peakExecutionMemory value when the task end
URL: https://github.com/apache/spark/pull/25949#discussion_r329337420
##
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##
@@ -647,6 +647,13 @@ private[spark] object JsonProtocol {
val taskInfo = taskInfoFromJson(json \ "Task Info")
val executorMetrics = executorMetricsFromJson(json \ "Task Executor
Metrics")
val taskMetrics = taskMetricsFromJson(json \ "Task Metrics")
+val peakExecutionMemory = taskInfo.accumulables.find(accInfo => {
Review comment:
hi, @vanzin , I update this PR
the peakExecutionMemory value zero happend when replaying event log, so I
add the metrics value in SparkListenerTaskEnd json info and parsing it when
replaying and set the value back like other metrics do, this is a litter more
natural than the first commit for this PR
the unit test is the same as JsonProtocolSuite.testTaskMetrics
could you please kindly review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 012huang commented on a change in pull request #25949: [SPARK-29273] set the right peakExecutionMemory value when the task end
012huang commented on a change in pull request #25949: [SPARK-29273] set the
right peakExecutionMemory value when the task end
URL: https://github.com/apache/spark/pull/25949#discussion_r329337440
##
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##
@@ -647,6 +647,13 @@ private[spark] object JsonProtocol {
val taskInfo = taskInfoFromJson(json \ "Task Info")
val executorMetrics = executorMetricsFromJson(json \ "Task Executor
Metrics")
val taskMetrics = taskMetricsFromJson(json \ "Task Metrics")
+val peakExecutionMemory = taskInfo.accumulables.find(accInfo => {
Review comment:
hi, @vanzin , I update this PR
the peakExecutionMemory value zero happend when replaying event log, so I
add the metrics value in SparkListenerTaskEnd json info and parsing it when
replaying and set the value back like other metrics do, this is a litter more
natural than the first commit for this PR
the unit test is the same as JsonProtocolSuite.testTaskMetrics
could you please kindly review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536248124 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536248125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111549/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536248125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111549/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tony-huangweiyi commented on a change in pull request #25949: [SPARK-29273] set the right peakExecutionMemory value when the task end
Tony-huangweiyi commented on a change in pull request #25949: [SPARK-29273] set
the right peakExecutionMemory value when the task end
URL: https://github.com/apache/spark/pull/25949#discussion_r329337420
##
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##
@@ -647,6 +647,13 @@ private[spark] object JsonProtocol {
val taskInfo = taskInfoFromJson(json \ "Task Info")
val executorMetrics = executorMetricsFromJson(json \ "Task Executor
Metrics")
val taskMetrics = taskMetricsFromJson(json \ "Task Metrics")
+val peakExecutionMemory = taskInfo.accumulables.find(accInfo => {
Review comment:
hi, @vanzin , I update this PR
the peakExecutionMemory value zero happend when replaying event log, so I
add the metrics value in SparkListenerTaskEnd json info and parsing it when
replaying and set the value back like other metrics do, this is a litter more
natural than the first commit for this PR
the unit test is the same as JsonProtocolSuite.testTaskMetrics
could you please kindly review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536248124 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536248101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111547/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536248101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111547/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
SparkQA removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241667 **[Test build #111549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111549/testReport)** for PR 25954 at commit [`b056622`](https://github.com/apache/spark/commit/b056622068a300c44340fd02b3773fda81a52f73). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536248100 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
SparkQA commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536248073 **[Test build #111549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111549/testReport)** for PR 25954 at commit [`b056622`](https://github.com/apache/spark/commit/b056622068a300c44340fd02b3773fda81a52f73). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536248100 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
SparkQA removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536240130 **[Test build #111547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111547/testReport)** for PR 25797 at commit [`1919589`](https://github.com/apache/spark/commit/19195897ccf2c4facdfe2939340cb683b08acc5d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
SparkQA commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-536248026 **[Test build #111547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111547/testReport)** for PR 25797 at commit [`1919589`](https://github.com/apache/spark/commit/19195897ccf2c4facdfe2939340cb683b08acc5d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536247594 **[Test build #111554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111554/testReport)** for PR 25729 at commit [`fcfd7af`](https://github.com/apache/spark/commit/fcfd7afc1a60b94e82bca9c668acf63b9b2bbbee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536247272 **[Test build #111553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111553/testReport)** for PR 25729 at commit [`d070059`](https://github.com/apache/spark/commit/d070059f08dba815d0a7be271e85044e09a62cfc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536246875 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111546/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536246874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536246874 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536246875 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111546/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
SparkQA removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536240131 **[Test build #111546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111546/testReport)** for PR 25944 at commit [`49a0803`](https://github.com/apache/spark/commit/49a0803d68bf8e4f5911200b7a6200672d44b776). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
SparkQA commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536246817 **[Test build #111546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111546/testReport)** for PR 25944 at commit [`49a0803`](https://github.com/apache/spark/commit/49a0803d68bf8e4f5911200b7a6200672d44b776). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file
AmplabJenkins removed a comment on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file URL: https://github.com/apache/spark/pull/25869#issuecomment-536246195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file
AmplabJenkins removed a comment on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file URL: https://github.com/apache/spark/pull/25869#issuecomment-536246196 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16563/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file
AmplabJenkins commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file URL: https://github.com/apache/spark/pull/25869#issuecomment-536246195 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file
AmplabJenkins commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file URL: https://github.com/apache/spark/pull/25869#issuecomment-536246196 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16563/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file
SparkQA commented on issue #25869: [SPARK-29189][SQL] Add an option to ignore block locations when listing file URL: https://github.com/apache/spark/pull/25869#issuecomment-536246135 **[Test build #111552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111552/testReport)** for PR 25869 at commit [`e500bcd`](https://github.com/apache/spark/commit/e500bcd8b375a61ad9410511db5bcf4d92c79a9f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write
advancedxy commented on a change in pull request #25863:
[SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate
result and support concurrent file source write operations write to different
partitions in the same table.
URL: https://github.com/apache/spark/pull/25863#discussion_r329336071
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala
##
@@ -156,4 +189,66 @@ class PartitionedWriteSuite extends QueryTest with
SharedSparkSession {
}
}
}
+
+ test("Output path should be a staging output dir, whose last level path name
is jobId," +
+" when dynamicPartitionOverwrite is enabled") {
+withSQLConf(SQLConf.PARTITION_OVERWRITE_MODE.key ->
PartitionOverwriteMode.DYNAMIC.toString) {
+ withTable("t") {
+withSQLConf(SQLConf.FILE_COMMIT_PROTOCOL_CLASS.key ->
+ classOf[DetectCorrectOutputPathFileCommitProtocol].getName) {
+ Seq((1, 2)).toDF("a", "b")
+.write
+.partitionBy("b")
+.mode("overwrite")
+.saveAsTable("t")
+}
+ }
+}
+ }
+
+ test("Concurrent write to the same table with different partitions should be
possible") {
+withSQLConf(SQLConf.PARTITION_OVERWRITE_MODE.key ->
PartitionOverwriteMode.DYNAMIC.toString) {
+ withTable("t") {
+val sem = new Semaphore(0)
+Seq((1, 2)).toDF("a", "b")
+ .write
+ .partitionBy("b")
+ .mode("overwrite")
+ .saveAsTable("t")
+
+val df1 = spark.range(0, 10).map(x => (x, 1)).toDF("a", "b")
+val df2 = spark.range(0, 10).map(x => (x, 2)).toDF("a", "b")
+val dfs = Seq(df1, df2)
+
+var throwable: Option[Throwable] = None
+for (i <- 0 until 2) {
+ new Thread {
+override def run(): Unit = {
+ try {
+dfs(i)
+ .write
+ .mode("overwrite")
+ .insertInto("t")
+ } catch {
+case t: Throwable =>
+ throwable = Some(t)
+ } finally {
+sem.release()
+ }
+}
+ }.start()
+}
+// make sure writing table in two threads are executed.
+sem.acquire(2)
+throwable.foreach { t => throw improveStackTrace(t) }
+checkAnswer(spark.sql("select a, b from t where b = 1"), df1)
+checkAnswer(spark.sql("select a, b from t where b = 2"), df2)
+ }
+}
Review comment:
Ah, that's a limitation of `DataFrameWriter`, we may need to extend
`DataFrameWriter` to support that.
But currently, i think we can simply use the SQL syntax since we can use
`spark.sql` and get the same behaviour.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write
advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write operations write to different partitions in the same table. URL: https://github.com/apache/spark/pull/25863#discussion_r329335138 ## File path: core/src/main/scala/org/apache/spark/internal/io/FileSourceWriteDesc.scala ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.internal.io + +/** + * A class to describe the properties for file source write operation. + * + * @param isInsertIntoHadoopFsRelation whether is a InsertIntoHadoopFsRelation operation + * @param escapedStaticPartitionKVs static partition key and value pairs, which have been escaped + */ +case class FileSourceWriteDesc( Review comment: Since this might be used/subclassed by user, we may just use a normal class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write
advancedxy commented on a change in pull request #25863:
[SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate
result and support concurrent file source write operations write to different
partitions in the same table.
URL: https://github.com/apache/spark/pull/25863#discussion_r329335652
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala
##
@@ -103,103 +111,131 @@ case class InsertIntoHadoopFsRelationCommand(
val dynamicPartitionOverwrite = enableDynamicOverwrite && mode ==
SaveMode.Overwrite &&
staticPartitions.size < partitionColumns.length
+val appId = SparkEnv.get.conf.getAppId
+val jobId = java.util.UUID.randomUUID().toString
+
+val escapedStaticPartitionKVs = partitionColumns
+ .filter(c => staticPartitions.contains(c.name))
+ .map { attr =>
+val escapedKey = ExternalCatalogUtils.escapePathName(attr.name)
+val escapedValue =
ExternalCatalogUtils.escapePathName(staticPartitions.get(attr.name).get)
+(escapedKey, escapedValue)
+ }
+
val committer = FileCommitProtocol.instantiate(
sparkSession.sessionState.conf.fileCommitProtocolClass,
- jobId = java.util.UUID.randomUUID().toString,
+ jobId = jobId,
outputPath = outputPath.toString,
- dynamicPartitionOverwrite = dynamicPartitionOverwrite)
+ dynamicPartitionOverwrite = dynamicPartitionOverwrite,
+ fileSourceWriteDesc = Some(FileSourceWriteDesc(true,
escapedStaticPartitionKVs)))
-val doInsertion = if (mode == SaveMode.Append) {
- true
-} else {
- val pathExists = fs.exists(qualifiedOutputPath)
- (mode, pathExists) match {
-case (SaveMode.ErrorIfExists, true) =>
- throw new AnalysisException(s"path $qualifiedOutputPath already
exists.")
-case (SaveMode.Overwrite, true) =>
- if (ifPartitionNotExists && matchingPartitions.nonEmpty) {
-false
- } else if (dynamicPartitionOverwrite) {
-// For dynamic partition overwrite, do not delete partition
directories ahead.
-true
- } else {
-deleteMatchingPartitions(fs, qualifiedOutputPath,
customPartitionLocations, committer)
+try {
+ val doInsertion = if (mode == SaveMode.Append) {
Review comment:
Can this modification be simplified?
We keep the `val doInsertion = ...` intact, issues `detectConflict` when
doInsertion is true.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write
advancedxy commented on a change in pull request #25863:
[SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate
result and support concurrent file source write operations write to different
partitions in the same table.
URL: https://github.com/apache/spark/pull/25863#discussion_r329335511
##
File path:
core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
##
@@ -91,7 +96,62 @@ class HadoopMapReduceCommitProtocol(
*/
private def stagingDir = new Path(path, ".spark-staging-" + jobId)
+ /**
+ * Whether is a InsertIntoHadoopFsRelation operation, the default is false.
+ */
+ private def isInsertIntoHadoopFsRelation =
+fileSourceWriteDesc.map(_.isInsertIntoHadoopFsRelation).getOrElse(false)
+
+ /**
+ * Get escaped static partition key and value pairs, the default is empty.
+ */
+ private def escapedStaticPartitionKVs =
+fileSourceWriteDesc.map(_.escapedStaticPartitionKVs).getOrElse(Seq.empty)
+
+ /**
+ * The staging root directory for InsertIntoHadoopFsRelation operation.
+ */
+ @transient private var insertStagingDir: Path = null
+
+ /**
+ * The staging output path for InsertIntoHadoopFsRelation operation.
+ */
+ @transient private var stagingOutputPath: Path = null
+
+ /**
+ * Get the desired output path for the job. The output will be [[path]] when
current operation
+ * is not a InsertIntoHadoopFsRelation operation. Otherwise, we choose a sub
path composed of
+ * [[escapedStaticPartitionKVs]] under [[insertStagingDir]] over [[path]] to
mark this operation
+ * and we can detect whether there is a operation conflict with current by
checking the existence
+ * of relative output path.
+ *
+ * @return Path the desired output path.
+ */
+ protected def getOutputPath(context: TaskAttemptContext): Path = {
+if (isInsertIntoHadoopFsRelation) {
+ val insertStagingPath = ".spark-staging-" +
escapedStaticPartitionKVs.size
+ insertStagingDir = new Path(path, insertStagingPath)
+ val appId = SparkEnv.get.conf.getAppId
+ val outputPath = new Path(path, Array(insertStagingPath,
+getEscapedStaticPartitionPath(escapedStaticPartitionKVs), appId, jobId)
+.mkString(File.separator))
+
insertStagingDir.getFileSystem(context.getConfiguration).makeQualified(outputPath)
+ outputPath
+} else {
+ new Path(path)
+}
+ }
+
protected def setupCommitter(context: TaskAttemptContext): OutputCommitter =
{
+if (isInsertIntoHadoopFsRelation) {
+ stagingOutputPath = getOutputPath(context)
+ context.getConfiguration.set(FileOutputFormat.OUTDIR,
stagingOutputPath.toString)
+ logWarning("Set file output committer algorithm version to 2
implicitly," +
+" for that the task output would be committed to staging output path
firstly," +
+" which is equivalent to algorithm 1.")
Review comment:
15843 is a lot, however, it would be not that much inside one spark
application.
One way to solve this, is to use an object level counter to only log the
first warning log(or logs).
But I am not sure if that's worth it. Also, the head of logs may get rotated
and discarded...
Or use logDebug is fine, but normally user won't set log level to DEBUG.
I am not sure which one is better. It's up to you then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] advancedxy commented on a change in pull request #25863: [SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate result and support concurrent file source write
advancedxy commented on a change in pull request #25863:
[SPARK-28945][SPARK-29037][CORE][SQL] Fix the issue that spark gives duplicate
result and support concurrent file source write operations write to different
partitions in the same table.
URL: https://github.com/apache/spark/pull/25863#discussion_r329335222
##
File path:
core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala
##
@@ -169,4 +171,84 @@ object FileCommitProtocol extends Logging {
ctor.newInstance(jobId, outputPath)
}
}
+
+ /**
+ * Instantiates a FileCommitProtocol with file source write description.
+ */
+ def instantiate(
+ className: String,
+ jobId: String,
+ outputPath: String,
+ dynamicPartitionOverwrite: Boolean,
+ fileSourceWriteDesc: Option[FileSourceWriteDesc]): FileCommitProtocol = {
+
+logDebug(s"Creating committer $className; job $jobId; output=$outputPath;"
+
+ s" dynamic=$dynamicPartitionOverwrite; fileSourceWriteDesc=
$fileSourceWriteDesc")
+val clazz = Utils.classForName[FileCommitProtocol](className)
+// First try the constructor with arguments (jobId: String, outputPath:
String,
+// dynamicPartitionOverwrite: Boolean, fileSourceWriteDesc:
Option[FileSourceWriteDesc]).
+// If that doesn't exist, try to invoke
`FileCommitProtocol.instance(className,
+// JobId, outputPath, dynamicPartitionOverwrite)`.
+try {
+ val ctor = clazz.getDeclaredConstructor(classOf[String],
classOf[String], classOf[Boolean],
+classOf[Option[FileSourceWriteDesc]])
+ logDebug("Using (String, String, Boolean, FileSourceWriteDesc)
constructor")
+ ctor.newInstance(jobId, outputPath,
dynamicPartitionOverwrite.asInstanceOf[java.lang.Boolean],
+fileSourceWriteDesc)
+} catch {
+ case _: NoSuchMethodException =>
+logDebug("Falling back to invoke instance(className, JobId,
outputPath," +
+ " dynamicPartitionOverwrite)")
+instantiate(className, jobId, outputPath, dynamicPartitionOverwrite)
+}
+ }
Review comment:
Kind of, but I think we can also add `dynamicPartitionOverwrite` to
`fileSourceWriteDesc`.
For user defined `FileCommitProtocol` class, We can extract
`fileSourceWriteDesc` and pass it to the old instantiate method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
LantaoJin commented on issue #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table URL: https://github.com/apache/spark/pull/25840#issuecomment-536244804 CC @HyukjinKwon @cloud-fan @srowen who probably have a chance to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
AmplabJenkins commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536243967 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111543/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
AmplabJenkins removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536243964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
AmplabJenkins removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536243967 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111543/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
AmplabJenkins commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536243964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
SparkQA removed a comment on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536232922 **[Test build #111543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111543/testReport)** for PR 24890 at commit [`d2d511e`](https://github.com/apache/spark/commit/d2d511ecc8daa307549d8013b24ff31c2a3045fc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query
SparkQA commented on issue #24890: [SPARK-28074][SS] Log warn message on possible correctness issue for multiple stateful operations in single query URL: https://github.com/apache/spark/pull/24890#issuecomment-536243845 **[Test build #111543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111543/testReport)** for PR 24890 at commit [`d2d511e`](https://github.com/apache/spark/commit/d2d511ecc8daa307549d8013b24ff31c2a3045fc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25561: [SPARK-28810][DOC][SQL] Document SHOW TABLES in SQL Reference.
srowen commented on a change in pull request #25561: [SPARK-28810][DOC][SQL]
Document SHOW TABLES in SQL Reference.
URL: https://github.com/apache/spark/pull/25561#discussion_r329335485
##
File path: docs/sql-ref-syntax-aux-show-tables.md
##
@@ -18,5 +18,86 @@ license: |
See the License for the specific language governing permissions and
limitations under the License.
---
+### Description
-**This page is under construction**
+`SHOW TABLES` statement returns all the tables for an optionally specified
`database`.
+Additionally, the output of this statement may be filtered via an optional
matching
+pattern. If no database is specified then the tables are returned from the
+current database.
+
+### Syntax
+{% highlight sql %}
+SHOW TABLES [{FROM|IN} database_name] [LIKE 'regex_pattern']
+{% endhighlight %}
+
+### Parameters
+
+ {FROM|IN} database_name
+
+ Specifies the `database` name from which tables are listed.
+
+ LIKE 'regex_pattern'
+
+ Specifies the regex pattern that is used to filter out unwanted tables.
+- Only `*` and `|` are allowed as wildcard pattern.
Review comment:
Is it a SQL-like pattern that supports wildcards like `%`? then I'd just
call it a `pattern`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536243457 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111551/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536242913 **[Test build #111551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111551/testReport)** for PR 25729 at commit [`7692893`](https://github.com/apache/spark/commit/7692893d8ca1474c04407cd412dce416f0709af6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins removed a comment on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536243455 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536243450 **[Test build #111551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111551/testReport)** for PR 25729 at commit [`7692893`](https://github.com/apache/spark/commit/7692893d8ca1474c04407cd412dce416f0709af6). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536243455 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
AmplabJenkins commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536243457 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111551/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536243023 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16560/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536243025 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536243028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16560/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536243025 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536243028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16560/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
SparkQA commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536242913 **[Test build #111551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111551/testReport)** for PR 25729 at commit [`7692893`](https://github.com/apache/spark/commit/7692893d8ca1474c04407cd412dce416f0709af6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242150 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16562/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242148 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242150 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16562/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
AmplabJenkins removed a comment on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242148 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
SparkQA commented on issue #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#issuecomment-536242073 **[Test build #111550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111550/testReport)** for PR 25933 at commit [`ce4d184`](https://github.com/apache/spark/commit/ce4d184e1cf42bea35f08de8ade8666bf217c48c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536242080 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16560/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities.
beliefer commented on a change in pull request #25933: [SPARK-29252][BUILD] Upgrade zookeeper to 3.4.14 and fix vulnerabilities. URL: https://github.com/apache/spark/pull/25933#discussion_r329334818 ## File path: dev/deps/spark-deps-hadoop-2.7 ## @@ -20,6 +20,7 @@ arpack_combined_all-0.1.jar arrow-format-0.12.0.jar arrow-memory-0.12.0.jar arrow-vector-0.12.0.jar +audience-annotations-0.5.0.jar Review comment: OK. I will append the `NOTICE-binary`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
AmplabJenkins commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536241826 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
AmplabJenkins removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536241826 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
AmplabJenkins commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536241828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111544/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
AmplabJenkins removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536241828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111544/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16561/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241749 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
SparkQA removed a comment on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536233808 **[Test build #111544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111544/testReport)** for PR 25953 at commit [`9d5c2d1`](https://github.com/apache/spark/commit/9d5c2d1580f99a4efad377491c8105b63413ccd6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins removed a comment on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241749 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AmplabJenkins commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16561/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
SparkQA commented on issue #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#issuecomment-536241728 **[Test build #111544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111544/testReport)** for PR 25953 at commit [`9d5c2d1`](https://github.com/apache/spark/commit/9d5c2d1580f99a4efad377491c8105b63413ccd6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
SparkQA commented on issue #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state URL: https://github.com/apache/spark/pull/25954#issuecomment-536241667 **[Test build #111549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111549/testReport)** for PR 25954 at commit [`b056622`](https://github.com/apache/spark/commit/b056622068a300c44340fd02b3773fda81a52f73). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25954: [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
AngersZh commented on a change in pull request #25954: [SPARK-29247][SQL]
Redact sensitive information in when construct HiveClientHive.state
URL: https://github.com/apache/spark/pull/25954#discussion_r329334623
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##
@@ -181,7 +182,7 @@ private[hive] class HiveClientImpl(
logDebug(
s"""
|Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
- |$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx"
else v}
+ |$k=${SQLConf.get.redactOptions(k.toLowerCase(Locale.ROOT) -> v)._2}
Review comment:
Change back use origin method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound
wangyum commented on issue #25729: [SPARK-29022][SQL][test-hadoop3.2][test-java11] Fix spark 'add jar', CliSessionState's hiveConf 's classLoader ClassNotFound URL: https://github.com/apache/spark/pull/25729#issuecomment-536241439 This PR is needed. but this change can not handle `isolatedLoader` is enabled(`spark.sql.hive.metastore.jars != builtin`). It seems [this commit](https://github.com/apache/spark/pull/25729/commits/0fb149c25e995bd5d2f08ece1ac65ef5f587d722) can handle this case. But we can not add a test case for this case because of [SPARK-29268](https://issues.apache.org/jira/browse/SPARK-29268). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111548/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AngersZh commented on a change in pull request #25944: [SPARK-29254][SQL]
Failed to include jars passed in through --jars when isolatedLoader is enabled
URL: https://github.com/apache/spark/pull/25944#discussion_r329334338
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##
@@ -80,10 +80,21 @@ private[hive] object IsolatedClientLoader extends Logging {
resolvedVersions((resolvedVersion, actualHadoopVersion))
}
+// HiveMetastoreConnection using maven missed the jars in classpath which
passed
+// through --jars. They need to be added back before new
IsolatedClientLoader()
+val jarsInClasspath = Utils.getContextOrSparkClassLoader match {
+ case m: MutableURLClassLoader =>
+val jars = m.getURLs.toSeq
+logInfo(s"Added jars in classpath to IsolatedClientLoader: $jars")
+jars
+ case _ =>
+Seq.empty
+}
+
new IsolatedClientLoader(
hiveVersion(hiveMetastoreVersion),
sparkConf,
- execJars = files,
+ execJars = files ++ jarsInClasspath,
Review comment:
That's good, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111548/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
SparkQA removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240135 **[Test build #111548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111548/testReport)** for PR 25648 at commit [`92306f8`](https://github.com/apache/spark/commit/92306f807452c68b5dabf9e31b3f5bfcd4403eee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins removed a comment on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240914 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
AmplabJenkins commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240914 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness
SparkQA commented on issue #25648: [SPARK-28947][K8S] Status logging not happens at an interval for liveness URL: https://github.com/apache/spark/pull/25648#issuecomment-536240890 **[Test build #111548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111548/testReport)** for PR 25648 at commit [`92306f8`](https://github.com/apache/spark/commit/92306f807452c68b5dabf9e31b3f5bfcd4403eee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins commented on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536240213 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16559/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled
AmplabJenkins removed a comment on issue #25944: [SPARK-29254][SQL] Failed to include jars passed in through --jars when isolatedLoader is enabled URL: https://github.com/apache/spark/pull/25944#issuecomment-536240213 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16559/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
