[GitHub] [spark] dongjoon-hyun commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite
dongjoon-hyun commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864#issuecomment-533417576 Please use `[TESTS]` when the PR touches only `src/test` files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533417233 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533417235 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111042/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533417233 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533417235 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111042/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
SparkQA removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533388635 **[Test build #111042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111042/testReport)** for PR 25850 at commit [`b9dc92b`](https://github.com/apache/spark/commit/b9dc92b8500c0c9d2bcce9d45c739c53f54fb209). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
SparkQA commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533416763 **[Test build #111042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111042/testReport)** for PR 25850 at commit [`b9dc92b`](https://github.com/apache/spark/commit/b9dc92b8500c0c9d2bcce9d45c739c53f54fb209). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533416241 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111034/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533416232 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533416241 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111034/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533416232 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add
parameters to limit the number of dynamic partitions for data source table
URL: https://github.com/apache/spark/pull/25840#discussion_r326471315
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala
##
@@ -63,7 +70,29 @@ class SQLHadoopMapReduceCommitProtocol(
committer = ctor.newInstance()
}
}
+totalPartitions = new AtomicInteger(0)
logInfo(s"Using output committer class
${committer.getClass.getCanonicalName}")
committer
}
+
+ override def newTaskTempFile(
+ taskContext: TaskAttemptContext, dir: Option[String], ext: String):
String = {
+val path = super.newTaskTempFile(taskContext, dir, ext)
+totalPartitions.incrementAndGet()
+if (dynamicPartitionOverwrite) {
+ if (totalPartitions.get > maxDynamicPartitions) {
Review comment:
If my understanding on SQLConf.DYNAMIC_PARTITION_MAX_PARTITIONS is correct,
that it is the max number of partitions a data source can have at any given
time, then I am not sure that this is the right place to check the total number
of partitions in the data source. The best would have been
[here](https://github.com/apache/spark/blob/76ebf2241a3f2149de13d6c89adcb86325b06004/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L137)
But it might be too late by the time control reaches there.
One way is we can pass the initial partitions in the data source at the
beginning, and here can check if the combined number is more than the
configured value. But I am not sure if this is the safest way (can another
insert happen in parallel on the data source and commit, this affecting the
value known here.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533366185 **[Test build #111034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111034/testReport)** for PR 24232 at commit [`722f0e6`](https://github.com/apache/spark/commit/722f0e68a9a4a73118ea636761f8e888a2dc48fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-533415718 **[Test build #111034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111034/testReport)** for PR 24232 at commit [`722f0e6`](https://github.com/apache/spark/commit/722f0e68a9a4a73118ea636761f8e888a2dc48fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on issue #25855: [SPARK-29053][WEBUI] Spark UI JDBC/ODBC Server tab sorting is not working on Duration column
amanomer commented on issue #25855: [SPARK-29053][WEBUI] Spark UI JDBC/ODBC Server tab sorting is not working on Duration column URL: https://github.com/apache/spark/pull/25855#issuecomment-533415033 @dongjoon-hyun thanks for your feedback. I am working on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
HyukjinKwon commented on a change in pull request #25849: [SPARK-29175][SQL]
Make additional remote maven repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326478196
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1979,6 +1979,13 @@ object SQLConf {
.doc("When true, the ArrayExists will follow the three-valued boolean
logic.")
.booleanConf
.createWithDefault(true)
+
+ private[spark] val ADDITIONAL_REMOTE_REPOSITORIES =
+ConfigBuilder("spark.sql.additionalRemoteRepositories")
+ .doc("A comma-delimited string config of the optional additional remote
maven mirror " +
+"repositories, this can be used for downloading Hive jars in
IsolatedClientLoader.")
+ .stringConf
+ .createWithDefault("")
Review comment:
We're not going to set the default option, can we use `createOptional`
instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
viirya commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533414410 Ok for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec
viirya commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec URL: https://github.com/apache/spark/pull/25830#issuecomment-533414231 As I said, I think it is possible an aggregate function accesses complex data input like array but uses a buffer attribute which is supported by HashAggregateExec. If you just filter out complex data types out, the split function for such aggregation function won't work. So currently this looks good to me. I am not sure if we want to turn off split mode just because of array argument as @maropu suggested. cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
HyukjinKwon commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533413708 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
HyukjinKwon closed pull request #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
HyukjinKwon commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533413162 I think we can just leave it without other options. It's rather corner case and I think it's fine to break such stuff since we're moving to Spark 3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec
HeartSaVioR commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in HashAggregateExec URL: https://github.com/apache/spark/pull/25830#issuecomment-533411970 Just to determine the next action, would we want to include newer discussion (@viirya and @maropu are discussing) for the scope of this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
HeartSaVioR edited a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533410631 Uh, shall we talk about which option is preferred one? It needs to be done before describing in migration note. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
HeartSaVioR commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533410631 Uh, could we talk about which option is preferred one? It needs to be done before describing in migration note. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL]
Make additional remote maven repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326474439
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##
@@ -112,18 +115,18 @@ private[hive] object IsolatedClientLoader extends
Logging {
private def downloadVersion(
version: HiveVersion,
hadoopVersion: String,
- ivyPath: Option[String]): Seq[URL] = {
+ ivyPath: Option[String],
+ centralRepo: String): Seq[URL] = {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1",
s"org.apache.hadoop:hadoop-client:$hadoopVersion")
-
val classpath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
- Some("http://www.datanucleus.org/downloads/maven2;),
Review comment:
Yes, as the discussion above, the flakiness is caused by when the Jenkins
blocked by maven central repo and the additional datanucleus remote repo still
not work.
I updated this PR to set google mirror as an additional remote repo for hive
tests in 49ea1cd.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
AmplabJenkins removed a comment on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable URL: https://github.com/apache/spark/pull/25849#issuecomment-533409666 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
AmplabJenkins removed a comment on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable URL: https://github.com/apache/spark/pull/25849#issuecomment-533409669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16134/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
AmplabJenkins commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable URL: https://github.com/apache/spark/pull/25849#issuecomment-533409666 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
AmplabJenkins commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable URL: https://github.com/apache/spark/pull/25849#issuecomment-533409669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16134/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
SparkQA commented on issue #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable URL: https://github.com/apache/spark/pull/25849#issuecomment-533409448 **[Test build #111046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111046/testReport)** for PR 25849 at commit [`49ea1cd`](https://github.com/apache/spark/commit/49ea1cd1d50a7524257890542c8c5c2d094ed1eb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL]
Make additional remote maven repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326473765
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1979,6 +1979,13 @@ object SQLConf {
.doc("When true, the ArrayExists will follow the three-valued boolean
logic.")
.booleanConf
.createWithDefault(true)
+
+ private[spark] val CENTRAL_REPOSITORY =
+ConfigBuilder("spark.sql.centralRepository")
+ .doc("The default central repository used for downloading Hive jars " +
+"in IsolatedClientLoader.")
+ .stringConf
+ .createWithDefault("https://repo1.maven.org/maven2;)
Review comment:
That's right, I also test locally without setting any additional remote
repo, it will pass.
Change the default value and set the config to google mirror for hive tests
in 49ea1cd.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL]
Make additional remote maven repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326473765
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1979,6 +1979,13 @@ object SQLConf {
.doc("When true, the ArrayExists will follow the three-valued boolean
logic.")
.booleanConf
.createWithDefault(true)
+
+ private[spark] val CENTRAL_REPOSITORY =
+ConfigBuilder("spark.sql.centralRepository")
+ .doc("The default central repository used for downloading Hive jars " +
+"in IsolatedClientLoader.")
+ .stringConf
+ .createWithDefault("https://repo1.maven.org/maven2;)
Review comment:
That's right, I also test locally without setting any additional remote
repo, it will pass.
Change the default value and set the config to google mirror in 49ea1cd.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable
xuanyuanking commented on a change in pull request #25849: [SPARK-29175][SQL]
Make additional remote maven repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326473540
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1979,6 +1979,13 @@ object SQLConf {
.doc("When true, the ArrayExists will follow the three-valued boolean
logic.")
.booleanConf
.createWithDefault(true)
+
+ private[spark] val CENTRAL_REPOSITORY =
+ConfigBuilder("spark.sql.centralRepository")
+ .doc("The default central repository used for downloading Hive jars " +
Review comment:
Thanks for the correcting, fix the description in 49ea1cd.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
SparkQA commented on issue #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty URL: https://github.com/apache/spark/pull/25837#issuecomment-533408147 **[Test build #111045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111045/testReport)** for PR 25837 at commit [`8e661f7`](https://github.com/apache/spark/commit/8e661f7d66993e9921a2df72cf9c21573c7edfac). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533407397 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111038/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533407393 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533407393 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
AmplabJenkins commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533407397 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111038/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
SparkQA removed a comment on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533376774 **[Test build #111038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111038/testReport)** for PR 25797 at commit [`1c36bfe`](https://github.com/apache/spark/commit/1c36bfe2208cf21cde8ef7c13fa97438c9303787). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server
SparkQA commented on issue #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#issuecomment-533407049 **[Test build #111038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111038/testReport)** for PR 25797 at commit [`1c36bfe`](https://github.com/apache/spark/commit/1c36bfe2208cf21cde8ef7c13fa97438c9303787). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite
AmplabJenkins commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864#issuecomment-533407020 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
HeartSaVioR commented on a change in pull request #25837:
[SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
URL: https://github.com/apache/spark/pull/25837#discussion_r326472009
##
File path:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
##
@@ -180,7 +180,7 @@ class CoarseGrainedSchedulerBackendSuite extends
SparkFunSuite with LocalSparkCo
backend.driverEndpoint.askSync[Boolean](
RegisterExecutor("3", mockEndpointRef, mockAddress.host, 1, logUrls,
attributes, Map.empty))
-sc.listenerBus.waitUntilEmpty(executorUpTimeout.toMillis)
+sc.listenerBus.waitUntilEmpty()
Review comment:
Rolled back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite
SparkQA commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864#issuecomment-533406906 **[Test build #111044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111044/testReport)** for PR 25864 at commit [`17417fb`](https://github.com/apache/spark/commit/17417fb4cb5e535f832c52b6d3f197f923676fe2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite
HeartSaVioR commented on issue #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864#issuecomment-533406228 Please refer below link to see rationalization of the change: https://issues.apache.org/jira/browse/SPARK-29139?focusedCommentId=16934034=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16934034 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR opened a new pull request #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite
HeartSaVioR opened a new pull request #25864: [SPARK-29139][CORE] Increase timeout to wait for executor(s) to be up in SparkContextSuite URL: https://github.com/apache/spark/pull/25864 ### What changes were proposed in this pull request? This patch proposes to increase timeout to wait for executor(s) to be up in SparkContextSuite, as we observed these tests failed due to wait timeout. ### Why are the changes needed? There's some case that CI build is extremely slow which requires 3x or more time to pass the test. (https://issues.apache.org/jira/browse/SPARK-29139?focusedCommentId=16934034=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16934034) Allocating higher timeout wouldn't bring additional latency, as the code checks the condition with sleeping 10 ms per loop iteration. ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A, as the case is not likely to be occurred frequently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add
parameters to limit the number of dynamic partitions for data source table
URL: https://github.com/apache/spark/pull/25840#discussion_r326471315
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala
##
@@ -63,7 +70,29 @@ class SQLHadoopMapReduceCommitProtocol(
committer = ctor.newInstance()
}
}
+totalPartitions = new AtomicInteger(0)
logInfo(s"Using output committer class
${committer.getClass.getCanonicalName}")
committer
}
+
+ override def newTaskTempFile(
+ taskContext: TaskAttemptContext, dir: Option[String], ext: String):
String = {
+val path = super.newTaskTempFile(taskContext, dir, ext)
+totalPartitions.incrementAndGet()
+if (dynamicPartitionOverwrite) {
+ if (totalPartitions.get > maxDynamicPartitions) {
Review comment:
If my understanding on SQLConf.DYNAMIC_PARTITION_MAX_PARTITIONS is correct,
that it is the max number of partitions a data source can have at any given
time, then I am not convinced that this is the right place to check the total
number of partitions in the data source. The best would have been
[here](https://github.com/apache/spark/blob/76ebf2241a3f2149de13d6c89adcb86325b06004/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L137)
But it might be too late by the time control reaches there.
One way is we can pass the initial partitions in the data source at the
beginning, and here can check if the combined number is more than the
configured value. But I am not sure if this is the safest way (can another
insert happen in parallel on the data source and commit, this affecting the
value known here.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
itskals commented on a change in pull request #25840: [SPARK-29166][SQL] Add
parameters to limit the number of dynamic partitions for data source table
URL: https://github.com/apache/spark/pull/25840#discussion_r326471315
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala
##
@@ -63,7 +70,29 @@ class SQLHadoopMapReduceCommitProtocol(
committer = ctor.newInstance()
}
}
+totalPartitions = new AtomicInteger(0)
logInfo(s"Using output committer class
${committer.getClass.getCanonicalName}")
committer
}
+
+ override def newTaskTempFile(
+ taskContext: TaskAttemptContext, dir: Option[String], ext: String):
String = {
+val path = super.newTaskTempFile(taskContext, dir, ext)
+totalPartitions.incrementAndGet()
+if (dynamicPartitionOverwrite) {
+ if (totalPartitions.get > maxDynamicPartitions) {
Review comment:
If my understanding on SQLConf.DYNAMIC_PARTITION_MAX_PARTITIONS is correct,
that it is the max number of partitions a data source can have at given time,
then I am not convinced that this is the right place to check the total number
of partitions in the data source. The best would have been
[here](https://github.com/apache/spark/blob/76ebf2241a3f2149de13d6c89adcb86325b06004/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L137)
But it might be too late by the time control reaches there.
One way is we can pass the initial partitions in the data source at the
beginning, and here can check if the combined number is more than the
configured value. But I am not sure if this is the safest way (can another
insert happen in parallel on the data source and commit, this affecting the
value known here.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25666: [SPARK-28962][SQL] Provide index argument to filter lambda functions
maropu commented on a change in pull request #25666: [SPARK-28962][SQL] Provide
index argument to filter lambda functions
URL: https://github.com/apache/spark/pull/25666#discussion_r326468768
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
##
@@ -357,10 +359,23 @@ case class ArrayFilter(
override def bind(f: (Expression, Seq[(DataType, Boolean)]) =>
LambdaFunction): ArrayFilter = {
val ArrayType(elementType, containsNull) = argument.dataType
-copy(function = f(function, (elementType, containsNull) :: Nil))
+function match {
+ case LambdaFunction(_, arguments, _) if arguments.size == 2 =>
+copy(function = f(function, (elementType, containsNull) ::
(IntegerType, false) :: Nil))
+ case _ =>
+copy(function = f(function, (elementType, containsNull) :: Nil))
Review comment:
nvm. I checked the error handling works well for the case, too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL] Make maven central repository in IsolatedClientLoader configurable
dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL]
Make maven central repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326468590
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##
@@ -112,18 +115,18 @@ private[hive] object IsolatedClientLoader extends
Logging {
private def downloadVersion(
version: HiveVersion,
hadoopVersion: String,
- ivyPath: Option[String]): Seq[URL] = {
+ ivyPath: Option[String],
+ centralRepo: String): Seq[URL] = {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1",
s"org.apache.hadoop:hadoop-client:$hadoopVersion")
-
val classpath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
- Some("http://www.datanucleus.org/downloads/maven2;),
Review comment:
@xuanyuanking . If then, could you make a separate JIRA and PR for this line
change with the following description?
> The repository currently used is
"http://www.datanucleus.org/downloads/maven2;, which is no longer maintained.
This will sometimes cause downloading failure and make hive test cases flaky.
End users can also set this config to the central repository they want to
access.
Then, we can backport your new PR to branch-2.4, too. After that, we can
proceed this PR on top of that. That will be very helpful for our LTS branch
`branch-2.4`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25666: [SPARK-28962][SQL] Provide index argument to filter lambda functions
maropu commented on a change in pull request #25666: [SPARK-28962][SQL] Provide
index argument to filter lambda functions
URL: https://github.com/apache/spark/pull/25666#discussion_r326468768
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
##
@@ -357,10 +359,23 @@ case class ArrayFilter(
override def bind(f: (Expression, Seq[(DataType, Boolean)]) =>
LambdaFunction): ArrayFilter = {
val ArrayType(elementType, containsNull) = argument.dataType
-copy(function = f(function, (elementType, containsNull) :: Nil))
+function match {
+ case LambdaFunction(_, arguments, _) if arguments.size == 2 =>
+copy(function = f(function, (elementType, containsNull) ::
(IntegerType, false) :: Nil))
+ case _ =>
+copy(function = f(function, (elementType, containsNull) :: Nil))
Review comment:
nvm. I checked the error handling works well for the case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL] Make maven central repository in IsolatedClientLoader configurable
dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL]
Make maven central repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326468590
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##
@@ -112,18 +115,18 @@ private[hive] object IsolatedClientLoader extends
Logging {
private def downloadVersion(
version: HiveVersion,
hadoopVersion: String,
- ivyPath: Option[String]): Seq[URL] = {
+ ivyPath: Option[String],
+ centralRepo: String): Seq[URL] = {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1",
s"org.apache.hadoop:hadoop-client:$hadoopVersion")
-
val classpath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
- Some("http://www.datanucleus.org/downloads/maven2;),
Review comment:
@xuanyuanking . If then, could you make a separate PR for this line change?
Then, we can backport your new PR to branch-2.4, too. After that, we can
proceed this PR on top of that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL] Make maven central repository in IsolatedClientLoader configurable
dongjoon-hyun commented on a change in pull request #25849: [SPARK-29175][SQL]
Make maven central repository in IsolatedClientLoader configurable
URL: https://github.com/apache/spark/pull/25849#discussion_r326468438
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##
@@ -112,18 +115,18 @@ private[hive] object IsolatedClientLoader extends
Logging {
private def downloadVersion(
version: HiveVersion,
hadoopVersion: String,
- ivyPath: Option[String]): Seq[URL] = {
+ ivyPath: Option[String],
+ centralRepo: String): Seq[URL] = {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Seq("com.google.guava:guava:14.0.1",
s"org.apache.hadoop:hadoop-client:$hadoopVersion")
-
val classpath = quietly {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
- Some("http://www.datanucleus.org/downloads/maven2;),
Review comment:
Interesting. So, with this PR, the side-effect benefit is the removal of the
flakiness by default, @xuanyuanking?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533400512 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111040/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533400512 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111040/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
SparkQA removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533381859 **[Test build #111040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111040/testReport)** for PR 25850 at commit [`d1e744e`](https://github.com/apache/spark/commit/d1e744eca601d75770011b08d39cc09fa8bcab89). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
SparkQA commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533400332 **[Test build #111040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111040/testReport)** for PR 25850 at commit [`d1e744e`](https://github.com/apache/spark/commit/d1e744eca601d75770011b08d39cc09fa8bcab89). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533400506 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533400506 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
AmplabJenkins removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533399599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111036/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
AmplabJenkins removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533399588 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
AmplabJenkins commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533399588 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
AmplabJenkins commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533399599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111036/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
SparkQA removed a comment on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533368855 **[Test build #111036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111036/testReport)** for PR 25838 at commit [`9f1f561`](https://github.com/apache/spark/commit/9f1f5617437fba231337495d68c4454fa8058b07). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers
SparkQA commented on issue #25838: [SPARK-29158][SQL] Expose SerializableConfiguration for DataSource V2 developers URL: https://github.com/apache/spark/pull/25838#issuecomment-533399253 **[Test build #111036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111036/testReport)** for PR 25838 at commit [`9f1f561`](https://github.com/apache/spark/commit/9f1f5617437fba231337495d68c4454fa8058b07). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SerializableConfigurationSuite ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
HeartSaVioR commented on a change in pull request #25862:
[SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not
rely on timing
URL: https://github.com/apache/spark/pull/25862#discussion_r326466225
##
File path:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala
##
@@ -106,19 +106,22 @@ class ReceiverSuite extends TestSuiteBase with
TimeLimits with Serializable {
assert(executor.errors.head.eq(exception))
// Verify restarting actually stops and starts the receiver
-receiver.restart("restarting", null, 600)
-eventually(timeout(300.milliseconds), interval(10.milliseconds)) {
- // receiver will be stopped async
- assert(receiver.isStopped)
- assert(receiver.onStopCalled)
-}
-eventually(timeout(1.second), interval(10.milliseconds)) {
- // receiver will be started async
- assert(receiver.onStartCalled)
- assert(executor.isReceiverStarted)
+executor.callsRecorder.reset()
+receiver.callsRecorder.reset()
+receiver.restart("restarting", null, 100)
+eventually(timeout(10.seconds), interval(10.milliseconds)) {
Review comment:
Yes that was actually 1.3 seconds (300ms + 1s) and it hasn't been failing
for high probability so it should be pretty enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
AmplabJenkins removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533398945 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111029/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
AmplabJenkins commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533398943 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
AmplabJenkins removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533398943 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
AmplabJenkins commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533398945 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111029/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on issue #25771: [SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2
imback82 commented on issue #25771: [SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 URL: https://github.com/apache/spark/pull/25771#issuecomment-533398732 @cloud-fan / @rdblue, what should be the behavior `USE ` if the `` doesn't exist? Should we follow the same behavior as v1 where database is required to exist? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
HeartSaVioR commented on a change in pull request #25837:
[SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
URL: https://github.com/apache/spark/pull/25837#discussion_r326465954
##
File path:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
##
@@ -180,7 +180,7 @@ class CoarseGrainedSchedulerBackendSuite extends
SparkFunSuite with LocalSparkCo
backend.driverEndpoint.askSync[Boolean](
RegisterExecutor("3", mockEndpointRef, mockAddress.host, 1, logUrls,
attributes, Map.empty))
-sc.listenerBus.waitUntilEmpty(executorUpTimeout.toMillis)
+sc.listenerBus.waitUntilEmpty()
Review comment:
Ah right. My bad that's 6 ms. I'll just roll them back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
SparkQA commented on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533398582 **[Test build #111029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111029/testReport)** for PR 25854 at commit [`fa55b3a`](https://github.com/apache/spark/commit/fa55b3ab2783972c55ae0c8c21aed88fe1b1c220). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN"
SparkQA removed a comment on issue #25854: [SPARK-29145][SQL]Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-533351174 **[Test build #111029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111029/testReport)** for PR 25854 at commit [`fa55b3a`](https://github.com/apache/spark/commit/fa55b3ab2783972c55ae0c8c21aed88fe1b1c220). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
dongjoon-hyun commented on a change in pull request #25862:
[SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not
rely on timing
URL: https://github.com/apache/spark/pull/25862#discussion_r326465739
##
File path:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala
##
@@ -106,19 +106,22 @@ class ReceiverSuite extends TestSuiteBase with
TimeLimits with Serializable {
assert(executor.errors.head.eq(exception))
// Verify restarting actually stops and starts the receiver
-receiver.restart("restarting", null, 600)
-eventually(timeout(300.milliseconds), interval(10.milliseconds)) {
- // receiver will be stopped async
- assert(receiver.isStopped)
- assert(receiver.onStopCalled)
-}
-eventually(timeout(1.second), interval(10.milliseconds)) {
- // receiver will be started async
- assert(receiver.onStartCalled)
- assert(executor.isReceiverStarted)
+executor.callsRecorder.reset()
+receiver.callsRecorder.reset()
+receiver.restart("restarting", null, 100)
+eventually(timeout(10.seconds), interval(10.milliseconds)) {
Review comment:
So, `10.seconds` is enough? Or, do you need to re-trigger this PR to
validate more?
BTW, thank you so much for taking care of this case! This is really an long
standing issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
dongjoon-hyun commented on a change in pull request #25862:
[SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not
rely on timing
URL: https://github.com/apache/spark/pull/25862#discussion_r326465739
##
File path:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala
##
@@ -106,19 +106,22 @@ class ReceiverSuite extends TestSuiteBase with
TimeLimits with Serializable {
assert(executor.errors.head.eq(exception))
// Verify restarting actually stops and starts the receiver
-receiver.restart("restarting", null, 600)
-eventually(timeout(300.milliseconds), interval(10.milliseconds)) {
- // receiver will be stopped async
- assert(receiver.isStopped)
- assert(receiver.onStopCalled)
-}
-eventually(timeout(1.second), interval(10.milliseconds)) {
- // receiver will be started async
- assert(receiver.onStartCalled)
- assert(executor.isReceiverStarted)
+executor.callsRecorder.reset()
+receiver.callsRecorder.reset()
+receiver.restart("restarting", null, 100)
+eventually(timeout(10.seconds), interval(10.milliseconds)) {
Review comment:
So, `10.seconds` is enough? Or, do you need to re-trigger this PR to
validate more?
BTW, thank you so much for taking care of this case!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
dongjoon-hyun commented on a change in pull request #25862:
[SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not
rely on timing
URL: https://github.com/apache/spark/pull/25862#discussion_r326465739
##
File path:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala
##
@@ -106,19 +106,22 @@ class ReceiverSuite extends TestSuiteBase with
TimeLimits with Serializable {
assert(executor.errors.head.eq(exception))
// Verify restarting actually stops and starts the receiver
-receiver.restart("restarting", null, 600)
-eventually(timeout(300.milliseconds), interval(10.milliseconds)) {
- // receiver will be stopped async
- assert(receiver.isStopped)
- assert(receiver.onStopCalled)
-}
-eventually(timeout(1.second), interval(10.milliseconds)) {
- // receiver will be started async
- assert(receiver.onStartCalled)
- assert(executor.isReceiverStarted)
+executor.callsRecorder.reset()
+receiver.callsRecorder.reset()
+receiver.restart("restarting", null, 100)
+eventually(timeout(10.seconds), interval(10.milliseconds)) {
Review comment:
So, `10.seconds` is enough? Do you need to retriever this PR to validate
more?
BTW, thank you so much for taking care of this case!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
dongjoon-hyun commented on a change in pull request #25837:
[SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
URL: https://github.com/apache/spark/pull/25837#discussion_r326465263
##
File path:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
##
@@ -180,7 +180,7 @@ class CoarseGrainedSchedulerBackendSuite extends
SparkFunSuite with LocalSparkCo
backend.driverEndpoint.askSync[Boolean](
RegisterExecutor("3", mockEndpointRef, mockAddress.host, 1, logUrls,
attributes, Map.empty))
-sc.listenerBus.waitUntilEmpty(executorUpTimeout.toMillis)
+sc.listenerBus.waitUntilEmpty()
Review comment:
For my understanding, is this reducing the original timeout because it was 1
minutes before (`private val executorUpTimeout = 1.minute`)? It seems that this
file's two instances are the only place to reduce the timeout unlike the other
places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25837: [SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
dongjoon-hyun commented on a change in pull request #25837:
[SPARK-29161][CORE][SQL][STREAMING] Unify default wait time for waitUntilEmpty
URL: https://github.com/apache/spark/pull/25837#discussion_r326465263
##
File path:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
##
@@ -180,7 +180,7 @@ class CoarseGrainedSchedulerBackendSuite extends
SparkFunSuite with LocalSparkCo
backend.driverEndpoint.askSync[Boolean](
RegisterExecutor("3", mockEndpointRef, mockAddress.host, 1, logUrls,
attributes, Map.empty))
-sc.listenerBus.waitUntilEmpty(executorUpTimeout.toMillis)
+sc.listenerBus.waitUntilEmpty()
Review comment:
For my understanding, is this reducing the original timeout because it was 1
minutes before (`private val executorUpTimeout = 1.minute`)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
AmplabJenkins removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533397117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
AmplabJenkins removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533397119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111039/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
AmplabJenkins commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533397117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
AmplabJenkins commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533397119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111039/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
SparkQA removed a comment on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533380341 **[Test build #111039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111039/testReport)** for PR 25862 at commit [`f70b3c1`](https://github.com/apache/spark/commit/f70b3c1149322162401dce1533e05ef909a1ec07). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itsvikramagr edited a comment on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
itsvikramagr edited a comment on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-533396055 > @itsvikramagr are you planning to resolve the remaining comments or waiting on second opinion? I think the config is not yet resolved. I was waiting for more comments. I think I have given enough time for more comments. Will fix the config changes and any other pending changes over the weekend. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing
SparkQA commented on issue #25862: [SPARK-23197][STREAMING][TESTS] Fix ReceiverSuite."receiver_life_cycle" to not rely on timing URL: https://github.com/apache/spark/pull/25862#issuecomment-533396953 **[Test build #111039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111039/testReport)** for PR 25862 at commit [`f70b3c1`](https://github.com/apache/spark/commit/f70b3c1149322162401dce1533e05ef909a1ec07). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-533396055 > @itsvikramagr are you planning to resolve the remaining comments or waiting on second opinion? I think the config is not yet resolved. I was waiting for more comments. Will fix the config changes and any other pending changes over the weekend. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25861: [SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism
dongjoon-hyun closed pull request #25861: [SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism URL: https://github.com/apache/spark/pull/25861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25861: [SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism
dongjoon-hyun commented on issue #25861: [SPARK-27460][TESTS][2.4] Running slowest test suites in their own forked JVMs for higher parallelism URL: https://github.com/apache/spark/pull/25861#issuecomment-533395820 Merged to `branch-2.4`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunzhusz commented on issue #18987: [SPARK-21775][Core]Dynamic Log Level Settings for executors
sunzhusz commented on issue #18987: [SPARK-21775][Core]Dynamic Log Level Settings for executors URL: https://github.com/apache/spark/pull/18987#issuecomment-533394197 spark not to supported the feature that is foolish This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25795: [WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed
cloud-fan commented on issue #25795: [WIP][SPARK-29037][Core] Spark gives
duplicate result when an application was killed
URL: https://github.com/apache/spark/pull/25795#issuecomment-533393396
> Before per insert, we should check the path whose name is started with
.spark-staging and find the longest path with sp_ prefix.
The problem here is that, how do you detect `insert overwrite table ta
select ...`? `.spark-staging-${UUID}` always exists if there is an ongoing
write.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #25776: [SPARK-28985][PYTHON][ML] Add common classes (JavaPredictor/JavaClassificationModel/JavaProbabilisticClassifier) in PYTHON
zhengruifeng commented on issue #25776: [SPARK-28985][PYTHON][ML] Add common classes (JavaPredictor/JavaClassificationModel/JavaProbabilisticClassifier) in PYTHON URL: https://github.com/apache/spark/pull/25776#issuecomment-533393082 @zero323 hi, how newly add common classes in this PR affects the end users to implement their own hierarchy? Could you please provide a user case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
LantaoJin commented on a change in pull request #25840: [SPARK-29166][SQL] Add
parameters to limit the number of dynamic partitions for data source table
URL: https://github.com/apache/spark/pull/25840#discussion_r326460911
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##
@@ -654,6 +654,45 @@ class InsertSuite extends DataSourceTest with
SharedSparkSession {
}
}
+ test("SPARK-29166: dynamic partition overwrite with limitation") {
+withSQLConf(
+ SQLConf.PARTITION_OVERWRITE_MODE.key ->
PartitionOverwriteMode.DYNAMIC.toString) {
+ withTable("t") {
+sql(
+ """
+|create table t(i int, part1 int, part2 int) using parquet
+|partitioned by (part1, part2)
+ """.stripMargin)
+
+// no restriction
+sql("insert overwrite table t partition(part1=2, part2)" +
+ " select 2, explode(array(2, 3, 4, 5))")
+checkAnswer(spark.table("t"),
Review comment:
The limitation parameter won't act on the SQLs above, it doesn't restrict
how many total partitions a table can be inserted whatever dynamic or not. They
are 4 normal queries. It only to limit some unexpected insert like "insert
overwrite table t partition(part1=2, part2) select 2, id" when "id" is a
mistake partition column. I think the behaviour in Hive is same with here, but
I will try this in Hive.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-533388922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16133/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533388908 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16132/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins removed a comment on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533388904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
AmplabJenkins removed a comment on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-533388912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533388908 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16132/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
AmplabJenkins commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-533388922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16133/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2
AmplabJenkins commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-533388912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort
AmplabJenkins commented on issue #25850: [SPARK-29177] [Core] fix zombie tasks after stage abort URL: https://github.com/apache/spark/pull/25850#issuecomment-533388904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
