[GitHub] [spark] HeartSaVioR commented on a change in pull request #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
HeartSaVioR commented on a change in pull request #22282: [SPARK-23539][SS] Add
support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#discussion_r307124628
##
File path:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala
##
@@ -158,19 +157,21 @@ abstract class KafkaRelationSuiteBase extends QueryTest
with SharedSQLContext wi
val topic = newTopic()
testUtils.createTopic(topic, partitions = 3)
testUtils.sendMessage(
- topic, ("1", Array(("once", "1".getBytes), ("twice", "2".getBytes))),
Some(0)
+ topic, ("1", Seq()), Some(0)
)
testUtils.sendMessage(
- topic, ("2", Array(("once", "2".getBytes), ("twice", "4".getBytes))),
Some(1)
+ topic, ("2", Seq(("a", "b".getBytes("UTF-8")), ("c",
"d".getBytes("UTF-8", Some(1)
Review comment:
I would be happy to deal with cleaning up. Just want to avoid adding more
for what Spark seems to use less. (I sought with ".getBytes" and
StandardCharsets.UTF_8 is used majorly.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514904039 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108146/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514904035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514904035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514904039 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108146/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514867350 **[Test build #108146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108146/testReport)** for PR 25007 at commit [`b8b7b8d`](https://github.com/apache/spark/commit/b8b7b8d00418ccc735ec7bdcc10de5e71384e8bc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514903739 **[Test build #108146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108146/testReport)** for PR 25007 at commit [`b8b7b8d`](https://github.com/apache/spark/commit/b8b7b8d00418ccc735ec7bdcc10de5e71384e8bc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hddong removed a comment on issue #23952: [SPARK-26929][SQL]fix table owner use user instead of principal when create table through spark-sql or beeline
hddong removed a comment on issue #23952: [SPARK-26929][SQL]fix table owner use user instead of principal when create table through spark-sql or beeline URL: https://github.com/apache/spark/pull/23952#issuecomment-514876679 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler
AmplabJenkins removed a comment on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler URL: https://github.com/apache/spark/pull/25244#issuecomment-514903013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler
SparkQA commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler URL: https://github.com/apache/spark/pull/25244#issuecomment-514903481 **[Test build #108150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108150/testReport)** for PR 25244 at commit [`e63bfdc`](https://github.com/apache/spark/commit/e63bfdc71586e17ce1bcacf150b1cf302155041e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler
AmplabJenkins removed a comment on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler URL: https://github.com/apache/spark/pull/25244#issuecomment-514903015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler
AmplabJenkins commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler URL: https://github.com/apache/spark/pull/25244#issuecomment-514903013 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler
AmplabJenkins commented on issue #25244: [SPARK-28499][ML] Optimize MinMaxScaler URL: https://github.com/apache/spark/pull/25244#issuecomment-514903015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13252/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
HyukjinKwon commented on a change in pull request #25245:
[SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#discussion_r307093160
##
File path:
mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
##
@@ -98,3 +103,163 @@ private[image] class ImageFileFormat extends FileFormat
with DataSourceRegister
}
}
}
+
+object ImageFileFormat {
+
+ val undefinedImageType = "Undefined"
+
+ /**
+ * (Scala-specific) OpenCV type mapping supported
+ */
+ val ocvTypes: Map[String, Int] = Map(
+undefinedImageType -> -1,
+"CV_8U" -> 0, "CV_8UC1" -> 0, "CV_8UC3" -> 16, "CV_8UC4" -> 24
+ )
+
+ /**
+ * (Java-specific) OpenCV type mapping supported
+ */
+ val javaOcvTypes: java.util.Map[String, Int] = ocvTypes.asJava
+
+ /**
+ * Schema for the image column: Row(String, Int, Int, Int, Int, Array[Byte])
+ */
+ private[image] val columnSchema = StructType(
+StructField("origin", StringType, true) ::
+StructField("height", IntegerType, true) ::
+StructField("width", IntegerType, true) ::
+StructField("nChannels", IntegerType, true) ::
+// OpenCV-compatible type: CV_8UC3 in most cases
+StructField("mode", IntegerType, true) ::
Review comment:
That's not true in structured streaming. Shall we don't change this here for
now? Sounds like orthogonal with this current PR..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514900645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514900645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514900649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108144/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514900649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108144/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
SparkQA removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514862115 **[Test build #108144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108144/testReport)** for PR 25245 at commit [`6bbb35b`](https://github.com/apache/spark/commit/6bbb35b06f518fd6febe9c65e2edcc90c7e5d095). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0
SparkQA commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0 URL: https://github.com/apache/spark/pull/25245#issuecomment-514900221 **[Test build #108144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108144/testReport)** for PR 25245 at commit [`6bbb35b`](https://github.com/apache/spark/commit/6bbb35b06f518fd6febe9c65e2edcc90c7e5d095). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
AmplabJenkins removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514899237 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108142/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
AmplabJenkins removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514899234 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
AmplabJenkins commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514899237 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108142/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
AmplabJenkins commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514899234 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
SparkQA removed a comment on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514847450 **[Test build #108142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108142/testReport)** for PR 24560 at commit [`3aa9ebd`](https://github.com/apache/spark/commit/3aa9ebd92569afd918189959ffdf602438278e39). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API
SparkQA commented on issue #24560: [SPARK-27661][SQL] Add SupportsNamespaces API URL: https://github.com/apache/spark/pull/24560#issuecomment-514898887 **[Test build #108142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108142/testReport)** for PR 24560 at commit [`3aa9ebd`](https://github.com/apache/spark/commit/3aa9ebd92569afd918189959ffdf602438278e39). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` final class SetProperty implements NamespaceChange ` * ` final class RemoveProperty implements NamespaceChange ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514895405 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514895405 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514895410 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108135/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514895410 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108135/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514834214 **[Test build #108135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108135/testReport)** for PR 25198 at commit [`9791914`](https://github.com/apache/spark/commit/979191464d803834317a2c2281b6afc3c3b09f7b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514894900 **[Test build #108135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108135/testReport)** for PR 25198 at commit [`9791914`](https://github.com/apache/spark/commit/979191464d803834317a2c2281b6afc3c3b09f7b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] chenjunjiedada commented on a change in pull request #24879: [SPARK-28042][K8S] Support using volume mount as local storage
chenjunjiedada commented on a change in pull request #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#discussion_r307114140 ## File path: docs/running-on-kubernetes.md ## @@ -285,7 +285,16 @@ The configuration properties for mounting volumes into the executor pods use pre ## Local Storage -Spark uses temporary scratch space to spill data to disk during shuffles and other operations. When using Kubernetes as the resource manager the pods will be created with an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume mounted for each directory listed in `SPARK_LOCAL_DIRS`. If no directories are explicitly specified then a default directory is created and configured appropriately. +Spark supports using volumes to spill data during shuffles and other operations. To use a volume as local storage, the volume's name should starts with `spark-local-dir-` and the mount path should be set in the spark configuration `spark.local.dir` or in the pod environment variable `SPARK_LOCAL_DIRS`, for example: Review comment: OK, now I understood your consideration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #22282: [SPARK-23539][SS] Add support for Kafka headers in Structured Streaming
HeartSaVioR commented on a change in pull request #22282: [SPARK-23539][SS] Add
support for Kafka headers in Structured Streaming
URL: https://github.com/apache/spark/pull/22282#discussion_r307108692
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
##
@@ -67,8 +67,7 @@ private[kafka010] class KafkaSourceProvider extends
DataSourceRegister
schema: Option[StructType],
providerName: String,
parameters: Map[String, String]): (String, StructType) = {
-val caseInsensitiveParams = parameters.map { case (k, v) =>
(k.toLowerCase(Locale.ROOT), v) }
-validateStreamOptions(caseInsensitiveParams)
+validateStreamOptions(parameters)
Review comment:
Ah OK my bad. That came from only reviewing last commit. Please ignore this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query
maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#discussion_r307107508 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -884,6 +885,10 @@ class Analyzer( if AttributeSet(windowExpressions.map(_.toAttribute)).intersect(conflictingAttributes) .nonEmpty => (oldVersion, oldVersion.copy(windowExpressions = newAliases(windowExpressions))) + +case oldVersion @ RecursiveReference(_, output, _) +if oldVersion.outputSet.intersect(conflictingAttributes).nonEmpty => + (oldVersion, oldVersion.copy(output = output.map(_.newInstance( Review comment: Can you add tests for this case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query
maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#discussion_r307107449 ## File path: docs/sql-keywords.md ## @@ -214,6 +214,7 @@ Below is a list of all the keywords in Spark SQL. RECORDREADERnon-reservednon-reservednon-reserved RECORDWRITERnon-reservednon-reservednon-reserved RECOVERnon-reservednon-reservednon-reserved + RECURSIVEreservednon-reservedreserved Review comment: Also, you need to update `TableIdentifierParserSuite`, too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25251: [MINOR] Trivial cleanups
maropu commented on issue #25251: [MINOR] Trivial cleanups URL: https://github.com/apache/spark/pull/25251#issuecomment-514886977 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjinleekr commented on issue #25251: [MINOR] Trivial cleanups
dongjinleekr commented on issue #25251: [MINOR] Trivial cleanups URL: https://github.com/apache/spark/pull/25251#issuecomment-514886740 @maropu I will have a try. Let me see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514886379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108138/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
SparkQA removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514836024 **[Test build #108138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108138/testReport)** for PR 25068 at commit [`474c16e`](https://github.com/apache/spark/commit/474c16ea5eae33da3f198c8f240a77ae8b1ce13e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514886374 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins removed a comment on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514886379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108138/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
AmplabJenkins commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514886374 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view
SparkQA commented on issue #25068: [SPARK-28156][SQL][BACKPORT-2.4] Self-join should not miss cached view URL: https://github.com/apache/spark/pull/25068#issuecomment-514886133 **[Test build #108138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108138/testReport)** for PR 25068 at commit [`474c16e`](https://github.com/apache/spark/commit/474c16ea5eae33da3f198c8f240a77ae8b1ce13e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514885083 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13251/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514885080 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514885053 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13251/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514885080 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514885083 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13251/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514883978 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13250/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514883973 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable
brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL]
DataSourceV2: InsertTable
URL: https://github.com/apache/spark/pull/24832#discussion_r307103748
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/TestInMemoryTableCatalog.scala
##
@@ -157,43 +170,86 @@ class InMemoryTable(
override def newWriteBuilder(options: CaseInsensitiveStringMap):
WriteBuilder = {
TestInMemoryTableCatalog.maybeSimulateFailedTableWrite(options)
-new WriteBuilder with SupportsTruncate {
- private var shouldTruncate: Boolean = false
+
+new WriteBuilder with SupportsTruncate with SupportsOverwrite with
SupportsDynamicOverwrite {
+ private var writer: BatchWrite = Append
override def truncate(): WriteBuilder = {
-shouldTruncate = true
+assert(writer == Append)
+writer = TruncateAndAppend
this
}
- override def buildForBatch(): BatchWrite = {
-if (shouldTruncate) TruncateAndAppend else Append
+ override def overwrite(filters: Array[Filter]): WriteBuilder = {
+assert(writer == Append)
+writer = new Overwrite(filters)
+this
}
+
+ override def overwriteDynamicPartitions(): WriteBuilder = {
+assert(writer == Append)
+writer = DynamicOverwrite
+this
+ }
+
+ override def buildForBatch(): BatchWrite = writer
}
}
- private object TruncateAndAppend extends BatchWrite {
+ private abstract class TestBatchWrite extends BatchWrite {
override def createBatchWriterFactory(): DataWriterFactory = {
BufferedRowsWriterFactory
}
-override def commit(messages: Array[WriterCommitMessage]): Unit = {
- replaceData(messages.map(_.asInstanceOf[BufferedRows]))
+override def abort(messages: Array[WriterCommitMessage]): Unit = {
}
+ }
-override def abort(messages: Array[WriterCommitMessage]): Unit = {
+ private object Append extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ withData(messages.map(_.asInstanceOf[BufferedRows]))
}
}
- private object Append extends BatchWrite {
-override def createBatchWriterFactory(): DataWriterFactory = {
- BufferedRowsWriterFactory
+ private object DynamicOverwrite extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ val newData = messages.map(_.asInstanceOf[BufferedRows])
+ dataMap --= newData.flatMap(_.rows.map(getKey))
+ withData(newData)
}
+ }
-override def commit(messages: Array[WriterCommitMessage]): Unit = {
- replaceData(data ++ messages.map(_.asInstanceOf[BufferedRows]))
+ private class Overwrite(filters: Array[Filter]) extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ val deleteKeys = dataMap.keys.filter { partValues =>
+filters.flatMap(splitAnd).forall {
+ case EqualTo(attr, value) =>
+partFieldNames.zipWithIndex.find(_._1 == attr) match {
+ case Some((_, partIndex)) =>
+value == partValues(partIndex)
+ case _ =>
+throw new IllegalArgumentException(s"Unknown filter attribute:
$attr")
+}
+ case f @ _ =>
Review comment:
nit, no need for `@ _`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable
brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable URL: https://github.com/apache/spark/pull/24832#discussion_r307058102 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -294,8 +294,8 @@ query ; insertInto -: INSERT OVERWRITE TABLE tableIdentifier (partitionSpec (IF NOT EXISTS)?)? #insertOverwriteTable -| INSERT INTO TABLE? tableIdentifier partitionSpec? #insertIntoTable +: INSERT OVERWRITE TABLE? multipartIdentifier (partitionSpec (IF NOT EXISTS)?)? #insertOverwriteTable +| INSERT INTO TABLE? multipartIdentifier partitionSpec? (IF NOT EXISTS)? #insertIntoTable Review comment: do we need to wrap with parentheses `(partitionSpec (IF NOT EXISTS)?)?` like above? Otherwise, what happens if there's no `partitionSpec` but the `IF NOT EXISTS`? - If the table not exists? Then wouldn't that be CTAS? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable
brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL]
DataSourceV2: InsertTable
URL: https://github.com/apache/spark/pull/24832#discussion_r307102960
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -284,9 +294,13 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
*/
override def visitInsertIntoTable(
ctx: InsertIntoTableContext): InsertTableParams = withOrigin(ctx) {
-val tableIdent = visitTableIdentifier(ctx.tableIdentifier)
+val tableIdent = visitMultipartIdentifier(ctx.multipartIdentifier)
val partitionKeys =
Option(ctx.partitionSpec).map(visitPartitionSpec).getOrElse(Map.empty)
+if (ctx.EXISTS != null) {
Review comment:
what's the point of adding this to the parser, if we're not going to support
it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable
brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL]
DataSourceV2: InsertTable
URL: https://github.com/apache/spark/pull/24832#discussion_r307103799
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/TestInMemoryTableCatalog.scala
##
@@ -149,49 +161,89 @@ private class InMemoryTable(
}
override def newWriteBuilder(options: CaseInsensitiveStringMap):
WriteBuilder = {
-new WriteBuilder with SupportsTruncate {
- private var shouldTruncate: Boolean = false
+new WriteBuilder with SupportsTruncate with SupportsOverwrite with
SupportsDynamicOverwrite {
+ private var writer: BatchWrite = Append
override def truncate(): WriteBuilder = {
-shouldTruncate = true
+assert(writer == Append)
+writer = TruncateAndAppend
+this
+ }
+
+ override def overwrite(filters: Array[Filter]): WriteBuilder = {
+assert(writer == Append)
+writer = new Overwrite(filters)
this
}
- override def buildForBatch(): BatchWrite = {
-if (shouldTruncate) TruncateAndAppend else Append
+ override def overwriteDynamicPartitions(): WriteBuilder = {
+assert(writer == Append)
+writer = DynamicOverwrite
+this
}
+
+ override def buildForBatch(): BatchWrite = writer
}
}
- private object TruncateAndAppend extends BatchWrite {
+ private abstract class TestBatchWrite extends BatchWrite {
override def createBatchWriterFactory(): DataWriterFactory = {
BufferedRowsWriterFactory
}
-override def commit(messages: Array[WriterCommitMessage]): Unit = {
- replaceData(messages.map(_.asInstanceOf[BufferedRows]))
+override def abort(messages: Array[WriterCommitMessage]): Unit = {
}
+ }
-override def abort(messages: Array[WriterCommitMessage]): Unit = {
+ private object Append extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ withData(messages.map(_.asInstanceOf[BufferedRows]))
}
}
- private object Append extends BatchWrite {
-override def createBatchWriterFactory(): DataWriterFactory = {
- BufferedRowsWriterFactory
+ private object DynamicOverwrite extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ val newData = messages.map(_.asInstanceOf[BufferedRows])
+ dataMap --= newData.flatMap(_.rows.map(getKey))
+ withData(newData)
}
+ }
-override def commit(messages: Array[WriterCommitMessage]): Unit = {
- replaceData(data ++ messages.map(_.asInstanceOf[BufferedRows]))
+ private class Overwrite(filters: Array[Filter]) extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ val deleteKeys = dataMap.keys.filter { partValues =>
+filters.exists {
+ case EqualTo(attr, value) =>
+partFieldNames.zipWithIndex.find(_._1 == attr) match {
+ case Some((_, partIndex)) =>
+value == partValues(partIndex)
+ case _ =>
+throw new IllegalArgumentException(s"Unknown filter attribute:
$attr")
+}
+ case f @ _ =>
+throw new IllegalArgumentException(s"Unsupported filter type: $f")
+}
+ }
+ dataMap --= deleteKeys
+ withData(messages.map(_.asInstanceOf[BufferedRows]))
}
+ }
-override def abort(messages: Array[WriterCommitMessage]): Unit = {
+ private object TruncateAndAppend extends TestBatchWrite {
+override def commit(messages: Array[WriterCommitMessage]): Unit =
dataMap.synchronized {
+ dataMap = mutable.Map.empty
Review comment:
@rdblue You forgot to address this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514883978 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13250/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514883973 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL] DataSourceV2: InsertTable
brkyvz commented on a change in pull request #24832: [SPARK-27845][SQL]
DataSourceV2: InsertTable
URL: https://github.com/apache/spark/pull/24832#discussion_r307103888
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala
##
@@ -23,17 +23,19 @@ import scala.collection.mutable
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalog.v2.{CatalogPlugin, Identifier,
LookupCatalog, TableCatalog}
-import org.apache.spark.sql.catalog.v2.expressions.Transform
+import org.apache.spark.sql.catalog.v2.expressions.{FieldReference,
IdentityTransform, Transform}
Review comment:
are any of the changes here needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514883941 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13250/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514881351 **[Test build #108149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108149/testReport)** for PR 24879 at commit [`6e5fcf6`](https://github.com/apache/spark/commit/6e5fcf64472cd25785673b3c9599cc59360d9381). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514883509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108149/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins removed a comment on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514883504 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514883450 **[Test build #108149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108149/testReport)** for PR 24879 at commit [`6e5fcf6`](https://github.com/apache/spark/commit/6e5fcf64472cd25785673b3c9599cc59360d9381). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514883509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108149/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
AmplabJenkins commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514883504 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514882742 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13251/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query
maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support
recursive SQL query
URL: https://github.com/apache/spark/pull/23531#discussion_r307102653
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
##
@@ -228,6 +234,156 @@ case class FilterExec(condition: Expression, child:
SparkPlan)
override def outputPartitioning: Partitioning = child.outputPartitioning
}
+/**
+ * Physical plan node for a recursive table that encapsulates the physical
plans of the anchor
+ * terms and the logical plans of the recursive terms and the maximum number
of rows to return.
+ *
+ * Anchor terms are physical plans and they are used to initialize the query
in the first run.
+ * Recursive terms are used to extend the result with new rows, They are
logical plans and contain
+ * references to the result of the previous iteration or to the so far
cumulated result. These
+ * references are updated with new statistics and compiled to physical plan
and then updated to
+ * reflect the appropriate RDD before execution.
+ *
+ * The execution terminates once the anchor terms or the current iteration of
the recursive terms
+ * return no rows or the number of cumulated rows reaches the limit.
+ *
+ * During the execution of a recursive query the previously computed results
are reused multiple
+ * times. To avoid massive recomputation of these pieces of the final result,
they are cached.
+ *
+ * @param name the name of the recursive table
+ * @param anchorTerms this child is used for initializing the query
+ * @param recursiveTerms this child is used for extending the set of results
with new rows based on
+ * the results of the previous iteration (or the anchor
in the first
+ * iteration)
+ * @param limit the maximum number of rows to return
+ */
+case class RecursiveTableExec(
+name: String,
+anchorTerms: Seq[SparkPlan],
+@transient
+val recursiveTerms: Seq[LogicalPlan],
+limit: Option[Long]) extends SparkPlan {
+ override def children: Seq[SparkPlan] = anchorTerms
+
+ override def output: Seq[Attribute] =
anchorTerms.head.output.map(_.withNullability(true))
+
+ override def simpleString(maxFields: Int): String =
+s"RecursiveTable $name${limit.map(", " + _).getOrElse("")}"
+
+ override def innerChildren: Seq[QueryPlan[_]] = recursiveTerms ++
super.innerChildren
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val storageLevel =
StorageLevel.fromString(conf.getConf(SQLConf.RECURSION_CACHE_STORAGE_LEVEL))
+
+val prevIterationRDDs = ArrayBuffer.empty[RDD[InternalRow]]
+var prevIterationCount = 0L
+
+val anchorTermsIterator = anchorTerms.iterator
+while (anchorTermsIterator.hasNext && limit.forall(_ >
prevIterationCount)) {
+ val anchorTerm = anchorTermsIterator.next()
+
+ lazy val cumulatedResult = if (prevIterationRDDs.size > 1) {
+sparkContext.union(prevIterationRDDs)
+ } else {
+prevIterationRDDs.head
+ }
+
+ anchorTerm.foreach {
+case rr: RecursiveReferenceExec if rr.name == name =>
rr.recursiveTable = cumulatedResult
+case _ =>
+ }
+
+ val rdd = anchorTerm.execute().map(_.copy()).persist(storageLevel)
+ val count = rdd.count()
+ if (count > 0) {
+prevIterationRDDs += rdd
+prevIterationCount += count
+ }
+}
+
+val cumulatedRDDs = ArrayBuffer(prevIterationRDDs: _*)
+var cumulatedCount = prevIterationCount
+var level = 0
+val levelLimit = conf.getConf(SQLConf.RECURSION_LEVEL_LIMIT)
Review comment:
`conf.recursionLevelLimit`? Also, can you move this definition outside the
loop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support recursive SQL query
maropu commented on a change in pull request #23531: [SPARK-24497][SQL] Support
recursive SQL query
URL: https://github.com/apache/spark/pull/23531#discussion_r307102482
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
##
@@ -228,6 +234,156 @@ case class FilterExec(condition: Expression, child:
SparkPlan)
override def outputPartitioning: Partitioning = child.outputPartitioning
}
+/**
+ * Physical plan node for a recursive table that encapsulates the physical
plans of the anchor
+ * terms and the logical plans of the recursive terms and the maximum number
of rows to return.
+ *
+ * Anchor terms are physical plans and they are used to initialize the query
in the first run.
+ * Recursive terms are used to extend the result with new rows, They are
logical plans and contain
+ * references to the result of the previous iteration or to the so far
cumulated result. These
+ * references are updated with new statistics and compiled to physical plan
and then updated to
+ * reflect the appropriate RDD before execution.
+ *
+ * The execution terminates once the anchor terms or the current iteration of
the recursive terms
+ * return no rows or the number of cumulated rows reaches the limit.
+ *
+ * During the execution of a recursive query the previously computed results
are reused multiple
+ * times. To avoid massive recomputation of these pieces of the final result,
they are cached.
+ *
+ * @param name the name of the recursive table
+ * @param anchorTerms this child is used for initializing the query
+ * @param recursiveTerms this child is used for extending the set of results
with new rows based on
+ * the results of the previous iteration (or the anchor
in the first
+ * iteration)
+ * @param limit the maximum number of rows to return
+ */
+case class RecursiveTableExec(
+name: String,
+anchorTerms: Seq[SparkPlan],
+@transient
+val recursiveTerms: Seq[LogicalPlan],
+limit: Option[Long]) extends SparkPlan {
+ override def children: Seq[SparkPlan] = anchorTerms
+
+ override def output: Seq[Attribute] =
anchorTerms.head.output.map(_.withNullability(true))
+
+ override def simpleString(maxFields: Int): String =
+s"RecursiveTable $name${limit.map(", " + _).getOrElse("")}"
+
+ override def innerChildren: Seq[QueryPlan[_]] = recursiveTerms ++
super.innerChildren
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val storageLevel =
StorageLevel.fromString(conf.getConf(SQLConf.RECURSION_CACHE_STORAGE_LEVEL))
Review comment:
`conf.recursionCacheStorageLevel`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514881458 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13250/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage
SparkQA commented on issue #24879: [SPARK-28042][K8S] Support using volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-514881351 **[Test build #108149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108149/testReport)** for PR 24879 at commit [`6e5fcf6`](https://github.com/apache/spark/commit/6e5fcf64472cd25785673b3c9599cc59360d9381). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25251: [MINOR] Trivial cleanups
maropu commented on issue #25251: [MINOR] Trivial cleanups URL: https://github.com/apache/spark/pull/25251#issuecomment-514880267 At the same time, can you check unused imports by using IDE supports? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514878393 **[Test build #108148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108148/testReport)** for PR 25236 at commit [`fd0a1d2`](https://github.com/apache/spark/commit/fd0a1d2cb240e4a0f0f8e442e083dfa221cfce00). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
dongjoon-hyun commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514878133 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25242: [SPARK-28497][SQL] Disallow upcasting complex data types to string type
HyukjinKwon commented on a change in pull request #25242: [SPARK-28497][SQL]
Disallow upcasting complex data types to string type
URL: https://github.com/apache/spark/pull/25242#discussion_r307098867
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala
##
@@ -196,6 +196,43 @@ class EncoderResolutionSuite extends PlanTest {
encoder.resolveAndBind(attrs)
}
+ test("SPARK-28497: complex type is not compatible with string encoder
schema") {
+val encoder = ExpressionEncoder[String]
+
+{
+ val attrs = Seq('a.struct('x.long))
+
assert(intercept[AnalysisException](encoder.resolveAndBind(attrs)).message ==
+s"""
+ |Cannot up cast `a` from struct to string.
+ |The type path of the target object is:
+ |- root class: "java.lang.String"
+ |You can either add an explicit cast to the input data or choose a
higher precision type
+""".stripMargin.trim + " of the field in the target object")
+}
+
+{
+ val attrs = Seq('a.array(StringType))
+
assert(intercept[AnalysisException](encoder.resolveAndBind(attrs)).message ==
+s"""
+ |Cannot up cast `a` from array to string.
Review comment:
Oh, it was the same comment as
https://github.com/apache/spark/pull/25242#discussion_r307064357
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] hddong commented on issue #23952: [SPARK-26929][SQL]fix table owner use user instead of principal when create table through spark-sql or beeline
hddong commented on issue #23952: [SPARK-26929][SQL]fix table owner use user instead of principal when create table through spark-sql or beeline URL: https://github.com/apache/spark/pull/23952#issuecomment-514876679 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25242: [SPARK-28497][SQL] Disallow upcasting complex data types to string type
cloud-fan commented on a change in pull request #25242: [SPARK-28497][SQL]
Disallow upcasting complex data types to string type
URL: https://github.com/apache/spark/pull/25242#discussion_r307097063
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala
##
@@ -196,6 +196,43 @@ class EncoderResolutionSuite extends PlanTest {
encoder.resolveAndBind(attrs)
}
+ test("SPARK-28497: complex type is not compatible with string encoder
schema") {
+val encoder = ExpressionEncoder[String]
+
+{
+ val attrs = Seq('a.struct('x.long))
+
assert(intercept[AnalysisException](encoder.resolveAndBind(attrs)).message ==
+s"""
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108136/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875176 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875176 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108136/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514834222 **[Test build #108136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108136/testReport)** for PR 25007 at commit [`9f17b9b`](https://github.com/apache/spark/commit/9f17b9bbf0d3d5677abf424c5b2d4d3b93dfc95a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875071 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108140/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875069 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875071 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108140/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514875069 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514837877 **[Test build #108140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108140/testReport)** for PR 25007 at commit [`56fa450`](https://github.com/apache/spark/commit/56fa450b4d703c7bc34f338e2ed0cd21fc82a98c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514874761 **[Test build #108136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108136/testReport)** for PR 25007 at commit [`9f17b9b`](https://github.com/apache/spark/commit/9f17b9bbf0d3d5677abf424c5b2d4d3b93dfc95a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
AmplabJenkins removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514874301 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108143/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514874681 **[Test build #108140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108140/testReport)** for PR 25007 at commit [`56fa450`](https://github.com/apache/spark/commit/56fa450b4d703c7bc34f338e2ed0cd21fc82a98c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
AmplabJenkins commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514874301 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108143/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
SparkQA removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514854663 **[Test build #108143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108143/testReport)** for PR 25249 at commit [`10678ad`](https://github.com/apache/spark/commit/10678ad436e9c50499703df96fae64f782e8722f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
AmplabJenkins commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514874295 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
AmplabJenkins removed a comment on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514874295 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
SparkQA commented on issue #25249: [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor URL: https://github.com/apache/spark/pull/25249#issuecomment-514874161 **[Test build #108143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108143/testReport)** for PR 25249 at commit [`10678ad`](https://github.com/apache/spark/commit/10678ad436e9c50499703df96fae64f782e8722f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514873120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514873123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108139/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514873123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108139/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514873120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514836023 **[Test build #108139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108139/testReport)** for PR 25007 at commit [`e53a001`](https://github.com/apache/spark/commit/e53a001b30a15ebc06df6b62c5650ae6f3213477). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-514872726 **[Test build #108139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108139/testReport)** for PR 25007 at commit [`e53a001`](https://github.com/apache/spark/commit/e53a001b30a15ebc06df6b62c5650ae6f3213477). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
