[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20659 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88260/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20659 **[Test build #88260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88260/testReport)** for PR 20659 at commit [`03138d8`](https://github.com/apache/spark/commit/03138d8cc3d0069600ffda37964d38b17a0f2e58). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20833 **[Test build #88264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88264/testReport)** for PR 20833 at commit [`a300848`](https://github.com/apache/spark/commit/a3008487a3db6ea56107f409334c4484f2ac4def). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88259/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20817 **[Test build #88259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88259/testReport)** for PR 20817 at commit [`cd73b4c`](https://github.com/apache/spark/commit/cd73b4c42051af17db1260b83a49290f6519b352). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class RandomUUIDGenerator(randomSeed: Long) ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88262/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88262/testReport)** for PR 20831 at commit [`a3f86b2`](https://github.com/apache/spark/commit/a3f86b23c7e75428446ac95d2e527bcdd5562a5f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20824: [SPARK-23683][SQL][FOLLOW-UP] FileCommitProtocol.instant...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20824 **[Test build #88263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88263/testReport)** for PR 20824 at commit [`64602ae`](https://github.com/apache/spark/commit/64602ae97a5318c674d09238f36ff1fec073c97e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20824: [SPARK-23683][SQL][FOLLOW-UP] FileCommitProtocol.instant...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20824 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20824: [SPARK-23683][SQL][FOLLOW-UP] FileCommitProtocol.instant...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20824 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1530/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20824: [SPARK-23683][SQL][FOLLOW-UP] FileCommitProtocol....
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/20824#discussion_r174740987 --- Diff: core/src/test/scala/org/apache/spark/internal/io/FileCommitProtocolInstantiationSuite.scala --- @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.internal.io + +import org.apache.spark.SparkFunSuite + +/** + * Unit tests for instantiation of FileCommitProtocol implementations. + */ +class FileCommitProtocolInstantiationSuite extends SparkFunSuite { + + test("Dynamic partitions require appropriate constructor") { + +// you cannot instantiate a two-arg client with dynamic partitions +// enabled. +val ex = intercept[IllegalArgumentException] { + instantiateClassic(true) +} +// check the contents of the message and rethrow if unexpected +if (!ex.toString.contains("Dynamic Partition Overwrite")) { --- End diff -- done --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20833 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20833 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88261/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20833 **[Test build #88261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88261/testReport)** for PR 20833 at commit [`91e53d8`](https://github.com/apache/spark/commit/91e53d87b0f5503ba7e9c9bb6a7258ef30f87c9d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20824: [SPARK-23683][SQL][FOLLOW-UP] FileCommitProtocol.instant...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/20824 Fixed the title, used the new JIRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88257/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20827 **[Test build #88257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88257/testReport)** for PR 20827 at commit [`68650ff`](https://github.com/apache/spark/commit/68650ff8c2f3a90c55b5bf4345c16a92fda3782a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class PrettyNamedExpression(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88258/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88258/testReport)** for PR 20831 at commit [`e1f28e2`](https://github.com/apache/spark/commit/e1f28e2c6c9ba99c92fea339946323c9490062d4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88262/testReport)** for PR 20831 at commit [`a3f86b2`](https://github.com/apache/spark/commit/a3f86b23c7e75428446ac95d2e527bcdd5562a5f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1529/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange w...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20831#discussion_r174715180 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala --- @@ -68,6 +69,15 @@ case class InMemoryRelation( override protected def innerChildren: Seq[SparkPlan] = Seq(child) + override def doCanonicalize(): logical.LogicalPlan = +copy(output = output.map(QueryPlan.normalizeExprId(_, child.output)), + storageLevel = new StorageLevel(), --- End diff -- Ok. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20833 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20833: [SPARK-23692][SQL]Print metadata of files when infer sch...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20833 **[Test build #88261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88261/testReport)** for PR 20833 at commit [`91e53d8`](https://github.com/apache/spark/commit/91e53d87b0f5503ba7e9c9bb6a7258ef30f87c9d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20833: [SPARK-23692][SQL]Print metadata of files when in...
GitHub user caneGuy opened a pull request: https://github.com/apache/spark/pull/20833 [SPARK-23692][SQL]Print metadata of files when infer schema failed ## What changes were proposed in this pull request? A trivial modify. Currently, when we had no input files to infer schema,we will throw below exception. For some users it may be misleading.If we can print files' metadata it will be more clearer. `Caused by: org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually.; at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:189) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:189) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$getOrInferFileFormatSchema(DataSource.scala:188) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152) at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:441) at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:425) at com.xiaomi.matrix.pipeline.jobspark.importer.MatrixAdEventDailyImportJob.(MatrixAdEventDailyImportJob.scala:18)` ## How was this patch tested? Exsist tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/caneGuy/spark zhoukang/modify-log Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20833.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20833 commit 91e53d87b0f5503ba7e9c9bb6a7258ef30f87c9d Author: zhoukangDate: 2018-03-15T08:53:06Z Print metadata of files when infer schema failed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20659 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20659 **[Test build #88260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88260/testReport)** for PR 20659 at commit [`03138d8`](https://github.com/apache/spark/commit/03138d8cc3d0069600ffda37964d38b17a0f2e58). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange w...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20831#discussion_r174707266 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala --- @@ -68,6 +69,15 @@ case class InMemoryRelation( override protected def innerChildren: Seq[SparkPlan] = Seq(child) + override def doCanonicalize(): logical.LogicalPlan = +copy(output = output.map(QueryPlan.normalizeExprId(_, child.output)), + storageLevel = new StorageLevel(), --- End diff -- `StorageLevel.NONE`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1528/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20832: [SPARK-20536][SQL] Extend ColumnName to create St...
Github user efimpoberezkin commented on a diff in the pull request: https://github.com/apache/spark/pull/20832#discussion_r174706413 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala --- @@ -1208,85 +1208,172 @@ class ColumnName(name: String) extends Column(name) { */ def boolean: StructField = StructField(name, BooleanType) + /** + * Creates a new `StructField` of type boolean. + * @since 2.3.0 --- End diff -- Done --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20832: [SPARK-20536][SQL] Extend ColumnName to create St...
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/20832#discussion_r174700327 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala --- @@ -1208,85 +1208,172 @@ class ColumnName(name: String) extends Column(name) { */ def boolean: StructField = StructField(name, BooleanType) + /** + * Creates a new `StructField` of type boolean. + * @since 2.3.0 + */ + def boolean(nullable: Boolean): StructField = StructField(name, BooleanType, nullable) + /** * Creates a new `StructField` of type byte. * @since 1.3.0 */ def byte: StructField = StructField(name, ByteType) + /** + * Creates a new `StructField` of type byte. + * @since 2.3.0 --- End diff -- `2.4.0` and in the other places too (unless they patch 2.3.0 and becomes 2.3.1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20800: [SPARK-23627][SQL] Provide isEmpty in Dataset
Github user goungoun commented on the issue: https://github.com/apache/spark/pull/20800 For additional check that I mentioned. The following code shows that Spark users does not need to add take(1). ds.rdd.take(1).isEmpty is redundant. [RDD.scala](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala) `def isEmpty(): Boolean = withScope { partitions.length == 0 || take(1).length == 0 }` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20812: [SPARK-23669] Executors fetch jars and name the jars wit...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20812 The idea looks good, just a few comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20812: [SPARK-23669] Executors fetch jars and name the j...
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20812#discussion_r174699658 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -453,8 +454,13 @@ private[spark] object Utils extends Logging { securityMgr: SecurityManager, hadoopConf: Configuration, timestamp: Long, - useCache: Boolean): File = { -val fileName = decodeFileNameInURI(new URI(url)) + useCache: Boolean, + withMD5Prefix: Boolean = false): File = { +val fileName = if (withMD5Prefix) { + DigestUtils.md5Hex(url) + "-" + decodeFileNameInURI(new URI(url)) --- End diff -- How about `s"${DigestUtils.md5Hex(url)}-${decodeFileNameInURI(new URI(url))}"`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20812: [SPARK-23669] Executors fetch jars and name the j...
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20812#discussion_r174699762 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -752,11 +752,10 @@ private[spark] class Executor( if (currentTimeStamp < timestamp) { logInfo("Fetching " + name + " with timestamp " + timestamp) // Fetch file with useCache mode, close cache for local mode. - Utils.fetchFile(name, new File(SparkFiles.getRootDirectory()), conf, -env.securityManager, hadoopConf, timestamp, useCache = !isLocal) + val url = Utils.fetchFile(name, new File(SparkFiles.getRootDirectory()), conf, +env.securityManager, hadoopConf, timestamp, useCache = !isLocal, +conf.getBoolean("spark.jars.withDecoratedName", false)).toURI.toURL --- End diff -- should we have this conf in `internal/conf` ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20832: [SPARK-20536][SQL] Extend ColumnName to create St...
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/20832#discussion_r174699743 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala --- @@ -1208,85 +1208,172 @@ class ColumnName(name: String) extends Column(name) { */ def boolean: StructField = StructField(name, BooleanType) + /** + * Creates a new `StructField` of type boolean. + * @since 2.3.0 --- End diff -- `2.4.0` I think --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1527/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20817 **[Test build #88259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88259/testReport)** for PR 20817 at commit [`cd73b4c`](https://github.com/apache/spark/commit/cd73b4c42051af17db1260b83a49290f6519b352). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20812: [SPARK-23669] Executors fetch jars and name the j...
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20812#discussion_r174699183 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -752,11 +752,10 @@ private[spark] class Executor( if (currentTimeStamp < timestamp) { logInfo("Fetching " + name + " with timestamp " + timestamp) // Fetch file with useCache mode, close cache for local mode. - Utils.fetchFile(name, new File(SparkFiles.getRootDirectory()), conf, -env.securityManager, hadoopConf, timestamp, useCache = !isLocal) + val url = Utils.fetchFile(name, new File(SparkFiles.getRootDirectory()), conf, +env.securityManager, hadoopConf, timestamp, useCache = !isLocal, +conf.getBoolean("spark.jars.withDecoratedName", false)).toURI.toURL currentJars(name) = timestamp - // Add it to our class loader - val url = new File(SparkFiles.getRootDirectory(), localName).toURI.toURL --- End diff -- shall we still need the `localName` here? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20825: add impurity stats in tree leaf node debug string
Github user davies commented on the issue: https://github.com/apache/spark/pull/20825 cc @zsxwing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1526/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88258/testReport)** for PR 20831 at commit [`e1f28e2`](https://github.com/apache/spark/commit/e1f28e2c6c9ba99c92fea339946323c9490062d4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20831 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI
Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20803 @cloud-fan, please review. The test result is: val df = spark.sql("x") spark.range(10).count() // noting show in UI df.collect() // show sql text "x" on the UI df.count() // show sql text "x" on the UI df.show() // show sql text "x" on the UI df.filter(...).collect() // show sql text "x" on the UI --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20832: [SPARK-20536][SQL] Extend ColumnName to create StructFie...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20832 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20832: [SPARK-20536][SQL] Extend ColumnName to create StructFie...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20832 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20832: [SPARK-20536][SQL] Extend ColumnName to create St...
GitHub user efimpoberezkin opened a pull request: https://github.com/apache/spark/pull/20832 [SPARK-20536][SQL] Extend ColumnName to create StructFields with explicit nullable ## What changes were proposed in this pull request? Extended ColumnName with methods that create StructFields with explicit nullable property ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/efimpoberezkin/spark pr/extend-ColumnName Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20832.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20832 commit 101f3451e07d64cb650699e3b49b1c35fc7cce20 Author: Efim PoberezkinDate: 2018-03-14T10:22:59Z [SPARK-20536][SQL] Extend ColumnName to create StructFields with explicit nullable --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20689: [SPARK-23533][SS] Add support for changing Contin...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20689 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20827 **[Test build #88257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88257/testReport)** for PR 20827 at commit [`68650ff`](https://github.com/apache/spark/commit/68650ff8c2f3a90c55b5bf4345c16a92fda3782a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1525/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20827 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20827 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88255/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20827: [SPARK-23666][SQL] Do not display exprIds of Alias in us...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20827 **[Test build #88255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88255/testReport)** for PR 20827 at commit [`68650ff`](https://github.com/apache/spark/commit/68650ff8c2f3a90c55b5bf4345c16a92fda3782a). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class PrettyNamedExpression(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88256/testReport)** for PR 20831 at commit [`e1f28e2`](https://github.com/apache/spark/commit/e1f28e2c6c9ba99c92fea339946323c9490062d4). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88256/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20689: [SPARK-23533][SS] Add support for changing ContinuousDat...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20689 Thanks! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20831 **[Test build #88256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88256/testReport)** for PR 20831 at commit [`e1f28e2`](https://github.com/apache/spark/commit/e1f28e2c6c9ba99c92fea339946323c9490062d4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange when cac...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20831 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1524/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20831: [SPARK-23614][SQL] Fix incorrect reuse exchange w...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/20831 [SPARK-23614][SQL] Fix incorrect reuse exchange when caching is used ## What changes were proposed in this pull request? We should provide customized canonicalize plan for `InMemoryRelation` and `InMemoryTableScanExec`. Otherwise, we can wrongly treat two different cached plans as same result. It causes wrongly reused exchange then. ## How was this patch tested? Added unit test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 SPARK-23614 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20831.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20831 commit e1f28e2c6c9ba99c92fea339946323c9490062d4 Author: Liang-Chi HsiehDate: 2018-03-15T06:16:22Z Fix incorrect reuse exchange when caching is used. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20689: [SPARK-23533][SS] Add support for changing ContinuousDat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20689 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20689: [SPARK-23533][SS] Add support for changing ContinuousDat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20689 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88253/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20689: [SPARK-23533][SS] Add support for changing ContinuousDat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20689 **[Test build #88253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88253/testReport)** for PR 20689 at commit [`992e2c1`](https://github.com/apache/spark/commit/992e2c1de84b9e82875f47ecc21aad2a299038a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88251/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20817: [SPARK-23599][SQL] Add a UUID generator from Pseudo-Rand...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20817 **[Test build #88251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88251/testReport)** for PR 20817 at commit [`b2062c7`](https://github.com/apache/spark/commit/b2062c78fa295f7374fe84f22144f923c242ada6). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class RandomUUIDGenerator(randomSeed: Long) ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org