[GitHub] spark pull request #16308: [SPARK-18350][SQL][WIP] Support session local tim...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/16308#discussion_r92982685 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -111,7 +112,8 @@ case class CatalogTablePartition( */ def toRow(partitionSchema: StructType): InternalRow = { InternalRow.fromSeq(partitionSchema.map { field => - Cast(Literal(spec(field.name)), field.dataType).eval() + Cast(Literal(spec(field.name)), field.dataType, +DateTimeUtils.defaultTimeZone().getID).eval() --- End diff -- Hmm, now I think we should use timezone settings for partition values, because the values are also parts of data so they should be affected by the settings. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16291 **[Test build #70345 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70345/testReport)** for PR 16291 at commit [`66e4f12`](https://github.com/apache/spark/commit/66e4f12c726d7cc49620f9c5747ab6b1d70d5bf3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16290 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70335/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16290 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16290 **[Test build #70335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70335/testReport)** for PR 16290 at commit [`f7b4772`](https://github.com/apache/spark/commit/f7b4772a4b72c28047afb7d614e3af3317af896d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/13909#discussion_r92979265 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) extends Expression { } override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { -val arrayClass = classOf[GenericArrayData].getName -val values = ctx.freshName("values") -ctx.addMutableState("Object[]", values, s"this.$values = null;") +val array = ctx.freshName("array") -ev.copy(code = s""" - this.$values = new Object[${children.size}];""" + +val et = dataType.elementType +val evals = children.map(e => e.genCode(ctx)) +val isPrimitiveArray = ctx.isPrimitiveType(et) +val primitiveTypeName = if (isPrimitiveArray) ctx.primitiveTypeName(et) else "" +val (preprocess, arrayData, arrayWriter) = + GenArrayData.getCodeArrayData(ctx, et, children.size, isPrimitiveArray, array) + +ev.copy(code = + preprocess + ctx.splitExpressions( ctx.INPUT_ROW, -children.zipWithIndex.map { case (e, i) => - val eval = e.genCode(ctx) - eval.code + s""" -if (${eval.isNull}) { - $values[$i] = null; +evals.zipWithIndex.map { case (eval, i) => + eval.code + +(if (isPrimitiveArray) { + (if (!children(i).nullable) { +s"\n$arrayWriter.write($i, ${eval.value});" + } else { +s""" +if (${eval.isNull}) { + $arrayWriter.setNull$primitiveTypeName($i); +} else { + $arrayWriter.write($i, ${eval.value}); +} + """ + }) } else { - $values[$i] = ${eval.value}; -} - """ + s""" + if (${eval.isNull}) { +$array[$i] = null; + } else { +$array[$i] = ${eval.value}; + } + """ +}) }) + - s""" -final ArrayData ${ev.value} = new $arrayClass($values); -this.$values = null; - """, isNull = "false") + s"\nfinal ArrayData ${ev.value} = $arrayData;\n", + isNull = "false") } override def prettyName: String = "array" } +private [sql] object GenArrayData { + // This function returns Java code pieces based on DataType and isPrimitive + // for allocation of ArrayData class + def getCodeArrayData( + ctx: CodegenContext, + dt: DataType, + size: Int, + isPrimitive : Boolean, + array: String): (String, String, String) = { +if (!isPrimitive) { + val arrayClass = classOf[GenericArrayData].getName + ctx.addMutableState("Object[]", array, +s"this.$array = new Object[${size}];") + ("", s"new $arrayClass($array)", null) +} else { + val holder = ctx.freshName("holder") + val arrayWriter = ctx.freshName("createArrayWriter") + val unsafeArrayClass = classOf[UnsafeArrayData].getName + val holderClass = classOf[BufferHolder].getName + val arrayWriterClass = classOf[UnsafeArrayWriter].getName + ctx.addMutableState(unsafeArrayClass, array, "") + ctx.addMutableState(holderClass, holder, "") + ctx.addMutableState(arrayWriterClass, arrayWriter, "") + val baseOffset = Platform.BYTE_ARRAY_OFFSET + val unsafeArraySizeInBytes = +UnsafeArrayData.calculateHeaderPortionInBytes(size) + +ByteArrayMethods.roundNumberOfBytesToNearestWord(dt.defaultSize * size) + + (s""" +$array = new $unsafeArrayClass(); +$holder = new $holderClass($unsafeArraySizeInBytes); +$arrayWriter = new $arrayWriterClass(); --- End diff -- @viirya `org.apache.spark.sql.DataFrameSuite.Star Expansion - CreateStruct and CreateArray` failed at [this test](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69969/testReport/). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi
[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/13909#discussion_r92979329 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) extends Expression { } override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { -val arrayClass = classOf[GenericArrayData].getName -val values = ctx.freshName("values") -ctx.addMutableState("Object[]", values, s"this.$values = null;") +val array = ctx.freshName("array") -ev.copy(code = s""" - this.$values = new Object[${children.size}];""" + +val et = dataType.elementType +val evals = children.map(e => e.genCode(ctx)) +val isPrimitiveArray = ctx.isPrimitiveType(et) +val primitiveTypeName = if (isPrimitiveArray) ctx.primitiveTypeName(et) else "" +val (preprocess, arrayData, arrayWriter) = + GenArrayData.getCodeArrayData(ctx, et, children.size, isPrimitiveArray, array) + +ev.copy(code = + preprocess + ctx.splitExpressions( ctx.INPUT_ROW, -children.zipWithIndex.map { case (e, i) => - val eval = e.genCode(ctx) - eval.code + s""" -if (${eval.isNull}) { - $values[$i] = null; +evals.zipWithIndex.map { case (eval, i) => + eval.code + +(if (isPrimitiveArray) { + (if (!children(i).nullable) { +s"\n$arrayWriter.write($i, ${eval.value});" + } else { +s""" +if (${eval.isNull}) { --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70334/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70334/testReport)** for PR 16335 at commit [`c5d6938`](https://github.com/apache/spark/commit/c5d69388823aa7fc9d75826cabacfb666ee665d5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16321 **[Test build #70344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70344/testReport)** for PR 16321 at commit [`471740b`](https://github.com/apache/spark/commit/471740b816014234359458fdcac73a6599e16a5b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16282 **[Test build #70343 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70343/testReport)** for PR 16282 at commit [`fd63513`](https://github.com/apache/spark/commit/fd63513d9e1ac3978f7ffe29382aff46ad86027b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16322 **[Test build #70342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70342/testReport)** for PR 16322 at commit [`2044bc0`](https://github.com/apache/spark/commit/2044bc0ebcb7eb760ba1b32a5a1d3cec6066a6dc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16335 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70333/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70333/testReport)** for PR 16335 at commit [`e309285`](https://github.com/apache/spark/commit/e3092856fa207d1fcc118580d77a64bc653c62be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16337 cc @hvanhovell @nsyca --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16337 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16282 SPARK-18907 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16282 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16337 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14365 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16336 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16336 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70340/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14365 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70336/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16336 **[Test build #70340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70340/testReport)** for PR 16336 at commit [`2aa11d8`](https://github.com/apache/spark/commit/2aa11d831401969a1c6052ece749ec4d56beb117). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14365 **[Test build #70336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)** for PR 14365 at commit [`141cb1d`](https://github.com/apache/spark/commit/141cb1d5232a14d72f6411afdb1212cf7ef4de0a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN s...
GitHub user kevinyu98 opened a pull request: https://github.com/apache/spark/pull/16337 [SPARK-18871][SQL] New test cases for IN/NOT IN subquery ## What changes were proposed in this pull request? This PR extends the existing IN/NOT IN subquery test cases coverage, adding these cases: - simple case - GROUP BY - ORDER BY - HAVING - JOIN - UNION | UNION ALL | UNION DISTINCT | INTERSECT | EXCEPT - CTE ## How was this patch tested? This patch is adding tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevinyu98/spark spark-18871 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16337.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16337 commit 3b44c5978bd44db986621d3e8511e9165b66926b Author: Kevin Yu Date: 2016-04-20T18:06:30Z adding testcase commit 18b4a31c687b264b50aa5f5a74455956911f738a Author: Kevin Yu Date: 2016-04-22T21:48:00Z Merge remote-tracking branch 'upstream/master' commit 4f4d1c8f2801b1e662304ab2b33351173e71b427 Author: Kevin Yu Date: 2016-04-23T16:50:19Z Merge remote-tracking branch 'upstream/master' get latest code from upstream commit f5f0cbed1eb5754c04c36933b374c3b3d2ae4f4e Author: Kevin Yu Date: 2016-04-23T22:20:53Z Merge remote-tracking branch 'upstream/master' adding trim characters support commit d8b2edbd13ee9a4f057bca7dcb0c0940e8e867b8 Author: Kevin Yu Date: 2016-04-25T20:24:33Z Merge remote-tracking branch 'upstream/master' get latest code for pr12646 commit 196b6c66b0d55232f427c860c0e7c6876c216a67 Author: Kevin Yu Date: 2016-04-25T23:45:57Z Merge remote-tracking branch 'upstream/master' merge latest code commit f37a01e005f3e27ae2be056462d6eb6730933ba5 Author: Kevin Yu Date: 2016-04-27T14:15:06Z Merge remote-tracking branch 'upstream/master' merge upstream/master commit bb5b01fd3abeea1b03315eccf26762fcc23f80c0 Author: Kevin Yu Date: 2016-04-30T23:49:31Z Merge remote-tracking branch 'upstream/master' commit bde5820a181cf84e0879038ad8c4cebac63c1e24 Author: Kevin Yu Date: 2016-05-04T03:52:31Z Merge remote-tracking branch 'upstream/master' commit 5f7cd96d495f065cd04e8e4cc58461843e45bc8d Author: Kevin Yu Date: 2016-05-10T21:14:50Z Merge remote-tracking branch 'upstream/master' commit 893a49af0bfd153ccb59ba50b63a232660e0eada Author: Kevin Yu Date: 2016-05-13T18:20:39Z Merge remote-tracking branch 'upstream/master' commit 4bbe1fd4a3ebd50338ccbe07dc5887fe289cd53d Author: Kevin Yu Date: 2016-05-17T21:58:14Z Merge remote-tracking branch 'upstream/master' commit b2dd795e23c36cbbd022f07a10c0cf21c85eb421 Author: Kevin Yu Date: 2016-05-18T06:37:13Z Merge remote-tracking branch 'upstream/master' commit 8c3e5da458dbff397ed60fcb68f2a46d87ab7ba4 Author: Kevin Yu Date: 2016-05-18T16:18:16Z Merge remote-tracking branch 'upstream/master' commit a0eaa408e847fbdc3ac5b26348588ee0a1e276c7 Author: Kevin Yu Date: 2016-05-19T04:28:20Z Merge remote-tracking branch 'upstream/master' commit d03c940ed89795fa7fe1d1e9f511363b22cdf19d Author: Kevin Yu Date: 2016-05-19T21:24:33Z Merge remote-tracking branch 'upstream/master' commit d728d5e002082e571ac47292226eb8b2614f479f Author: Kevin Yu Date: 2016-05-24T20:32:57Z Merge remote-tracking branch 'upstream/master' commit ea104ddfbf7d180ed1bc53dd9a1005010264aa1f Author: Kevin Yu Date: 2016-05-25T22:52:57Z Merge remote-tracking branch 'upstream/master' commit 6ab1215b781ad0cccf1752f3a625b4e4e371c38e Author: Kevin Yu Date: 2016-05-27T17:18:46Z Merge remote-tracking branch 'upstream/master' commit 0c566533705331697eb1b287b30c8b16111f6fa2 Author: Kevin Yu Date: 2016-06-01T06:48:57Z Merge remote-tracking branch 'upstream/master' commit d7a187490b31185d0a803cbbdeda67cb26c40056 Author: Kevin Yu Date: 2016-06-01T22:55:17Z Merge remote-tracking branch 'upstream/master' commit 85d35002ce864d5ce6fd3be7215a868a8867caf9 Author: Kevin Yu Date: 2016-06-02T14:08:30Z Merge remote-tracking branch 'upstream/master' commit c056f91036ec75d1e2c93f6f47ad842eb28a3e0b Author: Kevin Yu Date: 2016-06-03T06:06:51Z Merge remote-tracking branch 'upstream/master' commit 0b8189dd454897ae73bb3a5ffc245b2c65f6b226 Author: Kevin Yu Date: 2016-06-03T23:33:39Z Merge remote-tracking branch 'upstream/master' commit c2ea31de654bd8c32ab0dd9a0362ee90b47c8756 Author: Kevin Yu Date: 2016-06-06T21:04:09Z Merge remote-tracking branch 'upstream/master' commit a2d3056e148a1b38b1b2bf84eabf13962a452e36 Author: Kevin Yu Date: 2016-06-08T02:52:43Z Merge remote-tracking branch 'upstream/master' commit 39e564887667b57ec15b87f9ee6c760fd1938a15
[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16336 @dongjoon-hyun Just to make sure, could we actually build each individually, `cd ./python/docs && make html` for Python and `cd ./R && sh create-docs.sh` for R? (FWIW, I support this PR even if so because now these can be done in a single place). BTW, I believe it would be nicer if the options are documented in https://github.com/apache/spark/blob/master/docs/README.md --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16290 After a research, for avoiding this flaky testcase, the simplest way is to remove the contents in `metastore_db` (whose location is specified through `javax.jdo.option.ConnectionURL`). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70341/testReport)** for PR 16335 at commit [`7ad462a`](https://github.com/apache/spark/commit/7ad462ade573688dcd3ad963d65395689d9b9bcc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16335 Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=6AC0B3E0-95BF-4B24-AF41-47DE7360FAB9&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/6AC0B3E0-95BF-4B24-AF41-47DE7360FAB9) Build started: [TESTS] `org.apache.spark.sql.execution.datasources.json.JsonSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=5164A852-4B91-485F-ACF6-40CE627F2E83&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/5164A852-4B91-485F-ACF6-40CE627F2E83) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.state.StateStoreSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=2ED3B413-EF9F-4BF1-9CFE-BDC956D37E9A&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/2ED3B413-EF9F-4BF1-9CFE-BDC956D37E9A) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F9445F0A-49C1-45AF-9B14-C1328E5EC740&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F9445F0A-49C1-45AF-9B14-C1328E5EC740) Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=98335C35-A44F-4C47-B3ED-4A8F73525878&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/98335C35-A44F-4C47-B3ED-4A8F73525878) Diff: https://github.com/apache/spark/compare/master...spark-test:98335C35-A44F-4C47-B3ED-4A8F73525878 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92975296 --- Diff: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala --- @@ -35,6 +36,8 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers { tests.foreach { case (name, master) => test(s"$name: launcher handle") { + // The tests here are failed due to the cmd length limitation up to 8K on Windows. + assume(!Utils.isWindows) --- End diff -- This test is being failed due to cmd length limitation on Windows up to 8K. The actual command is roughly 16K (see https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16336 **[Test build #70340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70340/testReport)** for PR 16336 at commit [`2aa11d8`](https://github.com/apache/spark/commit/2aa11d831401969a1c6052ece749ec4d56beb117). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16336: [SPARK-API-DOC][DOC][BUILD] Support skipping R/Py...
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/16336 [SPARK-API-DOC][DOC][BUILD] Support skipping R/Python API docs ## What changes were proposed in this pull request? This PR aims to support `SKIP_PYTHONDOC` and `SKIP_RDOC` for documentation build. Currently, we can use `SKIP_SCALADOC` or `SKIP_API`. The reason providing additional options is that the Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Python and R. Especially, - Python API docs requires `sphinx`. - R API docs requires `R` installation and `knitr` (and more others libraries). In other words, we cannot generate Python API docs without R installation. Also, we cannot generate R API docs without Python `sphinx` installation. If Spark provides `SKIP_PYTHONDOC` and `SKIP_RDOC` like `SKIP_SCALADOC`, it would be more convenient. ## How was this patch tested? Manual. **Skipping Scala/Java/Python API Doc Build** ```bash $ cd docs $ SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 jekyll build $ ls api DESCRIPTION R ``` **Skipping Scala/Java/R API Doc Build** ```bash $ cd docs $ SKIP_SCALADOC=1 SKIP_RDOC=1 jekyll build $ ls api python ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark SPARK-18923 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16336.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16336 commit 2aa11d831401969a1c6052ece749ec4d56beb117 Author: Dongjoon Hyun Date: 2016-12-19T05:19:48Z [SPARK-API-DOC][DOC][BUILD] Support skipping R/Python API docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16313 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16313 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70331/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16313 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16313 **[Test build #70331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70331/testReport)** for PR 16313 at commit [`32857e6`](https://github.com/apache/spark/commit/32857e6c5fa89094b84d4ed78469217af8c515c7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70337/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #70337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)** for PR 16330 at commit [`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16308: [SPARK-18350][SQL][WIP] Support session local tim...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/16308#discussion_r92973560 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -111,7 +112,8 @@ case class CatalogTablePartition( */ def toRow(partitionSchema: StructType): InternalRow = { InternalRow.fromSeq(partitionSchema.map { field => - Cast(Literal(spec(field.name)), field.dataType).eval() + Cast(Literal(spec(field.name)), field.dataType, +DateTimeUtils.defaultTimeZone().getID).eval() --- End diff -- Currently the behavior doesn't change by timezone setting, i.e. using system timezone. This is a part that I was not sure which we should handle the partition values, use timezone settings or system timezone. Should we use timezone settings? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16323: [SPARK-18911] [SQL] Define CatalogStatistics to interact...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16323 **[Test build #70339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70339/testReport)** for PR 16323 at commit [`bd5eacc`](https://github.com/apache/spark/commit/bd5eaccabe6bdfcf3d7afbdcdb15676aaad78275). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16323: [SPARK-18911] [SQL] Define CatalogStatistics to interact...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16323 Ok, I think it's doable. But since it's not a small change, let's wait @rxin for his comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16334: estimate filter cardinality
Github user ron8hu commented on the issue: https://github.com/apache/spark/pull/16334 Sorry. This is a mistake. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16334: estimate filter cardinality
Github user ron8hu closed the pull request at: https://github.com/apache/spark/pull/16334 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/13909#discussion_r92972394 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) extends Expression { } override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { -val arrayClass = classOf[GenericArrayData].getName -val values = ctx.freshName("values") -ctx.addMutableState("Object[]", values, s"this.$values = null;") +val array = ctx.freshName("array") -ev.copy(code = s""" - this.$values = new Object[${children.size}];""" + +val et = dataType.elementType +val evals = children.map(e => e.genCode(ctx)) +val isPrimitiveArray = ctx.isPrimitiveType(et) +val primitiveTypeName = if (isPrimitiveArray) ctx.primitiveTypeName(et) else "" +val (preprocess, arrayData, arrayWriter) = + GenArrayData.getCodeArrayData(ctx, et, children.size, isPrimitiveArray, array) + +ev.copy(code = + preprocess + ctx.splitExpressions( ctx.INPUT_ROW, -children.zipWithIndex.map { case (e, i) => - val eval = e.genCode(ctx) - eval.code + s""" -if (${eval.isNull}) { - $values[$i] = null; +evals.zipWithIndex.map { case (eval, i) => + eval.code + +(if (isPrimitiveArray) { + (if (!children(i).nullable) { +s"\n$arrayWriter.write($i, ${eval.value});" + } else { +s""" +if (${eval.isNull}) { + $arrayWriter.setNull$primitiveTypeName($i); +} else { + $arrayWriter.write($i, ${eval.value}); +} + """ + }) } else { - $values[$i] = ${eval.value}; -} - """ + s""" + if (${eval.isNull}) { +$array[$i] = null; + } else { +$array[$i] = ${eval.value}; + } + """ +}) }) + - s""" -final ArrayData ${ev.value} = new $arrayClass($values); -this.$values = null; - """, isNull = "false") + s"\nfinal ArrayData ${ev.value} = $arrayData;\n", + isNull = "false") } override def prettyName: String = "array" } +private [sql] object GenArrayData { + // This function returns Java code pieces based on DataType and isPrimitive + // for allocation of ArrayData class + def getCodeArrayData( + ctx: CodegenContext, + dt: DataType, + size: Int, + isPrimitive : Boolean, + array: String): (String, String, String) = { +if (!isPrimitive) { + val arrayClass = classOf[GenericArrayData].getName + ctx.addMutableState("Object[]", array, +s"this.$array = new Object[${size}];") + ("", s"new $arrayClass($array)", null) +} else { + val holder = ctx.freshName("holder") + val arrayWriter = ctx.freshName("createArrayWriter") + val unsafeArrayClass = classOf[UnsafeArrayData].getName + val holderClass = classOf[BufferHolder].getName + val arrayWriterClass = classOf[UnsafeArrayWriter].getName + ctx.addMutableState(unsafeArrayClass, array, "") + ctx.addMutableState(holderClass, holder, "") + ctx.addMutableState(arrayWriterClass, arrayWriter, "") + val baseOffset = Platform.BYTE_ARRAY_OFFSET + val unsafeArraySizeInBytes = +UnsafeArrayData.calculateHeaderPortionInBytes(size) + +ByteArrayMethods.roundNumberOfBytesToNearestWord(dt.defaultSize * size) + + (s""" +$array = new $unsafeArrayClass(); +$holder = new $holderClass($unsafeArraySizeInBytes); +$arrayWriter = new $arrayWriterClass(); --- End diff -- @kiszk Do you remember which test failed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16308 **[Test build #70338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70338/testReport)** for PR 16308 at commit [`b860379`](https://github.com/apache/spark/commit/b8603790c3201c691415461a132b84ba882dc758). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #70337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)** for PR 16330 at commit [`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehous...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16290#discussion_r92971642 --- Diff: R/pkg/inst/tests/testthat/test_context.R --- @@ -72,6 +72,20 @@ test_that("repeatedly starting and stopping SparkSession", { } }) +test_that("Default warehouse dir should be set to tempdir", { + sparkR.session.stop() + sparkR.session(enableHiveSupport = FALSE) + + # Create a temporary table + sql("CREATE TABLE people_warehouse_test") + # spark-warehouse should be written only tempdir() and not current working directory + res <- list.files(path = ".", pattern = ".*spark-warehouse.*", --- End diff -- That I agree completely --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16330 that's weird, I'm seeing a lot of seemingly unrelated flaky test failure lately? ``` org.apache.spark.util.collection.ExternalSorterSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(ExternalSorterSuite.scala:32) java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.FutureTask.report(FutureTask.java:122) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16330 jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16290 I checked the most two recent failed test cases in Jenkins. They are not related to the changes in the PR. In the local environment, I can reproduce the error you mentioned above. ``` [info] 2016-12-18 20:47:23.328 - stdout> path1: file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse [info] 2016-12-18 20:47:23.328 - stdout> path2: file:/Users/xiaoli/IdeaProjects/sparkDelivery/spark-warehouse/ ``` The location of default database is still pointing to the original value of `hive.metastore.warehouse.dir` or `spark.sql.warehouse.dir` that was set in the previous test case or our previous local spark job. Ideally, our test suite should directly connect to Derby and drop the default database. Let me do more search. Also cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14365 **[Test build #70336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)** for PR 14365 at commit [`141cb1d`](https://github.com/apache/spark/commit/141cb1d5232a14d72f6411afdb1212cf7ef4de0a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16290 **[Test build #70335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70335/testReport)** for PR 16290 at commit [`f7b4772`](https://github.com/apache/spark/commit/f7b4772a4b72c28047afb7d614e3af3317af896d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16290 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13909 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70327/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13909 **[Test build #70327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70327/testReport)** for PR 13909 at commit [`d24c7b1`](https://github.com/apache/spark/commit/d24c7b1e13c00b722cafb7230e1e59021f42eee4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70332/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #70332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)** for PR 16330 at commit [`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehous...
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/16290#discussion_r92968560 --- Diff: R/pkg/inst/tests/testthat/test_context.R --- @@ -72,6 +72,20 @@ test_that("repeatedly starting and stopping SparkSession", { } }) +test_that("Default warehouse dir should be set to tempdir", { + sparkR.session.stop() + sparkR.session(enableHiveSupport = FALSE) + + # Create a temporary table + sql("CREATE TABLE people_warehouse_test") + # spark-warehouse should be written only tempdir() and not current working directory + res <- list.files(path = ".", pattern = ".*spark-warehouse.*", --- End diff -- I think my bigger concern for that is that usually tests are run all at time - i.e. core, sql, hive and then python, R. And there are no guarantees that other module tests won't create files inside `SPARK_HOME` afaik. So while we can check some basic things with our test, I dont think verifying a global property is always possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16232: [SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correct...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16232 ping @hvanhovell Do you think this fixing good now? Thanks. cc @davies @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16335 cc @srowen, do you mind if I ask to take a look please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16245: [SPARK-18824][SQL] Add optimizer rule to reorder Filter ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16245 ping @hvanhovell @rxin Any thing I need to address more? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16335 Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=83107543-90AA-4CEB-B33C-C6EDEBF5D7A7&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/83107543-90AA-4CEB-B33C-C6EDEBF5D7A7) Build started: [TESTS] `org.apache.spark.sql.execution.datasources.json.JsonSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=62FB21FD-889B-4A64-9BAF-D392C7997466&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/62FB21FD-889B-4A64-9BAF-D392C7997466) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.state.StateStoreSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=E0E0715A-4D36-430C-989F-7EBE7DBB00D1&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/E0E0715A-4D36-430C-989F-7EBE7DBB00D1) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F40EEC55-B520-4FD2-B49B-46E281C63887&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F40EEC55-B520-4FD2-B49B-46E281C63887) Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=ECB4A22F-AC5F-46B4-AE80-BCB22C4F6C4C&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/ECB4A22F-AC5F-46B4-AE80-BCB22C4F6C4C) Diff: https://github.com/apache/spark/compare/master...spark-test:F40EEC55-B520-4FD2-B49B-46E281C63887 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Consumes the returned local ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16263 ping @davies Can you take a look again? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92968297 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala --- @@ -209,14 +209,20 @@ class HDFSMetadataLogSuite extends SparkFunSuite with SharedSQLContext { } // Open and delete -val f1 = fm.open(path) -fm.delete(path) -assert(!fm.exists(path)) -intercept[IOException] { - fm.open(path) +if (Utils.isWindows) { + fm.open(path).close() + fm.delete(path) + fm.delete(path) // should not throw exception --- End diff -- Windows holds an exclusive lock so it does not allow to remove a file when it is open. So, I re-wrote the test cases for WIndows. cc @tdas, do you mind if I ask to see whether these tests on Windows are fine? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14365 ping @cloud-fan @yhuai May you take a look? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92968267 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala --- @@ -209,14 +209,20 @@ class HDFSMetadataLogSuite extends SparkFunSuite with SharedSQLContext { } // Open and delete -val f1 = fm.open(path) -fm.delete(path) -assert(!fm.exists(path)) -intercept[IOException] { - fm.open(path) +if (Utils.isWindows) { + fm.open(path).close() + fm.delete(path) + fm.delete(path) // should not throw exception +} else { + Utils.tryWithResource(fm.open(path)) { _ => --- End diff -- We should close when it emits an exception. Otherwise, it is possible to suppress the actual assertion error due to the exception in removing the path, that makes the debug harder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70334/testReport)** for PR 16335 at commit [`c5d6938`](https://github.com/apache/spark/commit/c5d69388823aa7fc9d75826cabacfb666ee665d5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16331 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70326/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16331 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16331 **[Test build #70326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70326/testReport)** for PR 16331 at commit [`1618d1d`](https://github.com/apache/spark/commit/1618d1d7855a6c2bdb01387330854b6e6159dd61). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16335 cc @srowen, do you mind if I ask to take a look please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92967906 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala --- @@ -209,14 +209,21 @@ class HDFSMetadataLogSuite extends SparkFunSuite with SharedSQLContext { } // Open and delete -val f1 = fm.open(path) -fm.delete(path) -assert(!fm.exists(path)) -intercept[IOException] { - fm.open(path) +// Open and delete +if (Utils.isWindows) { + Utils.tryWithResource(fm.open(path))(_ => ()) + fm.delete(path) + fm.delete(path) // should not throw exception --- End diff -- cc @tdas, do you mind if I ask to see whether these tests on Windows are fine? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92967880 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala --- @@ -209,14 +209,21 @@ class HDFSMetadataLogSuite extends SparkFunSuite with SharedSQLContext { } // Open and delete -val f1 = fm.open(path) -fm.delete(path) -assert(!fm.exists(path)) -intercept[IOException] { - fm.open(path) +// Open and delete +if (Utils.isWindows) { + Utils.tryWithResource(fm.open(path))(_ => ()) + fm.delete(path) + fm.delete(path) // should not throw exception +} else { + Utils.tryWithResource(fm.open(path)) { _ => --- End diff -- We should close when it emits an exception. Otherwise, it is possible to suppress the actual assertion error due to the exception in removing the path, that makes the debug harder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92967755 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala --- @@ -88,14 +88,14 @@ class HDFSMetadataLogSuite extends SparkFunSuite with SharedSQLContext { s"fs.$scheme.impl", classOf[FakeFileSystem].getName) withTempDir { temp => - val metadataLog = new HDFSMetadataLog[String](spark, s"$scheme://$temp") + val metadataLog = new HDFSMetadataLog[String](spark, s"$scheme://${temp.toURI.getPath}") --- End diff -- This is using a fake scheme. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16335#discussion_r92967723 --- Diff: core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala --- @@ -34,6 +35,8 @@ class LauncherBackendSuite extends SparkFunSuite with Matchers { "standalone/client" -> "local-cluster[1,1,1024]") tests.foreach { case (name, master) => +// The tests here are failed due to the cmd length limitation up to 8K on Windows. +assume(!Utils.isWindows) --- End diff -- This test is being failed due to cmd length limitation on Windows up to 8K. The actual command is roughly 16K (see https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16335 Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=BFE6F12C-21BF-414F-B6D2-2A01025A963A&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/BFE6F12C-21BF-414F-B6D2-2A01025A963A) Build started: [TESTS] `org.apache.spark.sql.execution.datasources.json.JsonSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F76F11AC-3219-4AEA-800C-5D54666043B0&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F76F11AC-3219-4AEA-800C-5D54666043B0) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.state.StateStoreSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=979C8BA8-CEB6-4056-B603-54168819FF35&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/979C8BA8-CEB6-4056-B603-54168819FF35) Build started: [TESTS] `org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=58D6D032-EE90-4060-9DAB-C2F5DBAF978C&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/58D6D032-EE90-4060-9DAB-C2F5DBAF978C) Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` [![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=5726D75A-8D35-4C0C-B571-8413F5E8E1FC&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/5726D75A-8D35-4C0C-B571-8413F5E8E1FC) Diff: https://github.com/apache/spark/compare/master...spark-test:F76F11AC-3219-4AEA-800C-5D54666043B0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16332 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70329/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16332 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16332 **[Test build #70329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70329/testReport)** for PR 16332 at commit [`b1b152b`](https://github.com/apache/spark/commit/b1b152b0cb94c1f6b0a41ee3f13750d96c3420a0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16335 **[Test build #70333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70333/testReport)** for PR 16335 at commit [`e309285`](https://github.com/apache/spark/commit/e3092856fa207d1fcc118580d77a64bc653c62be). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/16335 [SPARK-18922][TESTS] Fix more resource-closing-related and path-related test failures in identified ones on Windows ## What changes were proposed in this pull request? There are several tests failing due to resource-closing-related and path-related problems on Windows as below. - `SQLQuerySuite`: ``` - specifying database name for a temporary table is not allowed *** FAILED *** (125 milliseconds) org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark arget mpspark-1f4471ab-aac0-4239-ae35-833d54b37e52; at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370) ``` - `JsonSuite`: ``` - Loading a JSON dataset from a text file with SQL *** FAILED *** (94 milliseconds) org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark arget mpspark-c918a8b7-fc09-433c-b9d0-36c0f78ae918; at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370) ``` - `StateStoreSuite`: ``` - SPARK-18342: commit fails when rename fails *** FAILED *** (16 milliseconds) java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0 at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.(Path.java:116) at org.apache.hadoop.fs.Path.(Path.java:89) ... Cause: java.net.URISyntaxException: Relative path in absolute URI: StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0 at java.net.URI.checkPath(URI.java:1823) at java.net.URI.(URI.java:745) at org.apache.hadoop.fs.Path.initialize(Path.java:203) ``` - `HDFSMetadataLogSuite`: ``` - FileManager: FileContextManager *** FAILED *** (94 milliseconds) java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-415bb0bd-396b-444d-be82-04599e025f21 at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010) at org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127) at org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38) - FileManager: FileSystemManager *** FAILED *** (78 milliseconds) java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-ef8222cd-85aa-47c0-a396-bc7979e15088 at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010) at org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127) at org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38) ``` And, there are some tests being failed due to the length limitation on cmd in Windows as below: - `LauncherBackendSuite`: ``` - local: launcher handle *** FAILED *** (30 seconds, 120 milliseconds) The code passed to eventually never returned normally. Attempted 283 times over 30.0960053 seconds. Last failure message: The reference was null. (LauncherBackendSuite.scala:56) org.scalatest.exceptions.TestFailedDueToTimeoutException: at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438) - standalone/client: launcher handle *** FAILED *** (30 seconds, 47 milliseconds) The code passed to eventually never returned normally. Attempted 282 times over 30.03798710002 seconds. Last failure message: The reference was null. (LauncherBackendSuite.scala:56) org.scalatest.exceptions.TestFailedDueToTimeoutException: at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438) ``` The executed command is, https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd, which is 16K length; however, the length limitation is 8K. So, it is being failed to launch. This PR proposes to fix the test failures on Windows and skip the tests failed due to the length limitation ## How was this patch tested? Manually tested via AppVeyor **Before** `SQLQuerySuite `: https://c
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #70332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)** for PR 16330 at commit [`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16330 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16290 @shivaram with my PR #16330, `metastore_db` is in `tempdir` and is removed when the R process exits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16313 **[Test build #70331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70331/testReport)** for PR 16313 at commit [`32857e6`](https://github.com/apache/spark/commit/32857e6c5fa89094b84d4ed78469217af8c515c7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16142: [SPARK-18716][CORE] Restrict the disk usage of sp...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/16142 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/16142 @vanzin I will close this PR, but can't take your point completely. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16334: estimate filter cardinality
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16334 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16334: estimate filter cardinality
GitHub user ron8hu opened a pull request: https://github.com/apache/spark/pull/16334 estimate filter cardinality ## What changes were proposed in this pull request? This is a WIP PR. In this version, we set up the framework to traverse predicate and evaluate the equality (=) expression. ## How was this patch tested? We just have a simple test case for now. More tests need to be added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ron8hu/spark filterEstimate Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16334.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16334 commit f0bbb43f7b27c59a3181ce428ed2bb0a7c1fc89d Author: wangzhenhua Date: 2016-12-09T06:34:41Z estimation for inner join commit 64603b589bd1b971d2370d848f1dd19f11b52928 Author: wangzhenhua Date: 2016-12-09T07:30:33Z wrong position of comments commit 02bdf875babac7676a732a01fa41c8b8416a58d3 Author: Ron Hu Date: 2016-12-18T02:10:28Z implemented first version of filter cardinality estimation commit 6c9d06085f30dd19eb8b0a06f62742bda6ace736 Author: Ron Hu Date: 2016-12-18T02:15:57Z maintained a mutable colStats in Filter class commit 9e92b4b4c5033441f63e0e76544a1d609ac2a39e Author: Ron Hu Date: 2016-12-19T00:57:00Z Used mutableColStats to reflect changes for column stats commit 8e77acc8630df3696dbb12453ee3f469c03b1193 Author: Ron Hu Date: 2016-12-19T02:15:41Z use getOrElse to return filter stats commit 62315985cc6ec20fb26c8ed109b33160e9f3f80b Author: Ron Hu Date: 2016-12-19T03:27:40Z added the test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16291 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70325/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16291 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16291 **[Test build #70325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70325/testReport)** for PR 16291 at commit [`6763827`](https://github.com/apache/spark/commit/676382790b2e73c6ad3b4abe7c565ad266d9df1a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class StreamingQueryListenerBus(val sparkListenerBus: LiveListenerBus)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16325: [SPARK-18703] [SPARK-18675] [SQL] [BACKPORT-2.1] CTAS fo...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16325 LGTM, merging to 2.1! Let's send a new PR for 2.0 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16333: Filter estimate
Github user ron8hu commented on the issue: https://github.com/apache/spark/pull/16333 This is a mistake. I point to the wrong repository. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org