[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21527 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21475 **[Test build #91667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91667/testReport)** for PR 21475 at commit [`a103673`](https://github.com/apache/spark/commit/a103673073e3114b9a28cf183a8d7ec9271769e3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21475 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21475 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91667/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21530: [spark][pyspark][sql] Correcting code for Documentation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21530 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21530: [spark][pyspark][sql] Correcting code for Documentation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21530 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21530: [spark][pyspark][sql] Correcting code for Documentation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21530 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21530: [spark][pyspark][sql] Correcting code for Documen...
GitHub user DaniHoon opened a pull request: https://github.com/apache/spark/pull/21530 [spark][pyspark][sql] Correcting code for Documentation Strings(PEP8) ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) Just correcting an unnecessary interval on [column.py] line 133. It does not fit Documentation Strings(PEP8). ## How was this patch tested? Change is not affecting any other code. Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DaniHoon/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21530.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21530 commit 5682a10bbf023de8e237ee80775cb0efc05794fe Author: Danihoon Date: 2018-06-11T18:30:59Z [spark][pyspark][sql] Correcting code for Documentation Strings(PEP8) [column.py] There is an unnecessary interval on line 133. It does not fit PEP8. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21439: [SPARK-24391][SQL] Support arrays of any types by from_j...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21439 **[Test build #91668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91668/testReport)** for PR 21439 at commit [`e321e37`](https://github.com/apache/spark/commit/e321e37b0aaf2d17ab267b685ea14ee778f06192). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21475 **[Test build #91667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91667/testReport)** for PR 21475 at commit [`a103673`](https://github.com/apache/spark/commit/a103673073e3114b9a28cf183a8d7ec9271769e3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...
Github user redsanket commented on the issue: https://github.com/apache/spark/pull/21475 @tgravescs @squito fixed the description hope you can take a look thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3757/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91663/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting volumes
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21260 **[Test build #91663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91663/testReport)** for PR 21260 at commit [`d960e34`](https://github.com/apache/spark/commit/d960e34e6309f1ef98d5437acdf9af097f5cf4f9). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3904/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3757/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3903/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerialize...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21518#discussion_r194497168
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala
---
@@ -45,6 +44,14 @@ class ClosableByteArrayInputStream(buf: Array[Byte])
extends ByteArrayInputStrea
class UnsafeRowSerializerSuite extends SparkFunSuite with
LocalSparkContext {
+ override def beforeAll() {
+super.beforeAll()
+// This test suite calls `UnsafeProjection.create` which accesses
`SQLConf.get`, we should make
+// sure active session is cleaned so that `SQLConf.get` won't refer to
a stopped session.
+SparkSession.clearActiveSession()
+SparkSession.clearDefaultSession()
--- End diff --
the problem is that, `LocalSparkContext` is in core module not SQL, so we
can't access `SparkSession` there.
Maybe we should use `LocalSparkSession` for this test suite.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21529: [SPARK-24495][SQL] EnsureRequirement returns wrong plan ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21529 **[Test build #91666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91666/testReport)** for PR 21529 at commit [`06858cd`](https://github.com/apache/spark/commit/06858cd5a7e1b11cd5fd1edb206296d75550a8d4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3756/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21529: [SPARK-24495][SQL] EnsureRequirement returns wron...
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21529 [SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys ## What changes were proposed in this pull request? `EnsureRequirement` in its `reorder` method currently assumes that the same key appears only once in the join condition. This of course might not be the case, and when it is not satisfied, it returns a wrong plan which produces a wrong result of the query. ## How was this patch tested? added UT You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/spark SPARK-24495 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21529.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21529 commit 06858cd5a7e1b11cd5fd1edb206296d75550a8d4 Author: Marco Gaido Date: 2018-06-11T17:59:19Z [SPARK-24495][SQL] EnsureRequirement returns worng plan when reordering equal keys --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21510: [SPARK-24490][WebUI] Use WebUI.addStaticHandler in web U...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21510 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91662/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3756/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21510: [SPARK-24490][WebUI] Use WebUI.addStaticHandler in web U...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21510 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3902/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21510: [SPARK-24490][WebUI] Use WebUI.addStaticHandler in web U...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21510 **[Test build #91662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91662/testReport)** for PR 21510 at commit [`3ab14a1`](https://github.com/apache/spark/commit/3ab14a1eb2f8c9d5a50a031425fccb5c17bdd4b6). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21439: [SPARK-24391][SQL] Support arrays of any types by...
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21439#discussion_r194491865
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -101,6 +102,13 @@ class JacksonParser(
}
}
+ private def makeArrayRootConverter(at: ArrayType): JsonParser =>
Seq[InternalRow] = {
+val elemConverter = makeConverter(at.elementType)
+(parser: JsonParser) => parseJsonToken[Seq[InternalRow]](parser, at) {
+ case START_ARRAY => Seq(InternalRow(convertArray(parser,
elemConverter)))
--- End diff --
The code in line 87 returns `null` for json input `[]` if schema is
`StructType(StructField("a", IntegerType) :: Nil)`. I would explain why we
should return `null` in that case: we *extract* struct from the array. If the
array is _empty_, it means there is nothing to extract and we returns `null`
for the nothing.
In case when schema is `ArrayType(...)`, I believe we should return
`empty` array for empty JSON array `[]`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21523: [SPARK-24506][UI] Add UI filters also to thriftse...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21523#discussion_r194490802
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -91,7 +91,12 @@ private[spark] abstract class WebUI(
/** Attach a handler to this UI. */
def attachHandler(handler: ServletContextHandler) {
handlers += handler
-serverInfo.foreach(_.addHandler(handler))
+serverInfo.foreach { sInfo =>
--- End diff --
I have a slight preference for doing this in the `ServerInfo` class, so
that we're sure we're covering all paths.
I have a WIP patch for another thing where I did that, something like this:
```
@@ -507,17 +517,19 @@ private[spark] case class ServerInfo(
server: Server,
boundPort: Int,
securePort: Option[Int],
+private val conf: SparkConf,
private val rootHandler: ContextHandlerCollection) {
- def addHandler(handler: ContextHandler): Unit = {
+ def addHandler(handler: ServletContextHandler): Unit = {
handler.setVirtualHosts(JettyUtils.toVirtualHosts(JettyUtils.SPARK_CONNECTOR_NAME))
+JettyUtils.addFilters(Seq(handler), conf)
rootHandler.addHandler(handler)
if (!handler.isStarted()) {
handler.start()
}
}
```
It also avoids a race where an already started handler would be added
before filters are installed. Extremely unlikely that would be a problem, but
well.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91661/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21528: [SPARK-24520] Double braces in documentations
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21528 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21045 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21528: [SPARK-24520] Double braces in documentations
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21528 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21045 **[Test build #91661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91661/testReport)** for PR 21045 at commit [`d8f3dea`](https://github.com/apache/spark/commit/d8f3dea8b227a4ee44dedb6b8199c8a17f6bfdd4). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21528: [SPARK-24520] Double braces in documentations
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21528 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21523: [SPARK-24506][UI] Add UI filters also to thriftse...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21523#discussion_r194488465
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -91,7 +91,12 @@ private[spark] abstract class WebUI(
/** Attach a handler to this UI. */
def attachHandler(handler: ServletContextHandler) {
handlers += handler
-serverInfo.foreach(_.addHandler(handler))
+serverInfo.foreach { sInfo =>
+ sInfo.addHandler(handler)
+ // If the UI has already been bound, we need to add the filters to
the newly attached
+ // handlers. Otherwise, they will be attached when binding.
+ addFilters(Seq(handler), conf)
--- End diff --
Nevermind you already removed it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21523: [SPARK-24506][UI] Add UI filters also to thriftse...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21523#discussion_r194488116
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -91,7 +91,12 @@ private[spark] abstract class WebUI(
/** Attach a handler to this UI. */
def attachHandler(handler: ServletContextHandler) {
handlers += handler
-serverInfo.foreach(_.addHandler(handler))
+serverInfo.foreach { sInfo =>
+ sInfo.addHandler(handler)
+ // If the UI has already been bound, we need to add the filters to
the newly attached
+ // handlers. Otherwise, they will be attached when binding.
+ addFilters(Seq(handler), conf)
--- End diff --
There's a call to `addFilters` in `HistoryServer.scala` that becomes
redundant with this code, I think.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21528: [SPARK-24520] Double braces in documentations
GitHub user Fokko opened a pull request: https://github.com/apache/spark/pull/21528 [SPARK-24520] Double braces in documentations There are double braces in the markdown, which break the link. ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Fokko/spark patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21528 commit 4db8ff0b4c2ec1c4fad21ac21d49f7af1642f23c Author: Fokko Driesprong Date: 2018-06-11T17:44:21Z [SPARK-24520] Double braces in documentations There are double braces in the markdown, which break the link --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21510: [SPARK-24490][WebUI] Use WebUI.addStaticHandler i...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21510#discussion_r194484175
--- Diff: core/src/main/scala/org/apache/spark/ui/WebUI.scala ---
@@ -88,41 +90,41 @@ private[spark] abstract class WebUI(
handlers += renderHandler
}
- /** Attach a handler to this UI. */
+ /** Attaches a handler to this UI. */
def attachHandler(handler: ServletContextHandler) {
handlers += handler
serverInfo.foreach(_.addHandler(handler))
}
- /** Detach a handler from this UI. */
+ /** Detaches a handler from this UI. */
def detachHandler(handler: ServletContextHandler) {
handlers -= handler
serverInfo.foreach(_.removeHandler(handler))
}
/**
- * Add a handler for static content.
+ * Adds a handler for static content.
*
* @param resourceBase Root of where to find resources to serve.
* @param path Path in UI where to mount the resources.
*/
- def addStaticHandler(resourceBase: String, path: String): Unit = {
+ def addStaticHandler(resourceBase: String, path: String = "/static"):
Unit = {
attachHandler(JettyUtils.createStaticHandler(resourceBase, path))
}
/**
- * Remove a static content handler.
+ * Removes a static content handler.
*
* @param path Path in UI to unmount.
*/
def removeStaticHandler(path: String): Unit = {
--- End diff --
This method name is misleading. It will remove any handler, not just
"static" handlers... and since you're touching this code...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21518: [SPARK-24502][SQL] flaky test: UnsafeRowSerialize...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/21518#discussion_r194483889
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala
---
@@ -45,6 +44,14 @@ class ClosableByteArrayInputStream(buf: Array[Byte])
extends ByteArrayInputStrea
class UnsafeRowSerializerSuite extends SparkFunSuite with
LocalSparkContext {
+ override def beforeAll() {
+super.beforeAll()
+// This test suite calls `UnsafeProjection.create` which accesses
`SQLConf.get`, we should make
+// sure active session is cleaned so that `SQLConf.get` won't refer to
a stopped session.
+SparkSession.clearActiveSession()
+SparkSession.clearDefaultSession()
--- End diff --
should we actually do this in `LocalSparkContext.resetSparkContext()`
(which is called in `afterEach`)?
I worry there are a lot more places we should be doing this, eg.
`MLlibTestSparkContext.afterAll()` only calls
`SparkSession.clearActiveSession()`.
this class does seem to be one of the major sources of flakiness so I'm
also OK with just getting this in for now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21469: [SPARK-24441][SS] Expose total estimated size of ...
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r194483603
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala
---
@@ -112,14 +122,19 @@ trait StateStoreWriter extends StatefulOperator {
self: SparkPlan =>
val storeMetrics = store.metrics
longMetric("numTotalStateRows") += storeMetrics.numKeys
longMetric("stateMemory") += storeMetrics.memoryUsedBytes
-storeMetrics.customMetrics.foreach { case (metric, value) =>
- longMetric(metric.name) += value
+storeMetrics.customMetrics.foreach {
+ case (metric: StateStoreCustomAverageMetric, value) =>
+longMetric(metric.name).set(value * 1.0d)
--- End diff --
How does this get accumulated ? It seems the value last set may get
propagated to the driver.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3755/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21221 well, I think you should change the way PeakExecutorMetrics gets converted to json, so that it uses a name from the relevant `MetricGetter`. You should be able to customize the way it gets converted to json here: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala#L50 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21469: [SPARK-24441][SS] Expose total estimated size of ...
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r194480087
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -247,6 +253,14 @@ private[state] class HDFSBackedStateStoreProvider
extends StateStoreProvider wit
private lazy val fm = CheckpointFileManager.create(baseDir, hadoopConf)
private lazy val sparkConf =
Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+ private lazy val metricProviderLoaderMapSizeBytes:
StateStoreCustomSizeMetric =
+StateStoreCustomSizeMetric("providerLoadedMapSizeBytes",
+ "estimated size of states cache in provider")
+
+ private lazy val metricProviderLoaderCountOfVersionsInMap:
StateStoreCustomAverageMetric =
--- End diff --
Why is "metricProviderLoaderCountOfVersionsInMap" an average metrics? The
other metrics like "numTotalStateRows" and even "providerLoadedMapSizeBytes" is
count metric. Shouldn't this be similar?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3754/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3755/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3901/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20260 The `TODO` in comment should be removed, as I explained. We should close this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3754/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3900/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21523 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21523 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91658/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21523 **[Test build #91658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91658/testReport)** for PR 21523 at commit [`0ac1848`](https://github.com/apache/spark/commit/0ac1848efcb5d95ea3b3ab29aca8ff14dcc17dd2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3753/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 another comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3752/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3753/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3899/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21523 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91659/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21523 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21523: [SPARK-24506][UI] Add UI filters also to thriftserver ta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21523 **[Test build #91659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91659/testReport)** for PR 21523 at commit [`08cd576`](https://github.com/apache/spark/commit/08cd576f71b9ed4e7f6af62c2d90b3a650f6d82c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3752/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21260: [SPARK-23529][K8s] Support mounting volumes
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21260#discussion_r194464996
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtils.scala
---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s
+
+import java.util.NoSuchElementException
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.k8s.Config._
+
+private[spark] object KubernetesVolumeUtils {
+ /**
+ * Extract Spark volume configuration properties with a given name
prefix.
+ *
+ * @param sparkConf Spark configuration
+ * @param prefix the given property name prefix
+ * @return a Map storing with volume name as key and spec as value
+ */
+ def parseVolumesWithPrefix(
+sparkConf: SparkConf,
+prefix: String): Iterable[Try[KubernetesVolumeSpec[_ <:
KubernetesVolumeSpecificConf]]] = {
+val properties = sparkConf.getAllWithPrefix(prefix).toMap
+
+getVolumeTypesAndNames(properties).map { case (volumeType, volumeName)
=>
+ val pathKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_MOUNT_PATH_KEY"
+ val readOnlyKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_MOUNT_READONLY_KEY"
+
+ for {
+path <- properties.getTry(pathKey)
+readOnly <- properties.getTry(readOnlyKey)
+volumeConf <- parseVolumeSpecificConf(properties, volumeType,
volumeName)
+ } yield KubernetesVolumeSpec(
+volumeName = volumeName,
+mountPath = path,
+mountReadOnly = readOnly.toBoolean,
+volumeConf = volumeConf
+ )
+}
+ }
+
+ /**
+ * Get unique pairs of volumeType and volumeName,
+ * assuming options are formatted in this way:
+ * `volumeType`.`volumeName`.`property` = `value`
+ * @param properties flat mapping of property names to values
+ * @return Set[(volumeType, volumeName)]
+ */
+ private def getVolumeTypesAndNames(
+properties: Map[String, String]
+ ): Set[(String, String)] = {
+properties.keys.flatMap { k =>
+ k.split('.').toList match {
+case tpe :: name :: _ => Some((tpe, name))
+case _ => None
+ }
+}.toSet
+ }
+
+ private def parseVolumeSpecificConf(
+options: Map[String, String],
+volumeType: String,
+volumeName: String): Try[KubernetesVolumeSpecificConf] = {
+volumeType match {
+ case KUBERNETES_VOLUMES_HOSTPATH_TYPE =>
+val pathKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_OPTIONS_PATH_KEY"
+for {
+ path <- options.getTry(pathKey)
+} yield KubernetesHostPathVolumeConf(path)
+
+ case KUBERNETES_VOLUMES_PVC_TYPE =>
+val claimNameKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY"
+for {
+ claimName <- options.getTry(claimNameKey)
+} yield KubernetesPVCVolumeConf(claimName)
+
+ case KUBERNETES_VOLUMES_EMPTYDIR_TYPE =>
+val mediumKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY"
+val sizeLimitKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY"
+for {
+ medium <- options.getTry(mediumKey)
+ sizeLimit <- options.getTry(sizeLimitKey)
--- End diff --
Both `Medium` and `SizeLimit` are optional for `EmptyDirVolumeSource` so
users are not required to set them. What happens with `options.getTry` if users
do not set the key?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21260: [SPARK-23529][K8s] Support mounting volumes
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21260#discussion_r194467874
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtils.scala
---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s
+
+import java.util.NoSuchElementException
+
+import scala.util.{Failure, Success, Try}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.k8s.Config._
+
+private[spark] object KubernetesVolumeUtils {
+ /**
+ * Extract Spark volume configuration properties with a given name
prefix.
+ *
+ * @param sparkConf Spark configuration
+ * @param prefix the given property name prefix
+ * @return a Map storing with volume name as key and spec as value
+ */
+ def parseVolumesWithPrefix(
+sparkConf: SparkConf,
+prefix: String): Iterable[Try[KubernetesVolumeSpec[_ <:
KubernetesVolumeSpecificConf]]] = {
+val properties = sparkConf.getAllWithPrefix(prefix).toMap
+
+getVolumeTypesAndNames(properties).map { case (volumeType, volumeName)
=>
+ val pathKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_MOUNT_PATH_KEY"
+ val readOnlyKey =
s"$volumeType.$volumeName.$KUBERNETES_VOLUMES_MOUNT_READONLY_KEY"
+
+ for {
+path <- properties.getTry(pathKey)
+readOnly <- properties.getTry(readOnlyKey)
--- End diff --
`ReadOnly` is optional and defaults to false so users are not required to
set it explicitly. Is this semantics captured by `properties.getTry`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21526: [SPARK-24515][CORE] No need to warning when output commi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21526 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91656/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3898/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21260: [SPARK-23529][K8s] Support mounting volumes
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/21260#discussion_r194462215 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala --- @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.deploy.k8s.features + +import io.fabric8.kubernetes.api.model._ + +import org.apache.spark.deploy.k8s._ + +private[spark] class MountVolumesFeatureStep( + kubernetesConf: KubernetesConf[_ <: KubernetesRoleSpecificConf]) --- End diff -- The class parameter line should be indented 4 spaces. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21526: [SPARK-24515][CORE] No need to warning when output commi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21526 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21526: [SPARK-24515][CORE] No need to warning when output commi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21526 **[Test build #91656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91656/testReport)** for PR 21526 at commit [`6bac153`](https://github.com/apache/spark/commit/6bac1531929e914764d980e4eb4228a10436876b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...
Github user bkrieger commented on a diff in the pull request:
https://github.com/apache/spark/pull/21508#discussion_r194467198
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1568,11 +1568,32 @@ class Analyzer(
expr.find(_.isInstanceOf[Generator]).isDefined
}
-private def hasNestedGenerator(expr: NamedExpression): Boolean = expr
match {
- case UnresolvedAlias(_: Generator, _) => false
- case Alias(_: Generator, _) => false
- case MultiAlias(_: Generator, _) => false
- case other => hasGenerator(other)
+private def hasNestedGenerator(expr: NamedExpression): Boolean = {
+ trimNonTopLevelAliases(expr) match {
+case UnresolvedAlias(_: Generator, _) => false
+case Alias(_: Generator, _) => false
+case MultiAlias(_: Generator, _) => false
+case other => hasGenerator(other)
+ }
+}
+
+def trimNonTopLevelAliases(e: Expression): Expression = e match {
+ case a: UnresolvedAlias =>
--- End diff --
I don't think it'll hurt to handle it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3751/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3751/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3897/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user ssuchter commented on the issue: https://github.com/apache/spark/pull/21517 comment --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
