[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> Thank you for your first contribution, @suryag10 .
>
> * Could you file a SPARK JIRA issue since this is a code change?
Sure.
> * Could you update the PR title like the other PRs? e.g.
`[SPARK-XXX][SQL][K8S] ...`?
Sure.
>
> And, just out of curious, do we need this change?
>
> ```shell
> - exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name
"Thrift JDBC/ODBC Server" "$@"
> + exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name
"Thrift-JDBC-ODBC-Server" "$@"
> ```
Without the above change, it fails to start the driver pod as well. Spaces,
"/" are not allowed for the "name" in the kubernetes world.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22433
Thank you for your first contribution, @suryag10 .
- Could you file a SPARK JIRA issue since this is a code change?
- Could you update the PR title like the other PRs? e.g.
`[SPARK-XXX][SQL][K8S] ...`?
And, just out of curious, do we need this change?
```bash
- exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name "Thrift
JDBC/ODBC Server" "$@"
+ exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name
"Thrift-JDBC-ODBC-Server" "$@"
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3135/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22433 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3135/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22433 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky Ext...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217902367
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite extends SparkFunSuite
//
https://github.com/scala/scala/blob/2.13.x/test/junit/scala/tools/testing/AssertUtil.scala
// (lines 69-89)
// assert(map.currentMap == null)
-eventually {
+eventually(timeout(5 seconds), interval(200 milliseconds)) {
--- End diff --
This is a tiny issue annoying us. It's not a mess. :) Thank you again for
your quick response.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky Ext...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217902327
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite extends SparkFunSuite
//
https://github.com/scala/scala/blob/2.13.x/test/junit/scala/tools/testing/AssertUtil.scala
// (lines 69-89)
// assert(map.currentMap == null)
-eventually {
+eventually(timeout(5 seconds), interval(200 milliseconds)) {
--- End diff --
Thank you for review, @eyalfa .
If you are looking at Spark code, you must notice that the most test cases
are designed to have their suitable timeout and interval. We don't use a long
global value for all test cases. That will hide potential big issues.
Also, I already knew that the other instance you mentioned, and 4 instance
more on `ProcessingTimeExecutorSuite`. However, I didn't want to change it
because I usually don't touch the test cases if it's not flaky.
Specifically, this instance is really serious in our Jenkins environment.
The above 7 failures are only recent samples from 2 test series. We have
additional 3 test series on branch `master`; maven-hadoop2.6, maven-hadoop2.7,
sbt-scala-2.12. For `branch-2.4`, we have 4 more.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22433 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3135/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky Ext...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217902092
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite extends SparkFunSuite
//
https://github.com/scala/scala/blob/2.13.x/test/junit/scala/tools/testing/AssertUtil.scala
// (lines 69-89)
// assert(map.currentMap == null)
-eventually {
+eventually(timeout(5 seconds), interval(200 milliseconds)) {
--- End diff --
@dongjoon-hyun, thanks for cleaning up my mess! :sunglasses:
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21515 Problem solved. This script assumes the current user can run docker command without `sudo`, while it may not be true. But we can't run this script with root user as we can't add a root user in the docker container. Following the steps in https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user solves it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22343 Could you close this PR and JIRA, @seancxmao ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22060 Gentle ping, @maryannxue . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21517: Testing k8s change - please ignore (13)
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21517 Gentle ping, @ssuchter . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20405: [SPARK-23229][SQL] Dataset.hint should use planWithBarri...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20405 Gentle ping, @jaceklaskowski . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18304: [SPARK-21098] Set lineseparator csv multiline and csv wr...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18304 Gentle ping. If the issue is resolved, please close this PR, @cse68197 . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky Ext...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/22432#discussion_r217901988
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -457,7 +458,7 @@ class ExternalAppendOnlyMapSuite extends SparkFunSuite
//
https://github.com/scala/scala/blob/2.13.x/test/junit/scala/tools/testing/AssertUtil.scala
// (lines 69-89)
// assert(map.currentMap == null)
-eventually {
+eventually(timeout(5 seconds), interval(200 milliseconds)) {
--- End diff --
I think best practice to encounter this is specifying config patience in
sbt's test options.
having that said, I'v once had to 'grant' a longer timeout to a specific
class so I've achieved this by overriding the `spanScalaeFactor` method
`override def spanScaleFactor: Double = super.spanScaleFactor * 3`
please notice that there's another usage of `eventually` in line 519, this
one 'manually' waits 500 millis before testing which might explain why you
didn't see it failing in CI, looking at it now it seems like a bad practice
since `eventually` is designed to control both the timeout and the intervals
between trying.
to summarize: best practice is to control this in sbt's test settings, if
needed you can further control it in a specific class, in any case you have to
make sure you handle all invocations of `eventually` (which is easier and less
error prone by leveraging scalaTest's mechanisms).
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22433 **[Test build #96105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96105/testReport)** for PR 22433 at commit [`3a7fa57`](https://github.com/apache/spark/commit/3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21515#discussion_r217901993
--- Diff: dev/create-release/do-release-docker.sh ---
@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Creates a Spark release candidate. The script will update versions, tag
the branch,
+# build Spark binary packages and documentation, and upload maven
artifacts to a staging
+# repository. There is also a dry run mode where only local builds are
performed, and
+# nothing is uploaded to the ASF repos.
+#
+# Run with "-h" for options.
+#
+
+set -e
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/release-util.sh"
+
+function usage {
+ local NAME=$(basename $0)
+ cat < "$GPG_KEY_FILE"
+
+run_silent "Building spark-rm image with tag $IMGTAG..."
"docker-build.log" \
+ docker build -t "spark-rm:$IMGTAG" --build-arg UID=$UID "$SELF/spark-rm"
--- End diff --
got it. This is a system variable. So we can't run this script with root
user...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22433 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21515#discussion_r217901953
--- Diff: dev/create-release/do-release-docker.sh ---
@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Creates a Spark release candidate. The script will update versions, tag
the branch,
+# build Spark binary packages and documentation, and upload maven
artifacts to a staging
+# repository. There is also a dry run mode where only local builds are
performed, and
+# nothing is uploaded to the ASF repos.
+#
+# Run with "-h" for options.
+#
+
+set -e
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/release-util.sh"
+
+function usage {
+ local NAME=$(basename $0)
+ cat < "$GPG_KEY_FILE"
+
+run_silent "Building spark-rm image with tag $IMGTAG..."
"docker-build.log" \
+ docker build -t "spark-rm:$IMGTAG" --build-arg UID=$UID "$SELF/spark-rm"
--- End diff --
So we need to do `export UID=xxx` before running this script?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22433 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployment modes with spark de...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22433 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s cluster mode
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22433 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: Support STS to run in k8s cluster mode
GitHub user suryag10 opened a pull request: https://github.com/apache/spark/pull/22433 Support STS to run in k8s cluster mode ## What changes were proposed in this pull request? Code is enhanced to allow the STS run in kubernetes deployment with spark deploy mode of cluster. ## How was this patch tested? Started the sts in cluster mode in K8S deployment and was able to run some queries using the beeline client. You can merge this pull request into a Git repository by running: $ git pull https://github.com/suryag10/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22433 commit 3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce Author: Suryanarayana GARLAPATI Date: 2018-09-16T04:37:26Z Support STS to run in k8s cluster mode --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22393: [MINOR][DOCS] Axe deprecated doc refs
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22393 yes please - please double the doc created looks correct - there is no automatic test for that --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21515 UID already exists? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96101/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96101/testReport)** for PR 22355 at commit [`b1c6e86`](https://github.com/apache/spark/commit/b1c6e86e06c8d1e80af5d8bc6a3f1a6e08fa0026). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `class InterpretedMutableProjection(expressions: Seq[Expression]) extends MutableProjection ` * `abstract class MutableProjection extends Projection ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22418: [SPARK-25427][SQL][TEST] Add BloomFilter creation test c...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22418 Also, cc @dbtsai . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22431: [SPARK-24418][FOLLOWUP][DOC] Update docs to show ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22431 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/7#discussion_r217901635
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1803,6 +1803,10 @@ test_that("string operators", {
collect(select(df4, split_string(df4$a, "")))[1, 1],
list(list("a.b@c.d 1", "b"))
)
+ expect_equal(
+collect(select(df4, split_string(df4$a, "\\.", 2)))[1, 1],
+list(list("a", "b@c.d 1\\b"))
--- End diff --
let's add a test for `limit = 0` or `limit = -1` too - while it's the
default value, is any of the test cases changes behavior for limit = -1?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22431: [SPARK-24418][FOLLOWUP][DOC] Update docs to show Scala 2...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22431 Thank you so much, @dbtsai ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/22379#discussion_r217901558
--- Diff: R/pkg/NAMESPACE ---
@@ -275,6 +275,7 @@ exportMethods("%<=>%",
"format_number",
"format_string",
"from_json",
+ "from_csv",
--- End diff --
pleas sort this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/22379#discussion_r217901588
--- Diff: R/pkg/R/functions.R ---
@@ -2202,6 +2208,24 @@ setMethod("from_json", signature(x = "Column",
schema = "characterOrstructType")
column(jc)
})
+#' @details
+#' \code{from_csv}: Parses a column containing a CSV string into a Column
of \code{structType}
+#' with the specified \code{schema}.
+#' If the string is unparseable, the Column will contain the value NA.
+#'
+#' @rdname column_collection_functions
+#' @aliases from_csv from_csv,Column,character-method
+#'
--- End diff --
newline with `#'` is significant in ROxygen, please remove this line
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22431: [SPARK-24418][FOLLOWUP][DOC] Update docs to show Scala 2...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/22431 Thanks! Merged into both branch 2.4 and master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22393: [MINOR][DOCS] Axe deprecated doc refs
Github user MichaelChirico commented on the issue: https://github.com/apache/spark/pull/22393 @felixcheung is that to me? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21515: [SPARK-24372][build] Add scripts to help with preparing ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21515 I hit an issue when using this script to prepare 2.4.0 rc1 ``` useradd: UID 0 is not unique The command '/bin/sh -c useradd -m -s /bin/bash -p spark-rm -u $UID spark-rm' returned a non-zero code: 4 ``` Any ideas about how to resolve it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22393: [MINOR][DOCS] Axe deprecated doc refs
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22393 could you check the doc output manually for registerTempTable and createTable? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22430 I got it~ Oh, 2.3.2-rc6 is made. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/22010#discussion_r217901185
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -396,7 +396,26 @@ abstract class RDD[T: ClassTag](
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null):
RDD[T] = withScope {
-map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)
+partitioner match {
+ case Some(p) if numPartitions == partitions.length =>
+def key(x: T): (T, Null) = (x, null)
+val cleanKey = sc.clean(key _)
+val keyed = new MapPartitionsRDD[(T, Null), T](
+ this,
+ (context, pid, iter) => iter.map(cleanKey),
+ knownPartitioner = Some(new WrappedPartitioner(p)))
+val duplicatesRemoved = keyed.reduceByKey((x, y) => x)
--- End diff --
Ah yes, no partitioner specified => use parent's partitioner.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/22010#discussion_r217901179
--- Diff: core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
---
@@ -35,16 +35,24 @@ import org.apache.spark.{Partition, TaskContext}
* @param isOrderSensitive whether or not the function is order-sensitive.
If it's order
* sensitive, it may return totally different
result when the input order
* is changed. Mostly stateful functions are
order-sensitive.
+ * @param knownPartitioner If the result has a known partitioner.
*/
private[spark] class MapPartitionsRDD[U: ClassTag, T: ClassTag](
var prev: RDD[T],
f: (TaskContext, Int, Iterator[T]) => Iterator[U], // (TaskContext,
partition index, iterator)
preservesPartitioning: Boolean = false,
isFromBarrier: Boolean = false,
-isOrderSensitive: Boolean = false)
+isOrderSensitive: Boolean = false,
+knownPartitioner: Option[Partitioner] = None)
extends RDD[U](prev) {
- override val partitioner = if (preservesPartitioning)
firstParent[T].partitioner else None
+ override val partitioner = {
+if (preservesPartitioning) {
+ firstParent[T].partitioner
+} else {
+ knownPartitioner
+}
+ }
--- End diff --
Since we are already creating a `MapPartitionsRDD` in distinct, overriding
`partitioner` should be trivial.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22426 +1, LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96102/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19538: [SPARK-20393][WEBU UI][BACKPORT-2.0] Strengthen S...
Github user ambauma closed the pull request at: https://github.com/apache/spark/pull/19538 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96102/testReport)** for PR 22355 at commit [`f23705c`](https://github.com/apache/spark/commit/f23705cd8f214ffbda74a78a1c503c409ee7801f). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class InterpretedMutableProjection(expressions: Seq[Expression]) extends MutableProjection ` * `abstract class MutableProjection extends Projection ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96103/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96103/testReport)** for PR 22355 at commit [`76ca370`](https://github.com/apache/spark/commit/76ca37075398f890d19775e0037cb92072af53d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class InterpretedMutableProjection(expressions: Seq[Expression]) extends MutableProjection ` * `abstract class MutableProjection extends Projection ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22430 This is a pure test only PR. Backporting to 2.3 is optional. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22430 @gatorsmile . We need a backport to 2.3, too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky ExternalAp...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22432 cc @eyalfa @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky ExternalAp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22432 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky ExternalAp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22432 **[Test build #96104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96104/testReport)** for PR 22432 at commit [`04c3f7b`](https://github.com/apache/spark/commit/04c3f7b3c2a1b6a79d571ca2079ca6cc477027a7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky ExternalAp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22432 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3134/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite da...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22430 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22432: [SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky Ext...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22432
[SPARK-22713][CORE][TEST][FOLLOWUP] Fix flaky ExternalAppendOnlyMapSuite
due to timeout
## What changes were proposed in this pull request?
SPARK-22713 uses `eventually` with the default timeout `150ms`. It causes
flakiness because it's only executed once when GC is slow.
```scala
eventually {
System.gc()
...
}
```
**Failures**
```scala
org.scalatest.exceptions.TestFailedDueToTimeoutException:
The code passed to eventually never returned normally.
Attempted 1 times over 501.22261 milliseconds.
Last failure message: tmpIsNull was false.
```
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4916
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4906
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4907
## How was this patch tested?
Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-22713
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22432
commit 04c3f7b3c2a1b6a79d571ca2079ca6cc477027a7
Author: Dongjoon Hyun
Date: 2018-09-16T02:55:05Z
[SPARK-22713][CORE][FOLLOWUP] Fix flaky ExternalAppendOnlyMapSuite due to
timeout
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22430 Thanks! Merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21515: [SPARK-24372][build] Add scripts to help with pre...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21515#discussion_r217900617
--- Diff: dev/create-release/do-release-docker.sh ---
@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Creates a Spark release candidate. The script will update versions, tag
the branch,
+# build Spark binary packages and documentation, and upload maven
artifacts to a staging
+# repository. There is also a dry run mode where only local builds are
performed, and
+# nothing is uploaded to the ASF repos.
+#
+# Run with "-h" for options.
+#
+
+set -e
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/release-util.sh"
+
+function usage {
+ local NAME=$(basename $0)
+ cat <
[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22010#discussion_r217900574
--- Diff: core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
---
@@ -35,16 +35,24 @@ import org.apache.spark.{Partition, TaskContext}
* @param isOrderSensitive whether or not the function is order-sensitive.
If it's order
* sensitive, it may return totally different
result when the input order
* is changed. Mostly stateful functions are
order-sensitive.
+ * @param knownPartitioner If the result has a known partitioner.
*/
private[spark] class MapPartitionsRDD[U: ClassTag, T: ClassTag](
var prev: RDD[T],
f: (TaskContext, Int, Iterator[T]) => Iterator[U], // (TaskContext,
partition index, iterator)
preservesPartitioning: Boolean = false,
isFromBarrier: Boolean = false,
-isOrderSensitive: Boolean = false)
+isOrderSensitive: Boolean = false,
+knownPartitioner: Option[Partitioner] = None)
extends RDD[U](prev) {
- override val partitioner = if (preservesPartitioning)
firstParent[T].partitioner else None
+ override val partitioner = {
+if (preservesPartitioning) {
+ firstParent[T].partitioner
+} else {
+ knownPartitioner
+}
+ }
--- End diff --
`MapPartitionsRDD` is already private. But yes the other option is
sub-classing.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22010#discussion_r217900563
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -396,7 +396,26 @@ abstract class RDD[T: ClassTag](
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null):
RDD[T] = withScope {
-map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)
+partitioner match {
+ case Some(p) if numPartitions == partitions.length =>
+def key(x: T): (T, Null) = (x, null)
+val cleanKey = sc.clean(key _)
+val keyed = new MapPartitionsRDD[(T, Null), T](
+ this,
+ (context, pid, iter) => iter.map(cleanKey),
+ knownPartitioner = Some(new WrappedPartitioner(p)))
+val duplicatesRemoved = keyed.reduceByKey((x, y) => x)
--- End diff --
No reduceByKey on a known partioner is fine.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22430 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96100/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22430 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22430: [SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22430 **[Test build #96100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96100/testReport)** for PR 22430 at commit [`8277ff6`](https://github.com/apache/spark/commit/8277ff66286ed3a6438cfd4aa5c48c8f01ceb2ff). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchm...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22427 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22427 Merged to master/2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22427 Thank you, @maropu ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22418: [SPARK-25427][SQL][TEST] Add BloomFilter creation test c...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22418 Could you review this test case PR, @gatorsmile , @cloud-fan , @maropu , @HyukjinKwon ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22427 Thanks for the explanation! The change looks good to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22427 Thank you for review, @maropu . 1. Yes. It's the same. The first one limits the memory usage for write operation. The second one limits the memory usage for compression operation. 2. Yes. As you see in this PR, it's performance sensitive. Actually, all parameters of Parquet/ORC are performance sensitive. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96097/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22429 **[Test build #96097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96097/testReport)** for PR 22429 at commit [`9b2a3e6`](https://github.com/apache/spark/commit/9b2a3e664564182e325ef08785c998c6ff9b5367). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22413: [SPARK-25425][SQL] Extra options should override ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22413 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22413 Thank you, @MaxGekk . Merged to master/2.4/2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22414: [SPARK-25424][SQL] Window duration and slide duration wi...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22414 Please write the current behaviour (you described above) in the PR description? Also, can you format the PR description, too? you don't need the link to the jira. ``` â¦alues should fail fast (Link to Jira: https://issues.apache.org/jira/browse/SPARK-25424) ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22414: [SPARK-25424][SQL] Window duration and slide dura...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22414#discussion_r217898513
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimeWindow.scala
---
@@ -35,6 +35,10 @@ case class TimeWindow(
with ImplicitCastInputTypes
with Unevaluable
with NonSQLExpression {
+ require(windowDuration > 0, "The window duration must be " +
+s"a positive integer, long or string literal, found: $windowDuration")
+ require(slideDuration > 0, "The slide duration must be " +
+s"a positive integer, long or string literal, found: $slideDuration")
--- End diff --
We cannot check these requirements only in the constructor?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22424: [SPARK-25303][STREAMING] For checkpointed DStreams, remo...
Github user nikunjb commented on the issue: https://github.com/apache/spark/pull/22424 @dongjoon-hyun @tdas Can you please help review this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22423: [SPARK-25302][STREAMING] Checkpoint the reducedStream in...
Github user nikunjb commented on the issue: https://github.com/apache/spark/pull/22423 @dongjoon-hyun @tdas Can you please help review this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22427: [SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22427 Just a question; I'm not familiar with both internal logics though, these parameters (`Memory buffer for writing` and `Compression chunk size`) are internally treated in the same manner? Also, they are performace-sensitive parameters? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22413 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96098/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22413 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22413 **[Test build #96098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96098/testReport)** for PR 22413 at commit [`325b9c4`](https://github.com/apache/spark/commit/325b9c4790213b567153d6c9f85ae65ff64cc8e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96103/testReport)** for PR 22355 at commit [`76ca370`](https://github.com/apache/spark/commit/76ca37075398f890d19775e0037cb92072af53d2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3133/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96102/testReport)** for PR 22355 at commit [`f23705c`](https://github.com/apache/spark/commit/f23705cd8f214ffbda74a78a1c503c409ee7801f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3132/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22426: [SPARK-25436] Bump master branch version to 2.5.0...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22426 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22417: [SPARK-25426][SQL] Remove the duplicate fallback ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22417 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22426 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22355 **[Test build #96101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96101/testReport)** for PR 22355 at commit [`b1c6e86`](https://github.com/apache/spark/commit/b1c6e86e06c8d1e80af5d8bc6a3f1a6e08fa0026). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22355 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3131/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22426 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96095/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22426 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22426: [SPARK-25436] Bump master branch version to 2.5.0-SNAPSH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22426 **[Test build #96095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96095/testReport)** for PR 22426 at commit [`ee760b7`](https://github.com/apache/spark/commit/ee760b7e1a0485435de3ac94f1a920eda2fffa30). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22413 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
