[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161545059 @zsxwing Please take a look. Most of the change is in indentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10092#discussion_r46522204 --- Diff: python/pyspark/storagelevel.py --- @@ -49,12 +51,8 @@ def __str__(self): StorageLevel.DISK_ONLY = StorageLevel(True, False, False, False) StorageLevel.DISK_ONLY_2 = StorageLevel(True, False, False, False, 2) -StorageLevel.MEMORY_ONLY = StorageLevel(False, True, False, True) -StorageLevel.MEMORY_ONLY_2 = StorageLevel(False, True, False, True, 2) -StorageLevel.MEMORY_ONLY_SER = StorageLevel(False, True, False, False) --- End diff -- It's too late for 1.6, and this change (API change) is good for 2.0, sounds good? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161545237 +1 @yanboliang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12044] [SparkR] Fix usage of isnan, isn...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/10037#issuecomment-161545580 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12057] [SQL] Prevent failure on corrupt...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10043#issuecomment-161545561 @yhuai as an aside I moderately prefer not to introduce flags merely for the sake of being conservative or flexible. It rarely achieves that goal, just introduces complexity and rarely gets cleaned out, since you've just continued to promise a particular old behavior. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12044] [SparkR] Fix usage of isnan, isn...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10037#issuecomment-161545661 **[Test build #47127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47127/consoleFull)** for PR 10037 at commit [`3ee7d5c`](https://github.com/apache/spark/commit/3ee7d5c37a0b3815c2ff139964775d23e593837b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12044] [SparkR] Fix usage of isnan, isn...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10037#issuecomment-161545743 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12044] [SparkR] Fix usage of isnan, isn...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10037#issuecomment-161545748 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47127/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12016][MLlib][PySpark] Wrap Word2VecMod...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10100#issuecomment-161546106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47122/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12016][MLlib][PySpark] Wrap Word2VecMod...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10100#issuecomment-161545982 **[Test build #47122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47122/consoleFull)** for PR 10100 at commit [`56c250e`](https://github.com/apache/spark/commit/56c250e630a2fdc16809101a34a7eaa8b94e1a9e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12016][MLlib][PySpark] Wrap Word2VecMod...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10100#issuecomment-161546102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Working on spark 11827
GitHub user kevinyu98 opened a pull request: https://github.com/apache/spark/pull/10125 Working on spark 11827 Hello : Can you help check this PR? I am adding support for the java.math.BigInteger for java bean code path. I saw internally spark is converting the BigInteger to BigDecimal in ColumnType.scala and CatalystRowConverter.scala. I use the similar way and convert the BigInteger to the BigDecimal. . You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevinyu98/spark working_on_spark-11827 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10125.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10125 commit a67722094e8a9d0689ba022eb4f923e28791503e Author: Kevin Yu Date: 2015-12-01T16:38:09Z adding java.math.BigInteger support for java bean commit a58d92cd85719c6112c5cb0162be9b6104f9ba00 Author: Kevin Yu Date: 2015-12-02T05:37:56Z adding test case commit f400a825f38a2e3559e9b4f63b4e58bdd17c5e3b Author: Kevin Yu Date: 2015-12-03T07:38:15Z modify the JavaDataFrameSuite commit 3db875a7d9a331d3a200d26338c956d694001046 Author: Kevin Yu Date: 2015-12-03T07:50:43Z clean the JavaDataFrameSuite commit 0807550ae396231a19648c2f4db7e8946544d4a2 Author: Kevin Yu Date: 2015-12-03T07:57:20Z working on the JavaDataFrameSuite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/10092#discussion_r46522595 --- Diff: python/pyspark/storagelevel.py --- @@ -49,12 +51,8 @@ def __str__(self): StorageLevel.DISK_ONLY = StorageLevel(True, False, False, False) StorageLevel.DISK_ONLY_2 = StorageLevel(True, False, False, False, 2) -StorageLevel.MEMORY_ONLY = StorageLevel(False, True, False, True) -StorageLevel.MEMORY_ONLY_2 = StorageLevel(False, True, False, True, 2) -StorageLevel.MEMORY_ONLY_SER = StorageLevel(False, True, False, False) --- End diff -- Sure. Just changed it. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Working on spark 11827
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10125#issuecomment-161546768 @kevinyu98 please write a meaningful title and description. https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161546785 **[Test build #47128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47128/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161546939 **[Test build #47129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47129/consoleFull)** for PR 10092 at commit [`fef7ada`](https://github.com/apache/spark/commit/fef7ada1ec1d56c61de39c2268190bff6f6b0f42). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Working on spark 11827
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10125#issuecomment-161546895 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [DOCUMENTATION][KAFKA] fix typo in kafka/Offse...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10121#issuecomment-161547029 @microwishing can you have a look at the rest of the doc around this for any other typos? even small PRs have nonzero overhead, so I'm encouraging people to not just open a PR for single typos as they see them, but take a little time, if they're bothering, to evaluate similar issues around the change. Maybe there aren't any. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [DO NOT MERGE] Try to log some useful th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10120#issuecomment-161547681 **[Test build #47120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47120/consoleFull)** for PR 10120 at commit [`c8fc2ec`](https://github.com/apache/spark/commit/c8fc2ec5aea720c6f3b3553f9efa99bc5b545d70). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [DO NOT MERGE] Try to log some useful th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10120#issuecomment-161547818 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47120/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [DO NOT MERGE] Try to log some useful th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10120#issuecomment-161547817 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161548060 **[Test build #47125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47125/consoleFull)** for PR 10092 at commit [`014a3a8`](https://github.com/apache/spark/commit/014a3a8f31958bf1337a0c8df293fe15ac54cd9f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161548162 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161548165 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47125/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12044] [SparkR] Fix usage of isnan, isn...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10037#discussion_r46523622
--- Diff: R/pkg/R/generics.R ---
@@ -623,6 +623,10 @@ setGeneric("getItem", function(x, ...) {
standardGeneric("getItem") })
#' @rdname column
#' @export
+setGeneric("isNaN", function(x) { standardGeneric("isNaN") })
--- End diff --
open https://issues.apache.org/jira/browse/SPARK-12118 to explain in
release note.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10259] [ML] Add @since annotation to ml...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/8534#issuecomment-161548934 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12079][BUILD][SQL] Run Catalyst subproj...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10077#issuecomment-161553258 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161553502 **[Test build #47129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47129/consoleFull)** for PR 10092 at commit [`fef7ada`](https://github.com/apache/spark/commit/fef7ada1ec1d56c61de39c2268190bff6f6b0f42). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12088][SQL]check connection.isClosed be...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10095 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161553605 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47129/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12091] [PYSPARK] Deprecate the JAVA-spe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10092#issuecomment-161553604 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12088][SQL]check connection.isClosed be...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10095#issuecomment-161554438 Merged to master/1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11439][ML] Optimization of creating spa...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9756#discussion_r46525752
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala ---
@@ -131,35 +131,30 @@ object LinearDataGenerator {
eps: Double,
sparsity: Double): Seq[LabeledPoint] = {
require(0.0 <= sparsity && sparsity <= 1.0)
-val rnd = new Random(seed)
-val x = Array.fill[Array[Double]](nPoints)(
- Array.fill[Double](weights.length)(rnd.nextDouble()))
-
-val sparseRnd = new Random(seed)
-x.foreach { v =>
- var i = 0
- val len = v.length
- while (i < len) {
-if (sparseRnd.nextDouble() < sparsity) {
- v(i) = 0.0
-} else {
- v(i) = (v(i) - 0.5) * math.sqrt(12.0 * xVariance(i)) + xMean(i)
-}
-i += 1
- }
-}
-val y = x.map { xi =>
- blas.ddot(weights.length, xi, 1, weights, 1) + intercept + eps *
rnd.nextGaussian()
-}
-
-y.zip(x).map { p =>
- if (sparsity == 0.0) {
+val rnd = new Random(seed)
+if (sparsity == 0.0) {
+ (0 until nPoints).map { _ =>
+val features = Vectors.dense(weights.indices.map { i =>
+ (rnd.nextDouble() - 0.5) * math.sqrt(12.0 * xVariance(i)) +
xMean(i)
+}.toArray)
+val label = BLAS.dot(Vectors.dense(weights), features) +
+ intercept + eps * rnd.nextGaussian()
// Return LabeledPoints with DenseVector
-LabeledPoint(p._1, Vectors.dense(p._2))
- } else {
+LabeledPoint(label, features)
+ }
+} else {
+ val sparseRnd = new Random(seed)
--- End diff --
Why a second `Random` in this block?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12079][BUILD][SQL] Run Catalyst subproj...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10077#issuecomment-161556126 **[Test build #47131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47131/consoleFull)** for PR 10077 at commit [`12f3e3a`](https://github.com/apache/spark/commit/12f3e3a4a46e668ce2eecb4ff3bc21657608317d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12112][BUILD] Upgrade to SBT 0.13.9
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10112#issuecomment-161556223 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12112][BUILD] Upgrade to SBT 0.13.9
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10112#issuecomment-161556224 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47130/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12120][PYSPARK] Improve exception messa...
GitHub user zjffdu opened a pull request:
https://github.com/apache/spark/pull/10126
[SPARK-12120][PYSPARK] Improve exception message when failing to initâ¦
â¦ialize HiveContext in PySpark
This is the error message after this PR
```
15/12/03 16:59:53 WARN ObjectStore: Failed to get database default,
returning NoSuchObjectException
/Users/jzhang/github/spark/python/pyspark/sql/context.py:689: UserWarning:
You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt
assembly
warnings.warn("You must build Spark with Hive. "
Traceback (most recent call last):
File "", line 1, in
File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line
663, in read
return DataFrameReader(self)
File "/Users/jzhang/github/spark/python/pyspark/sql/readwriter.py", line
56, in __init__
self._jreader = sqlContext._ssql_ctx.read()
File "/Users/jzhang/github/spark/python/pyspark/sql/context.py", line
692, in _ssql_ctx
raise e
py4j.protocol.Py4JJavaError: An error occurred while calling
None.org.apache.spark.sql.hive.HiveContext.
: java.lang.RuntimeException: java.net.ConnectException: Call From
jzhangMBPr.local/127.0.0.1 to 0.0.0.0:9000 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at
org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:194)
at
org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at
org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
at
org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
at
org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
at
org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
at org.apache.spark.sql.UDFRegistration.(UDFRegistration.scala:40)
at org.apache.spark.sql.SQLContext.(SQLContext.scala:330)
at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:101)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:214)
at
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/spark SPARK-12120
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10126.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10126
commit 1878c700e9a39d612e05bf304a1a873d7a94bbde
Author: Jeff Zhang
Date: 2015-12-03T09:01:15Z
[SPARK-12120][PYSPARK] Improve exception message when failing to initialize
HiveContext in PySpark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161557846 **[Test build #47128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47128/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161557968 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47128/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161557967 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161558277 **[Test build #2156 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2156/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161558331 **[Test build #2157 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2157/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12120][PYSPARK] Improve exception messa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10126#issuecomment-161560614 **[Test build #47132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47132/consoleFull)** for PR 10126 at commit [`1878c70`](https://github.com/apache/spark/commit/1878c700e9a39d612e05bf304a1a873d7a94bbde). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161565087 **[Test build #2158 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2158/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161565148 **[Test build #2160 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2160/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161565449 **[Test build #2159 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2159/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161565810 **[Test build #47133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47133/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11439][ML] Optimization of creating spa...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9756#discussion_r46529174
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala ---
@@ -131,35 +131,30 @@ object LinearDataGenerator {
eps: Double,
sparsity: Double): Seq[LabeledPoint] = {
require(0.0 <= sparsity && sparsity <= 1.0)
-val rnd = new Random(seed)
-val x = Array.fill[Array[Double]](nPoints)(
- Array.fill[Double](weights.length)(rnd.nextDouble()))
-
-val sparseRnd = new Random(seed)
-x.foreach { v =>
- var i = 0
- val len = v.length
- while (i < len) {
-if (sparseRnd.nextDouble() < sparsity) {
- v(i) = 0.0
-} else {
- v(i) = (v(i) - 0.5) * math.sqrt(12.0 * xVariance(i)) + xMean(i)
-}
-i += 1
- }
-}
-val y = x.map { xi =>
- blas.ddot(weights.length, xi, 1, weights, 1) + intercept + eps *
rnd.nextGaussian()
-}
-
-y.zip(x).map { p =>
- if (sparsity == 0.0) {
+val rnd = new Random(seed)
+if (sparsity == 0.0) {
+ (0 until nPoints).map { _ =>
+val features = Vectors.dense(weights.indices.map { i =>
+ (rnd.nextDouble() - 0.5) * math.sqrt(12.0 * xVariance(i)) +
xMean(i)
--- End diff --
Maybe you could refactor this into a small `def` local to the method to
avoid repeating it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11439][ML] Optimization of creating spa...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9756#discussion_r46529273
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala ---
@@ -131,35 +131,30 @@ object LinearDataGenerator {
eps: Double,
sparsity: Double): Seq[LabeledPoint] = {
require(0.0 <= sparsity && sparsity <= 1.0)
-val rnd = new Random(seed)
-val x = Array.fill[Array[Double]](nPoints)(
- Array.fill[Double](weights.length)(rnd.nextDouble()))
-
-val sparseRnd = new Random(seed)
-x.foreach { v =>
- var i = 0
- val len = v.length
- while (i < len) {
-if (sparseRnd.nextDouble() < sparsity) {
- v(i) = 0.0
-} else {
- v(i) = (v(i) - 0.5) * math.sqrt(12.0 * xVariance(i)) + xMean(i)
-}
-i += 1
- }
-}
-val y = x.map { xi =>
- blas.ddot(weights.length, xi, 1, weights, 1) + intercept + eps *
rnd.nextGaussian()
-}
-
-y.zip(x).map { p =>
- if (sparsity == 0.0) {
+val rnd = new Random(seed)
+if (sparsity == 0.0) {
+ (0 until nPoints).map { _ =>
+val features = Vectors.dense(weights.indices.map { i =>
+ (rnd.nextDouble() - 0.5) * math.sqrt(12.0 * xVariance(i)) +
xMean(i)
+}.toArray)
+val label = BLAS.dot(Vectors.dense(weights), features) +
+ intercept + eps * rnd.nextGaussian()
// Return LabeledPoints with DenseVector
-LabeledPoint(p._1, Vectors.dense(p._2))
- } else {
+LabeledPoint(label, features)
+ }
+} else {
+ val sparseRnd = new Random(seed)
+ (0 until nPoints).map { _ =>
+val (values, indices) = weights.indices.filter { _ =>
+ sparseRnd.nextDouble() <= sparsity }.map { i =>
+ ((rnd.nextDouble() - 0.5) * math.sqrt(12.0 * xVariance(i)) +
xMean(i), i)
+}.unzip
--- End diff --
I might be over-thinking this, but I wonder if it's significantly more
efficient to choose the indices as an array, and then map that to values
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/10127 [SPARK-12122][STREAMING] Prevent batches from being submitted twice after recovering StreamingContext from checkpoint You can merge this pull request into a Git repository by running: $ git pull https://github.com/tdas/spark SPARK-12122 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10127 commit d904b25a7037e2b12693158f29e069f13aa0fa78 Author: Tathagata Das Date: 2015-12-03T09:30:27Z Remove duplicate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161567164 @zsxwing Please check this. I think this problem has been caused by the #9707 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161567492 @zsxwing Please merge this to older branches if possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161567552 **[Test build #2161 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2161/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11439][ML] Optimization of creating spa...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/9756#discussion_r46529657 --- Diff: mllib/src/test/scala/org/apache/spark/ml/evaluation/RegressionEvaluatorSuite.scala --- @@ -65,15 +65,15 @@ class RegressionEvaluatorSuite // default = rmse val evaluator = new RegressionEvaluator() -assert(evaluator.evaluate(predictions) ~== 0.1019382 absTol 0.001) +assert(evaluator.evaluate(predictions) ~== 0.1013829 absTol 0.001) --- End diff -- Yeah I expect some values change due to differing use of the RNG. Some of the differences are larger than the test's tolerance. That could just mean the tolerance is too tight. I wonder if it's worth loosening them where it's clearly too small (implicitly, by adjusting the expected value we're accepting this). Like, these could be +/- 0.01. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161567545 **[Test build #2162 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2162/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161569412 **[Test build #47134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47134/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12120][PYSPARK] Improve exception messa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10126#issuecomment-161570578 **[Test build #47132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47132/consoleFull)** for PR 10126 at commit [`1878c70`](https://github.com/apache/spark/commit/1878c700e9a39d612e05bf304a1a873d7a94bbde). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12120][PYSPARK] Improve exception messa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10126#issuecomment-161570708 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47132/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12120][PYSPARK] Improve exception messa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10126#issuecomment-161570707 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12084][Core]Fix codes that uses ByteBuf...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10083#issuecomment-161575365 **[Test build #47126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47126/consoleFull)** for PR 10083 at commit [`81d1812`](https://github.com/apache/spark/commit/81d18120bff0a772a566ddfe19e439f309b5d5df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12084][Core]Fix codes that uses ByteBuf...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10083#issuecomment-161575897 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12084][Core]Fix codes that uses ByteBuf...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10083#issuecomment-161575904 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47126/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r46532508
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ // The setApplicationTags method was only introduced in Hadoop
2.4+, so we need to use
+ // reflection to set it, printing a warning if a tag was
specified but the YARN version
+ // doesn't support it.
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+} catch {
+ case e: NoSuchMethodException =>
+logWarning(s"Ignoring $CONF_SPARK_YARN_APPLICATION_TAGS
because this version of " +
+ "YARN does not support it")
+}
+ }
--- End diff --
using reflect make the code reading more difficult. generally not be
recommanded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161579383 **[Test build #2156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2156/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [FLAKY-TEST-FIX][STREAMING][TEST] Make sure St...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10124#issuecomment-161581627 **[Test build #2157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2157/consoleFull)** for PR 10124 at commit [`a66723e`](https://github.com/apache/spark/commit/a66723e5451b9e001e578fb1bbc56aeeea9ba439). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161582530 **[Test build #2159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2159/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: pull out nondeterministic expressions from Joi...
GitHub user zhonghaihua opened a pull request: https://github.com/apache/spark/pull/10128 pull out nondeterministic expressions from Join Currently,`nondeterministic expressions` are only allowed in `Project` or `Filter`,And only when we use nondeterministic expressions in `UnaryNode` can be pulled out. But,Sometime in many case,we will use nondeterministic expressions to process `join keys` avoiding data skew.for example: ``` select * from tableA a join (select * from tableB) b on upper((case when (a.brand_code is null or a.brand_code = '' ) then cast( (-rand() * 1000 ) as string ) else a.brand_code end )) = b.brand_code ``` This PR introduce a mechanism to pull out nondeterministic expressions from `Join`,so we can use nondeterministic expression in `Join` appropriately. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhonghaihua/spark pulloutJoinNondeterministic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10128.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10128 commit 6e166578a5c1a1faf260389509663ac8c71ec015 Author: zhonghaihua <793507...@qq.com> Date: 2015-11-30T07:44:49Z pull out nondeterministic expressions from Join pull out nondeterministic expressions from Join pull out nondeterministic expressions from Join --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: pull out nondeterministic expressions from Joi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10128#issuecomment-161583163 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161584579 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47133/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161584196 **[Test build #47133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47133/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161584575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4117] [YARN] Spark on Yarn handle AM be...
GitHub user devaraj-kavali opened a pull request: https://github.com/apache/spark/pull/10129 [SPARK-4117] [YARN] Spark on Yarn handle AM being told command from RM Spark on Yarn handle AM being told command from RM When RM throws ApplicationAttemptNotFoundException for allocate invocation, making the ApplicationMaster to finish immediately without any retries. You can merge this pull request into a Git repository by running: $ git pull https://github.com/devaraj-kavali/spark SPARK-4117 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10129 commit 636fd782c9b18840d5fabb59f625993ad5d73fc5 Author: Devaraj K Date: 2015-12-03T10:21:34Z Spark on Yarn handle AM being told command from RM When RM throws ApplicationAttemptNotFoundException for allocate invocation, making the ApplicationMaster to finish immediately without any retries. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161585644 **[Test build #2158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2158/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4117] [YARN] Spark on Yarn handle AM be...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10129#issuecomment-161585597 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11932][STREAMING] Partition previous Tr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9988#issuecomment-161586067 **[Test build #2160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2160/consoleFull)** for PR 9988 at commit [`53846f5`](https://github.com/apache/spark/commit/53846f56eaa5f423cae44c7e287971325703b12d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [DOCUMENTATION][KAFKA] fix typo in kafka/Offse...
Github user microwishing commented on the pull request: https://github.com/apache/spark/pull/10121#issuecomment-161586166 @srowen thanks for your tips. I happened to work on some kafka project recently so I went through source codes including comments for spark kafka streaming part today. I spent some time setting up the spark dev environment and feel good so far due to rich doc. Those are indeed trivial typos. I can make check-ins next time after fixing some real bugs there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: pull out nondeterministic expressions from Joi...
Github user zhonghaihua commented on the pull request: https://github.com/apache/spark/pull/10128#issuecomment-161590938 @rxin @cloud-fan @chenghao-intel @jeanlyn Could you give some suggestions on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161590894 **[Test build #2161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2161/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161591828 **[Test build #47134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47134/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161592110 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47134/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161592106 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11987] Python API update for ChiSqSelec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10007#issuecomment-161592280 **[Test build #47136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47136/consoleFull)** for PR 10007 at commit [`3a33327`](https://github.com/apache/spark/commit/3a33327122ae94d59403d807255273180528d9a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161593111 **[Test build #47135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47135/consoleFull)** for PR 10123 at commit [`ed691eb`](https://github.com/apache/spark/commit/ed691ebd6bcbed28101a7af7760d9fcc01031c8d). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161593117 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47135/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161593116 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161593100 **[Test build #47135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47135/consoleFull)** for PR 10123 at commit [`ed691eb`](https://github.com/apache/spark/commit/ed691ebd6bcbed28101a7af7760d9fcc01031c8d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161598230 **[Test build #47137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47137/consoleFull)** for PR 10123 at commit [`94c596d`](https://github.com/apache/spark/commit/94c596d9e0d3be3b37b97b86d361f26d648a7bbf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161601793 **[Test build #47137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47137/consoleFull)** for PR 10123 at commit [`94c596d`](https://github.com/apache/spark/commit/94c596d9e0d3be3b37b97b86d361f26d648a7bbf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161601910 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12115] [SparkR] Change numPartitions() ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10123#issuecomment-161601912 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47137/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11987] Python API update for ChiSqSelec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10007#issuecomment-161602541 **[Test build #47136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47136/consoleFull)** for PR 10007 at commit [`3a33327`](https://github.com/apache/spark/commit/3a33327122ae94d59403d807255273180528d9a9). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class QuantileDiscretizer(JavaEstimator, HasInputCol, HasOutputCol):`\n * ` class QuantileDiscretizerModel(JavaModel):`\n * `class ChiSqSelector(JavaEstimator, HasFeaturesCol, HasOutputCol, HasLabelCol):`\n * `class ChiSqSelectorModel(JavaModel):`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12122][STREAMING] Prevent batches from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10127#issuecomment-161608484 **[Test build #2162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2162/consoleFull)** for PR 10127 at commit [`d904b25`](https://github.com/apache/spark/commit/d904b25a7037e2b12693158f29e069f13aa0fa78). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class CrossValidator @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String)`\n * `class ParamGridBuilder @Since(\"1.2.0\") `\n * `class TrainValidationSplit @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4117] [YARN] Spark on Yarn handle AM be...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/10129#discussion_r46546311
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -370,6 +371,12 @@ private[spark] class ApplicationMaster(
failureCount = 0
} catch {
case i: InterruptedException =>
+case a: ApplicationAttemptNotFoundException => {
+ val message = "ApplicationAttemptNotFoundException was
thrown from Reporter thread.";
--- End diff --
`;` is not needed for Scala, also `{...}` is not necessary for this code
block.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12125][SQL] pull out nondeterministic e...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/10128#issuecomment-161626933 I think it's not a good example to show that we need to allow nondeterministic expressions in join codition. We can use `Repartition` operator to fix data skew, like `sqlContext.table(tblName).repartition(numPartitions).registerTempTable`, which looks better than your random join approach. Do you find other cases that need to use nondeterministic expressions in join codition? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12079][BUILD][SQL] Run Catalyst subproj...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10077#issuecomment-161632276 **[Test build #47131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47131/consoleFull)** for PR 10077 at commit [`12f3e3a`](https://github.com/apache/spark/commit/12f3e3a4a46e668ce2eecb4ff3bc21657608317d). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12079][BUILD][SQL] Run Catalyst subproj...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10077#issuecomment-161632526 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47131/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12079][BUILD][SQL] Run Catalyst subproj...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10077#issuecomment-161632524 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9778][SQL] remove unnecessary evaluatio...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8066#issuecomment-161644398 **[Test build #47138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47138/consoleFull)** for PR 8066 at commit [`79f2dc9`](https://github.com/apache/spark/commit/79f2dc9c56bf9635446c522690bb998955756ca7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12105 - SPARK-SQL add convenient show fu...
GitHub user jbonofre opened a pull request: https://github.com/apache/spark/pull/10130 SPARK-12105 - SPARK-SQL add convenient show functions You can merge this pull request into a Git repository by running: $ git pull https://github.com/jbonofre/spark SPARK-12105 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10130.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10130 commit 1d583323a2d7ae01962bf707262ae6e66cdb5bd6 Author: Jean-Baptiste Onofré Date: 2015-12-03T13:50:59Z SPARK-12105 - SPARK-SQL add convenient show functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12105 - SPARK-SQL add convenient show fu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10130#issuecomment-161645600 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
