[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3282#issuecomment-63164349 [Test build #23421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23421/consoleFull) for PR 3282 at commit [`7a9d8be`](https://github.com/apache/spark/commit/7a9d8be32f2db2d7a023414e8ab85c1fcfd136fb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63164284 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23415/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63164283 [Test build #23415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23415/consoleFull) for PR 3281 at commit [`7a10123`](https://github.com/apache/spark/commit/7a10123aa35c8558f4913eb5d2b56a84d46f6e82). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3282#issuecomment-63164257 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3282#issuecomment-63164249 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63164213 [Test build #23419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23419/consoleFull) for PR 3281 at commit [`5f1fa5e`](https://github.com/apache/spark/commit/5f1fa5efe1f3894e7d09e841225eef3839b39b1a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4393] Fix memory leak in ConnectionMana...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3259#issuecomment-63164209 [Test build #23420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23420/consoleFull) for PR 3259 at commit [`afcc8d6`](https://github.com/apache/spark/commit/afcc8d66f58bcb803ea44651b2c119fe3888d2a3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4327] [PySpark] Python API for RDD.rand...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3193#issuecomment-63164193 I don't really feel qualified to give an opinion here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
GitHub user adampingel opened a pull request: https://github.com/apache/spark/pull/3282 SPARK-2811 upgrade algebird to 0.8.1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/adampingel/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3282.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3282 commit 7a9d8be32f2db2d7a023414e8ab85c1fcfd136fb Author: Adam Pingel Date: 2014-11-15T07:43:39Z SPARK-2811 upgrade algebird to 0.8.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3074] [PySpark] support groupByKey() wi...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1977#issuecomment-63164145 I agree that this is a good fix; I've been letting the review slip because this PR is pretty complex and it will take me a decent amount of time to be sure that it's correct in all cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63164138 Merging in master & branch-1.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added contains(key) to Metadata
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3273#issuecomment-63164092 Merging in master & branch-1.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2947#issuecomment-63164082 That might actually be easier (closing this one and submitting a new one) than fixing the merge on this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63164073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23413/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63164072 [Test build #23413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23413/consoleFull) for PR 3197 at commit [`30b0afa`](https://github.com/apache/spark/commit/30b0afa7e2f0b4e40f56867a515ec1cea07c43e9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3121#issuecomment-63163937 [Test build #23418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23418/consoleFull) for PR 3121 at commit [`c0987d3`](https://github.com/apache/spark/commit/c0987d373596284c05189635f992a0828df2e0eb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3121#issuecomment-63163914 I've made another pass which I think should address this last round of review feedback. Thanks for all of the careful review and commentary so far. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3121#discussion_r20401015
--- Diff: pom.xml ---
@@ -978,6 +978,7 @@
1
false
${test_classpath}
+
true
--- End diff --
I've created https://issues.apache.org/jira/browse/SPARK-4424 so that we
remember to finish the test cleanup / refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4363][Doc] Update the Broadcast example
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3226 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added contains(key) to Metadata
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3273#issuecomment-63163844 [Test build #23414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23414/consoleFull) for PR 3273 at commit [`74b3d03`](https://github.com/apache/spark/commit/74b3d03fad0a887091d8f0fb25f2d8f52705cad6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4379][Core] Change Exception to SparkEx...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3241 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added contains(key) to Metadata
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3273#issuecomment-63163846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23414/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4260] Httpbroadcast should set connecti...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3122 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63163715 [Test build #23417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23417/consoleFull) for PR 3281 at commit [`50783bd`](https://github.com/apache/spark/commit/50783bd617e0e177d6bb5b31340bfd101c5cd4ab). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user adampingel commented on the pull request: https://github.com/apache/spark/pull/2947#issuecomment-63163705 @rxin Worst case I'll create another pull request with the same 2-file changeset --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63163655 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63163568 [Test build #23416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23416/consoleFull) for PR 3278 at commit [`7fea558`](https://github.com/apache/spark/commit/7fea5587d708729eaf017029f3bc74cea74b64b7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63163298 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23412/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63163296 [Test build #23412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23412/consoleFull) for PR 3278 at commit [`cb2301a`](https://github.com/apache/spark/commit/cb2301a893397b7c4c08d3437eca4161a81c9031). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2811 upgrade algebird to 0.8.1
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2947#issuecomment-63163264 @adampingel looks like the merge screwed up. It now shows 200+ commits. Can you fix this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3262#issuecomment-63163240 Ok I finally went through the code. I like the change and it is pretty clever. I believe it should preserve both source compatibility and binary compatibility. To summarize, the changes are: 1. Deprecated the old implicit conversion functions: this preserves binary compatibility for code compiled against earlier versions of Spark. 2. Removed "implicit" from them so they are just normal functions: this made sure the compiler doesn't get confused and warn about multiple implicits in scope. 3. Created new implicit functions in package rdd object, which is part of the scope that scalac will search when looking for implicit conversions on various RDD objects. It is still a tricky change so it'd be great to get more eyes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/3262#issuecomment-63163199 @zsxwing just curious, with the old conversions being deprecated, is there any chance they'll create compiler warnings in common uses of the code? In any case this seems pretty cool if it doesn't actually break binary compatibility. I guess one risk is if it adds new implicits that cause something to compile differently, but it seems unlikely from a first glance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63163080 Thanks! LGTM except the imports should be organized. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3281#discussion_r20400659
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala ---
@@ -19,6 +19,8 @@ package org.apache.spark.mllib.linalg
import org.scalatest.FunSuite
+import breeze.linalg.{DenseVector => BDV, DenseMatrix => BDM}
--- End diff --
This should be in the same group as `org.scalatest.FunSuite` because both
are 3rd-party imports.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3262#discussion_r20400656
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1500,32 +1510,40 @@ object SparkContext extends Logging {
new WritableConverter[T](_ => wClass, x => convert(x.asInstanceOf[W]))
}
- implicit def intWritableConverter(): WritableConverter[Int] =
+ @deprecated("An API for backforward compatibility", "1.2.0")
--- End diff --
is the simpleWritableConverter in SparkContext still necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4309][SPARK-4407][SQL] Date type suppor...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3178#issuecomment-63162955 @marmbrus This should be good to go now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4393] Fix memory leak in ConnectionMana...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3259#discussion_r20400611
--- Diff:
core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala ---
@@ -899,22 +904,41 @@ private[nio] class ConnectionManager(
: Future[Message] = {
val promise = Promise[Message]()
-val timeoutTask = new TimerTask {
- override def run(): Unit = {
+// It's important that the TimerTask doesn't capture a reference to
`message`, which can cause
+// memory leaks since cancelled TimerTasks won't necessarily be
garbage collected until the time
+// at which they would originally be scheduled to run. Therefore,
extract the message id
+// from outside of the TimerTask closure (see SPARK-4393 for more
context).
+val messageId = message.id
+// Keep a weak reference to the promise so that the completed promise
may be garbage-collected
+val promiseReference = new WeakReference(promise)
+val timeoutTask: TimerTask = new TimerTask {
+ override def run(timeout: Timeout): Unit = {
messageStatuses.synchronized {
- messageStatuses.remove(message.id).foreach ( s => {
+ messageStatuses.remove(messageId).foreach ( s => {
val e = new IOException("sendMessageReliably failed because
ack " +
s"was not received within $ackTimeout sec")
-if (!promise.tryFailure(e)) {
- logWarning("Ignore error because promise is completed", e)
+Option(promiseReference.get) match {
--- End diff --
I was actually on the fence about this, but your comment tips me towards
the `== null` camp since it removes a level of nesting / indentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Bumping version to 1.3.0-SNAPSHOT.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3277#issuecomment-63162905 [Test build #23410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23410/consoleFull) for PR 3277 at commit [`4ad2f18`](https://github.com/apache/spark/commit/4ad2f18148ff3f50cc68ad19eb18e4afce9651d0). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Bumping version to 1.3.0-SNAPSHOT.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3277#issuecomment-63162907 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23410/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3262#discussion_r20400609
--- Diff: core/src/main/scala/org/apache/spark/rdd/package.scala ---
@@ -17,7 +17,34 @@
package org.apache.spark
+import scala.language.implicitConversions
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.io.Writable
+
/**
* Provides several RDD implementations. See [[org.apache.spark.rdd.RDD]].
*/
-package object rdd
+package object rdd {
+
+ implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
--- End diff --
would be great to add some comment mentioning the duplicate code in
SparkContext and say pre-1.2, these were in SparkContext and users had to
import SparkContext._
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3262#discussion_r20400598
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1427,46 +1427,56 @@ object SparkContext extends Logging {
private[spark] val DRIVER_IDENTIFIER = ""
- implicit object DoubleAccumulatorParam extends AccumulatorParam[Double] {
+ @deprecated("An API for backforward compatibility", "1.2.0")
+ object DoubleAccumulatorParam extends AccumulatorParam[Double] {
--- End diff --
Do you mind updating the deprecation message to say
"Replaced by implicit objects in AccumulatorParam. This is kept here only
for backward binary compatibility."
Do it for all the following.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Reorganize 'implicit's to i...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3262#discussion_r20400601
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1427,46 +1427,56 @@ object SparkContext extends Logging {
private[spark] val DRIVER_IDENTIFIER = ""
- implicit object DoubleAccumulatorParam extends AccumulatorParam[Double] {
+ @deprecated("An API for backforward compatibility", "1.2.0")
+ object DoubleAccumulatorParam extends AccumulatorParam[Double] {
def addInPlace(t1: Double, t2: Double): Double = t1 + t2
def zero(initialValue: Double) = 0.0
}
- implicit object IntAccumulatorParam extends AccumulatorParam[Int] {
+ @deprecated("An API for backforward compatibility", "1.2.0")
+ object IntAccumulatorParam extends AccumulatorParam[Int] {
def addInPlace(t1: Int, t2: Int): Int = t1 + t2
def zero(initialValue: Int) = 0
}
- implicit object LongAccumulatorParam extends AccumulatorParam[Long] {
+ @deprecated("An API for backforward compatibility", "1.2.0")
+ object LongAccumulatorParam extends AccumulatorParam[Long] {
def addInPlace(t1: Long, t2: Long) = t1 + t2
def zero(initialValue: Long) = 0L
}
- implicit object FloatAccumulatorParam extends AccumulatorParam[Float] {
+ @deprecated("An API for backforward compatibility", "1.2.0")
+ object FloatAccumulatorParam extends AccumulatorParam[Float] {
def addInPlace(t1: Float, t2: Float) = t1 + t2
def zero(initialValue: Float) = 0f
}
// TODO: Add AccumulatorParams for other types, e.g. lists and strings
- implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
+ @deprecated("An API for backforward compatibility", "1.2.0")
--- End diff --
update these accordingly too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4260] Httpbroadcast should set connecti...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3122#issuecomment-63162821 Thanks a lot! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4260] Httpbroadcast should set connecti...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3122#issuecomment-63162815 Ok this makes sense. I'm merging in master & branch-1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3121#discussion_r20400579
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1417,6 +1427,97 @@ class SparkContext(config: SparkConf) extends
SparkStatusAPI with Logging {
*/
object SparkContext extends Logging {
+ /**
+ * Lock that guards access to global variables that track SparkContext
construction.
+ */
+ private[spark] val SPARK_CONTEXT_CONSTRUCTOR_LOCK = new Object()
+
+ /**
+ * Records the creation site of the active, fully-constructed
SparkContext. If no SparkContext
+ * is active, then this is `None`.
+ *
+ * Access to this field is guarded by SPARK_CONTEXT_CONSTRUCTOR_LOCK
+ */
+ private[spark] var activeContextCreationSite: Option[CallSite] = None
+
+ /**
+ * Points to a partially-constructed SparkContext if some thread is in
the SparkContext
+ * constructor, or `None` if no SparkContext is being constructed.
+ *
+ * Access to this field is guarded by SPARK_CONTEXT_CONSTRUCTOR_LOCK
+ */
+ private[spark] var contextBeingConstructed: Option[SparkContext] = None
+
+ /**
+ * Called to ensure that no other SparkContext is running in this JVM.
+ *
+ * Throws an exception if a running context is detected and logs a
warning if another thread is
+ * constructing a SparkContext. This warning is necessary because the
current locking scheme
+ * prevents us from reliably distinguishing between cases where another
context is being
+ * constructed and cases where another constructor threw an exception.
+ */
+ private def assertNoOtherContextIsRunning(sc: SparkContext, conf:
SparkConf) {
+SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
+ contextBeingConstructed.foreach { otherContext =>
+if (otherContext ne sc) {
+ val warnMsg = "Another SparkContext is being constructed (or
threw an exception in its" +
+" constructor). This may indicate an error, since only one
SparkContext may be" +
+" running in this JVM (see SPARK-2243)."
+ logWarning(warnMsg)
+}
+
+activeContextCreationSite.foreach { creationSite =>
+ val errMsg = "Only one SparkContext may be running in this JVM
(see SPARK-2243)." +
+" To ignore this error, set spark.driver.allowMultipleContexts
= true. " +
+s"The currently running SparkContext was created
at:\n${creationSite.longForm}"
+ val exception = new SparkException(errMsg)
+ if (conf.getBoolean("spark.driver.allowMultipleContexts",
false)) {
+logWarning("Multiple running SparkContexts detected in the
same JVM!", exception)
+ } else {
+throw exception
+ }
+}
+ }
+}
+ }
+
+ /**
+ * Called at the beginning of the SparkContext constructor to ensure
that no SparkContext is
+ * running. Throws an exception if a running context is detected and
logs a warning if another
+ * thread is constructing a SparkContext. This warning is necessary
because the current locking
+ * scheme prevents us from reliably distinguishing between cases where
another context is being
+ * constructed and cases where another constructor threw an exception.
+ */
+ private[spark] def markPartiallyConstructed(sc: SparkContext, conf:
SparkConf) {
--- End diff --
Regarding the `conf` parameter, take a look at how
`markPartiallyConstructed` is called at the very top of SparkContext: at this
point, `sc.conf` hasn't been set and we only have the `config` that's passed
via the SparkContext constructor. I could make that `config` in the
constructor into a `private val`, but that seems like it has a high potential
for confusion. I could also move the call to `markPartiallyConstructed`
further down in the SparkContext constructor, but that creates confusing
implicit ordering dependencies.
Maybe it's clearer to change the constructor to accept a
`allowMultipleContexts` boolean and to populate it at the callsite from a
SparkConf. I suppose that this has the disadvantage of requiring each caller
to implement the logic for picking the default configuration value, but we
could just extract that into a private local variable in SparkContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail:
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63162772 Hmmm in case I haven't expressed this earlier, I really like this new API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] SPARK-4335, Mima false positives.
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3201#issuecomment-63162754 Yea I think we can close this one for now since we worked around it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3281#issuecomment-63162749 [Test build #23415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23415/consoleFull) for PR 3281 at commit [`7a10123`](https://github.com/apache/spark/commit/7a10123aa35c8558f4913eb5d2b56a84d46f6e82). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4422][MLLIB]In some cases, Vectors.from...
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/3281 [SPARK-4422][MLLIB]In some cases, Vectors.fromBreeze get wrong results. cc @mengxr You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-4422 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3281.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3281 commit 7a10123aa35c8558f4913eb5d2b56a84d46f6e82 Author: GuoQiang Li Date: 2014-11-15T06:27:42Z In some cases, Vectors.fromBreeze get wrong results. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4363][Doc] Update the Broadcast example
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3226#issuecomment-63162660 Thanks. Merging in master & branch-1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3665] Java API for GraphX
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3234#issuecomment-63162620 Ok since this is so big, we should probably put this for 1.3 now :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4180] [Core] Prevent creation of multip...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3121#discussion_r20400528
--- Diff: pom.xml ---
@@ -978,6 +978,7 @@
1
false
${test_classpath}
+
true
--- End diff --
Yeah, I think that we'd eventually want to have this be `false`, but there
are a few test suites for which this might be difficult (such as SparkSQL) and
I didn't want to undertake a huge refactoring in this PR. If you SSH into
Jenkins and look at the `unit-tests.log` files, you'll notice that this PR
introduces a very small number of warning log messages in the tests, since most
suites properly clean up their contexts. I did manage to fix a few of the
obvious cases that I found, so this PR does contain a bit of test refactoring.
I'll open a JIRA to finish this cleanup later.
By the way, if you look in `SparkContextSuite.scala`, there's a test
fixture that changes system properties in individual unit tests (and resets
them after the test); I use this to ensure that the property / configuration
option works as expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added contains(key) to Metadata
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3273#issuecomment-63162543 [Test build #23414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23414/consoleFull) for PR 3273 at commit [`74b3d03`](https://github.com/apache/spark/commit/74b3d03fad0a887091d8f0fb25f2d8f52705cad6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4379][Core] Change Exception to SparkEx...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3241#issuecomment-63162549 I'm merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4379][Core] Change Exception to SparkEx...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3241#issuecomment-63162528 I think this should be ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4393] Fix memory leak in ConnectionMana...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3259#discussion_r20400513
--- Diff:
core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala ---
@@ -899,22 +904,41 @@ private[nio] class ConnectionManager(
: Future[Message] = {
val promise = Promise[Message]()
-val timeoutTask = new TimerTask {
- override def run(): Unit = {
+// It's important that the TimerTask doesn't capture a reference to
`message`, which can cause
+// memory leaks since cancelled TimerTasks won't necessarily be
garbage collected until the time
+// at which they would originally be scheduled to run. Therefore,
extract the message id
+// from outside of the TimerTask closure (see SPARK-4393 for more
context).
+val messageId = message.id
+// Keep a weak reference to the promise so that the completed promise
may be garbage-collected
+val promiseReference = new WeakReference(promise)
+val timeoutTask: TimerTask = new TimerTask {
+ override def run(timeout: Timeout): Unit = {
messageStatuses.synchronized {
- messageStatuses.remove(message.id).foreach ( s => {
+ messageStatuses.remove(messageId).foreach ( s => {
--- End diff --
nitpick here - u can remove one layer of parenthesis/brackets
```scala
messageStatuses.remove(messageId).foreach { s =>
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4393] Fix memory leak in ConnectionMana...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3259#discussion_r20400510
--- Diff:
core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala ---
@@ -899,22 +904,41 @@ private[nio] class ConnectionManager(
: Future[Message] = {
val promise = Promise[Message]()
-val timeoutTask = new TimerTask {
- override def run(): Unit = {
+// It's important that the TimerTask doesn't capture a reference to
`message`, which can cause
+// memory leaks since cancelled TimerTasks won't necessarily be
garbage collected until the time
+// at which they would originally be scheduled to run. Therefore,
extract the message id
+// from outside of the TimerTask closure (see SPARK-4393 for more
context).
+val messageId = message.id
+// Keep a weak reference to the promise so that the completed promise
may be garbage-collected
+val promiseReference = new WeakReference(promise)
+val timeoutTask: TimerTask = new TimerTask {
+ override def run(timeout: Timeout): Unit = {
messageStatuses.synchronized {
- messageStatuses.remove(message.id).foreach ( s => {
+ messageStatuses.remove(messageId).foreach ( s => {
val e = new IOException("sendMessageReliably failed because
ack " +
s"was not received within $ackTimeout sec")
-if (!promise.tryFailure(e)) {
- logWarning("Ignore error because promise is completed", e)
+Option(promiseReference.get) match {
--- End diff --
why not
```scala
val p = promiseReference.get
if (p == null) {
...
} else {
...
}
```
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3197#issuecomment-63162463 [Test build #23413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23413/consoleFull) for PR 3197 at commit [`30b0afa`](https://github.com/apache/spark/commit/30b0afa7e2f0b4e40f56867a515ec1cea07c43e9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3862][SQL][WIP] MultiWayBroadcastInnerH...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2985#issuecomment-63162455 Closing this one since it's been subsumed by https://github.com/apache/spark/pull/3270 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3862][SQL][WIP] MultiWayBroadcastInnerH...
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/2985 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added contains(key) to Metadata
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3273#issuecomment-63162406 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4410][SQL] Add support for external sor...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3268#issuecomment-63162379 LGTM other than the minor comment. One thing I noticed is that we'd want to control the closure size at some point. Right now the entire query plan is being captured by every stage. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user watermen closed the pull request at: https://github.com/apache/spark/pull/2216 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4410][SQL] Add support for external sor...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/3268#discussion_r20400481 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala --- @@ -17,6 +17,8 @@ package org.apache.spark.sql.execution +import org.apache.spark.util.collection.ExternalSorter --- End diff -- import order here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4410][SQL] Add support for external sor...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/3268#discussion_r20400473 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala --- @@ -189,6 +191,7 @@ case class TakeOrdered(limit: Int, sortOrder: Seq[SortOrder], child: SparkPlan) /** * :: DeveloperApi :: + * Performs a sort on-heap. --- End diff -- can we document the parameters, e.g. "global" for both Sort and ExternalSort? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3279#issuecomment-63162179 I sent 2 PRs about [SPARK-4421](https://issues.apache.org/jira/browse/SPARK-4421) because the page name are different between Spark 1.2 and Spark 1.1. This is for Spark 1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3280#issuecomment-63162188 I sent 2 PRs about [SPARK-4421](https://issues.apache.org/jira/browse/SPARK-4421) because the page name are different between Spark 1.2 and Spark 1.1. This is for Spark 1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4365: Remove unnecessary filter call on ...
Github user saucam commented on the pull request: https://github.com/apache/spark/pull/3229#issuecomment-63162128 Thanks everyone! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3280#issuecomment-63162109 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/3280 [SPARK-4421] Wrong link in spark-standalone.html Modified the link of building Spark. (backport version of #3279.) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-4421-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3280.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3280 commit 3b4d38d30c71685804fa71ae9dbfaf0068f0e384 Author: Masayoshi TSUZUKI Date: 2014-11-15T01:58:54Z [SPARK-4421] Wrong link in spark-standalone.html Modified the link of building Spark. (backport version of #3279.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4309][SPARK-4407][SQL] Date type suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3178#issuecomment-63161982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23409/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4309][SPARK-4407][SQL] Date type suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3178#issuecomment-63161981 [Test build #23409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23409/consoleFull) for PR 3178 at commit [`6f71d0b`](https://github.com/apache/spark/commit/6f71d0ba9b4b01247c6844105e5d99db062a0948). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63161781 [Test build #23412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23412/consoleFull) for PR 3278 at commit [`cb2301a`](https://github.com/apache/spark/commit/cb2301a893397b7c4c08d3437eca4161a81c9031). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3279#issuecomment-63161717 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4421] Wrong link in spark-standalone.ht...
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/3279 [SPARK-4421] Wrong link in spark-standalone.html Modified the link of building Spark. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-4421 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3279.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3279 commit 56e31c1459044aaacd183ecb82f8ca6dcd040bb7 Author: Masayoshi TSUZUKI Date: 2014-11-15T01:38:02Z Modified the link of building Spark. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3197#discussion_r20400301
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusAPI.scala ---
@@ -140,3 +103,10 @@ private[spark] trait SparkStatusAPI { this:
SparkContext =>
}
}
}
+
+private[spark] object SparkStatusAPI {
--- End diff --
Sure, that's fine; I'll just make both constructors `private[spark]` and
add a note; as long as we've warned users not to call the constructor and
hidden it from the Scaladoc, then I don't think anyone should complain if we
need to change it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3197#discussion_r20400214
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusAPI.scala ---
@@ -140,3 +103,10 @@ private[spark] trait SparkStatusAPI { this:
SparkContext =>
}
}
}
+
+private[spark] object SparkStatusAPI {
--- End diff --
Sorry the main problem I have is that I don't get why we need to protect
the constructor at all. It is not something we expect the users to call. Why
don't you just remove all of these stuff, and add a line in javadoc for the
constructor saying we don't expect users to call this constructor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63161074 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23411/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63161073 [Test build #23411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23411/consoleFull) for PR 3278 at commit [`133bad5`](https://github.com/apache/spark/commit/133bad59c127c32198a09742fe747f8b77cb34d5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3197#discussion_r20400195
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusAPI.scala ---
@@ -140,3 +103,10 @@ private[spark] trait SparkStatusAPI { this:
SparkContext =>
}
}
}
+
+private[spark] object SparkStatusAPI {
--- End diff --
Also, I think that CompanionObject.apply() might be a fairly common idiom;
I think it's used in several of the Scala standard libraries. I don't really
care what we call it, one way or the other, so I can change it if you think
that `apply` is confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4327] [PySpark] Python API for RDD.rand...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3193#issuecomment-63161047 @JoshRosen How to you think of this? The MLlib tests may be blocked by this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3278#issuecomment-63160884 [Test build #23411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23411/consoleFull) for PR 3278 at commit [`133bad5`](https://github.com/apache/spark/commit/133bad59c127c32198a09742fe747f8b77cb34d5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4293][SQL] Make Cast be able to handle ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3150#issuecomment-63160796 [Test build #23408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23408/consoleFull) for PR 3150 at commit [`ba14003`](https://github.com/apache/spark/commit/ba14003fedbc13db8b40b1712070ae1ed44972f8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4293][SQL] Make Cast be able to handle ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3150#issuecomment-63160799 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23408/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Bumping version to 1.3.0-SNAPSHOT.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3277#issuecomment-63160797 [Test build #23410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23410/consoleFull) for PR 3277 at commit [`4ad2f18`](https://github.com/apache/spark/commit/4ad2f18148ff3f50cc68ad19eb18e4afce9651d0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2321] Several progress API improvements...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3197#discussion_r20400108
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusAPI.scala ---
@@ -140,3 +103,10 @@ private[spark] trait SparkStatusAPI { this:
SparkContext =>
}
}
}
+
+private[spark] object SparkStatusAPI {
--- End diff --
Is there a way to make a Scala constructor Java-package-private as opposed
to Scala-package-private, since that can become public from Java's point of
view? I think I originally used this factory pattern for JavaSparkStatusAPI
and just kept the same approach here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4420][SQL] Change nullability of Cast f...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/3278 [SPARK-4420][SQL] Change nullability of Cast from DoubleType/FloatType to DecimalType. This is follow-up of [SPARK-4390](https://issues.apache.org/jira/browse/SPARK-4390) (#3256). You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-4420 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3278.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3278 commit 133bad59c127c32198a09742fe747f8b77cb34d5 Author: Takuya UESHIN Date: 2014-11-15T04:45:42Z Change nullability of Cast from DoubleType/FloatType to DecimalType. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4048] Enhance and extend hadoop-provide...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2982#issuecomment-63160744 Failure seems unrelated. Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Bumping version to 1.3.0-SNAPSHOT.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3277#issuecomment-63160723 Probably a good idea now that branch-1.2 has been cut. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Bumping version to 1.3.0-SNAPSHOT.
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/3277 Bumping version to 1.3.0-SNAPSHOT. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark version-1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3277.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3277 commit 4ad2f18148ff3f50cc68ad19eb18e4afce9651d0 Author: Marcelo Vanzin Date: 2014-11-15T04:48:23Z Bumping version to 1.3.0-SNAPSHOT. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4418][SQL] Remove restriction of binary...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3276#issuecomment-63160638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23407/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4418][SQL] Remove restriction of binary...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3276#issuecomment-63160637 [Test build #23407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23407/consoleFull) for PR 3276 at commit [`95d64ad`](https://github.com/apache/spark/commit/95d64ad0e2a492d03b850294d511eb80e7d72fb2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4309][SPARK-4407][SQL] Date type suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3178#issuecomment-63160120 [Test build #23409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23409/consoleFull) for PR 3178 at commit [`6f71d0b`](https://github.com/apache/spark/commit/6f71d0ba9b4b01247c6844105e5d99db062a0948). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4415] [PySpark] JVM should exit after P...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3274 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4404]SparkSubmitDriverBootstrapper shou...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4415] [PySpark] JVM should exit after P...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3274#discussion_r20399877
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitDriverBootstrapper.scala
---
@@ -139,14 +139,15 @@ private[spark] object SparkSubmitDriverBootstrapper {
// subprocess there already reads directly from our stdin, so we
should avoid spawning a
// thread that contends with the subprocess in reading from System.in.
val isWindows = Utils.isWindows
-val isPySparkShell = sys.env.contains("PYSPARK_SHELL")
+val isSubprocess = sys.env.contains("IS_SUBPROCESS")
if (!isWindows) {
val stdinThread = new RedirectThread(System.in,
process.getOutputStream, "redirect stdin")
stdinThread.start()
- // For the PySpark shell, Spark submit itself runs as a python
subprocess, and so this JVM
- // should terminate on broken pipe, which signals that the parent
process has exited. In
- // Windows, the termination logic for the PySpark shell is handled
in java_gateway.py
- if (isPySparkShell) {
+ // Spark submit (JVM) may can runs as a subprocess, and so this JVM
should terminate on
--- End diff --
can run. I'll fix this when I merge it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4415] [PySpark] JVM should exit after P...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3274#issuecomment-63159445 Ok merging this master 1.2 thanks @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4404]SparkSubmitDriverBootstrapper shou...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3266#issuecomment-63159399 Ok I merge 1.2 master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1977][MLLIB] use immutable BitSet in AL...
Github user aaronlin commented on the pull request: https://github.com/apache/spark/pull/925#issuecomment-63159256 Spark-ec2.py gets the official release of spark, so even if i build spark with the registration of bitset and rating class manually, I cannot use it when deploying ec2 cluster. Can you give me some hints how to cope with the problem while using spark-ec2.py? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4228 SchemaRDD to JSON
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3213#issuecomment-63159199 I think it should be ``` def toJsonRDD(self): rdd = self._jschema_rdd.baseSchemaRDD().toJsonRDD() return RDD(rdd.toJavaRDD(), self.ctx, UTF8Deserializer()) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
