[GitHub] spark pull request: [BUILD] Fix for false positives reported by mi...

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54783013 @rxin The test failed as expected. I will fix it once you confirm the code change. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [BUILD] Fix for false positives reported by mi...

2014-09-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54783094 LGTM (although I don't really understand this part very well). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2274#issuecomment-54785322 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19964/consoleFull) for PR 2274 at commit [`c04b447`](https://github.com/a

[GitHub] spark pull request: [HOTFIX] A left over version change. It should...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2317#issuecomment-54785885 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19966/consoleFull) for PR 2317 at commit [`b6472d4`](https://github.com/ap

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2274#issuecomment-54785910 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19967/consoleFull) for PR 2274 at commit [`4a5332a`](https://github.com/ap

[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2014#issuecomment-54790978 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19968/consoleFull) for PR 2014 at commit [`91c921f`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54790980 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19969/consoleFull) for PR 1106 at commit [`2ca3091`](https://github.com/ap

[GitHub] spark pull request: [HOTFIX] A left over version change. It should...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2317#issuecomment-54791055 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19966/consoleFull) for PR 2317 at commit [`b6472d4`](https://github.com/a

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2274#issuecomment-54791106 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19967/consoleFull) for PR 2274 at commit [`4a5332a`](https://github.com/a

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54791541 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19969/consoleFull) for PR 1106 at commit [`2ca3091`](https://github.com/a

[GitHub] spark pull request: [SPARK-3437] Support crossbuilding in maven.

2014-09-08 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/2318 [SPARK-3437] Support crossbuilding in maven. This patch get rid of _2.10 in the artifact ids and adapts the install plugin to include it instead. This will make it easy to cross build for sca

[GitHub] spark pull request: [Docs] actorStream storageLevel default is MEM...

2014-09-08 Thread melrief
GitHub user melrief opened a pull request: https://github.com/apache/spark/pull/2319 [Docs] actorStream storageLevel default is MEMORY_AND_DISK_SER_2 Comment of the storageLevel param of actorStream says that it defaults to memory-only while the default is MEMORY_AND_DISK_SER_2. Y

[GitHub] spark pull request: [HOTFIX] A left over version change. It should...

2014-09-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2317 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [HOTFIX] A left over version change. It should...

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2317#issuecomment-54792700 Yay merged it ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: [SPARK-3437] Support crossbuilding in maven.

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2318#issuecomment-54792890 @pwendell If this is okay to do, then there is no need of a special maven plugin for this task. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54794153 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [Docs] actorStream storageLevel default is MEM...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2319#issuecomment-54796569 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2318#issuecomment-54796779 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19970/consoleFull) for PR 2318 at commit [`6c8f9f6`](https://github.com/ap

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54796777 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19972/consoleFull) for PR 2229 at commit [`d4ad660`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2318#issuecomment-54796972 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19970/consoleFull) for PR 2318 at commit [`6c8f9f6`](https://github.com/a

[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2014#issuecomment-54797341 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19968/consoleFull) for PR 2014 at commit [`91c921f`](https://github.com/a

[GitHub] spark pull request: [SPARK-3433][BUILD] Fix for Mima false-positiv...

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54797397 I will let @JoshRosen take look too, since he reported this issue as well. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54800273 This patch is like most prone to merge conflicts, would be good to merge it soon. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-54802000 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19974/consoleFull) for PR 2311 at commit [`2ebe62e`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2318#issuecomment-54801998 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19973/consoleFull) for PR 2318 at commit [`ed315ef`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3433][BUILD] Fix for Mima false-positiv...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54802027 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19975/consoleFull) for PR 2285 at commit [`35b6c71`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3012] Standardized Distance Functions b...

2014-09-08 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1964#issuecomment-54802662 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [SPARK-927] detect numpy at time of use

2014-09-08 Thread mattf
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2313#issuecomment-54804358 >> What's the behavior if some slaves have numpy but others do not have? > > Yeah, this was my original concern. Do both implementations give equal results, or will

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/2318#discussion_r17232343 --- Diff: bagel/pom.xml --- @@ -60,6 +60,53 @@ target/scala-${scala.binary.version}/test-classes +org.apache.maven.p

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/2318#discussion_r17232335 --- Diff: bagel/pom.xml --- @@ -60,6 +60,53 @@ target/scala-${scala.binary.version}/test-classes +org.apache.maven.p

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/2318#discussion_r17232323 --- Diff: assembly/pom.xml --- @@ -26,7 +26,7 @@ org.apache.spark - spark-assembly_2.10 + spark-assembly --- End diff --

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54805222 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19972/consoleFull) for PR 2229 at commit [`d4ad660`](https://github.com/a

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-54806033 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19974/consoleFull) for PR 2311 at commit [`2ebe62e`](https://github.com/a

[GitHub] spark pull request: [SPARK-3433][BUILD] Fix for Mima false-positiv...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54806878 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19975/consoleFull) for PR 2285 at commit [`35b6c71`](https://github.com/a

[GitHub] spark pull request: [SPARK-3012] Standardized Distance Functions b...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1964#issuecomment-54806883 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19976/consoleFull) for PR 1964 at commit [`a44c30c`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3425] do not set MaxPermSize for OpenJD...

2014-09-08 Thread mattf
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2301#issuecomment-54806924 > This patch fails unit tests. failures appear to be from mima and hive, unrelated to this patch --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Provide a default PYSPARK_PYTHON for python/ru...

2014-09-08 Thread mattf
GitHub user mattf reopened a pull request: https://github.com/apache/spark/pull/2300 Provide a default PYSPARK_PYTHON for python/run_tests Without this the version of python used in the test is not recorded. The error is, Testing with Python version: ./run-test

[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...

2014-09-08 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2087#issuecomment-54807921 FWIW, I think mostly-accurate metrics are much better than no metrics in this case. The read/write bytes are very useful from Hadoop FSes, and Hadoop <2.5 is still very

[GitHub] spark pull request: [SPARK-3425] do not set MaxPermSize for OpenJD...

2014-09-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/2301#discussion_r17233969 --- Diff: bin/spark-class --- @@ -105,7 +105,7 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 2>&1 | sed 's/java version "\(.

[GitHub] spark pull request: [SPARK-3425] do not set MaxPermSize for OpenJD...

2014-09-08 Thread mattf
Github user mattf commented on a diff in the pull request: https://github.com/apache/spark/pull/2301#discussion_r17234099 --- Diff: bin/spark-class --- @@ -105,7 +105,7 @@ else exit 1 fi fi -JAVA_VERSION=$($RUNNER -version 2>&1 | sed 's/java version "\(.*\)

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2318#issuecomment-54810834 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19973/consoleFull) for PR 2318 at commit [`ed315ef`](https://github.com/a

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread bbejeck
Github user bbejeck commented on the pull request: https://github.com/apache/spark/pull/2309#issuecomment-54811532 Can an admin take a look and verify this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-54812961 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-3012] Standardized Distance Functions b...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1964#issuecomment-54814790 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19976/consoleFull) for PR 1964 at commit [`a44c30c`](https://github.com/a

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-54819294 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19979/consoleFull) for PR 2311 at commit [`2ebe62e`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54819302 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19980/consoleFull) for PR 1106 at commit [`2835929`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54829230 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19980/consoleFull) for PR 1106 at commit [`2835929`](https://github.com/a

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2311#issuecomment-54830677 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19979/consoleFull) for PR 2311 at commit [`2ebe62e`](https://github.com/a

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2309#issuecomment-54831765 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [EC2] don't duplicate default values

2014-09-08 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54832517 @shivaram So then post the 1.1.0 release we should bump the default Spark version in `master`, right? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Added support for accessing secured HDFS

2014-09-08 Thread huozhanfeng
GitHub user huozhanfeng opened a pull request: https://github.com/apache/spark/pull/2320 Added support for accessing secured HDFS JIRA Issue:https://issues.apache.org/jira/browse/SPARK-3438 Reading data from secure HDFS into spark is a usefull feature. You can merge this pu

[GitHub] spark pull request: [EC2] don't duplicate default values

2014-09-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54839162 Yup, we should make it 1.1.0 after Patrick uploads the release binaries. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: Added support for accessing secured HDFS

2014-09-08 Thread huozhanfeng
Github user huozhanfeng commented on the pull request: https://github.com/apache/spark/pull/265#issuecomment-54840248 @pwendell @tgravescs I have done some improvement for it and have created a new PR base on the newest master, you can work on it . PR:https://github.com/apache

[GitHub] spark pull request: Added support for accessing secured HDFS

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2320#issuecomment-54840534 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2309#issuecomment-54840875 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19981/consoleFull) for PR 2309 at commit [`51cf915`](https://github.com/ap

[GitHub] spark pull request: Added support for accessing secured HDFS

2014-09-08 Thread huozhanfeng
Github user huozhanfeng commented on the pull request: https://github.com/apache/spark/pull/2320#issuecomment-54841227 This PR is base on https://github.com/apache/spark/pull/2320 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54841382 Seems like there are tests failed in streaming module, @andrewor14 could you check it? --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark pull request: [SPARK-2140] Updating heap memory calculation ...

2014-09-08 Thread copester
Github user copester commented on the pull request: https://github.com/apache/spark/pull/2253#issuecomment-54848702 Is this merge blocking on Jenkins? I can provide a JUnit test report if you want? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54848841 Sounds good to me. Were you thinking of adding these instructions under the _Accessing Data in S3_ section? --- If your project is set up for it, you can reply to thi

[GitHub] spark pull request: [SPARK-3418] Sparse Matrix support (CCS) and a...

2014-09-08 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2294#issuecomment-54849222 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3086] [SPARK-3043] [SPARK-3156] [mllib]...

2014-09-08 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2125#issuecomment-54849996 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-3418] Sparse Matrix support (CCS) and a...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2294#issuecomment-54850135 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19982/consoleFull) for PR 2294 at commit [`848406c`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3086] [SPARK-3043] [SPARK-3156] [mllib]...

2014-09-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2125 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54850308 That's what I was thinking at first, but then I noticed under ["Before You Start"](https://github.com/apache/spark/blob/master/docs/ec2-scripts.md#before-you-start) ther

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17250396 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -295,28 +295,34 @@ private[spark] class Master( val execOption

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2309#issuecomment-54850299 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19981/consoleFull) for PR 2309 at commit [`51cf915`](https://github.com/a

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17250489 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -295,28 +295,34 @@ private[spark] class Master( val execOption

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17250527 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -295,28 +295,34 @@ private[spark] class Master( val execOption

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17250993 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -234,7 +234,7 @@ private[spark] class Worker( try {

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17251037 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -355,11 +355,20 @@ private[spark] class ApplicationMaster(arg

[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2087#issuecomment-54851789 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17250755 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -295,28 +295,34 @@ private[spark] class Master( val execOption

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54852214 I think that `boto` will read its credentials from those environment variables [or through configuration files](http://boto.readthedocs.org/en/latest/boto_config_tut.ht

[GitHub] spark pull request: [SPARK-3293] yarn's web show "SUCCEEDED" when ...

2014-09-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2311#discussion_r17251254 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala --- @@ -369,7 +378,17 @@ private[spark] class ApplicationMaster(args

[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...

2014-09-08 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-54852242 this is ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...

2014-09-08 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-54852250 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17251334 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Reco

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17251378 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Reco

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2227#issuecomment-54852535 For those watching this PR, the new one is at #2309. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...

2014-09-08 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2087#issuecomment-54852759 Just to make sure it's clear, the issue isn't only that we can be a few bytes off when we're reading outside of split boundaries, but that it'll look like we read the full

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17251541 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Reco

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17251647 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Reco

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1106#issuecomment-54853445 This still doesn't deal with the fact that we can over-subscribe a worker by scheduling more drivers on it than the worker's resources allow. Since this wasn't dealt w

[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...

2014-09-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2309#issuecomment-54853820 Still think the fix should be made in the Utils class. This is not the only place where this could cause trouble. --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-08 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2274#issuecomment-54854337 LGTM, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread danosipov
Github user danosipov commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54854888 @JoshRosen Yes, the credentials could be supplied in boto specific config files, but if they're not there the spark_ec2 script will look in the env variables, and error

[GitHub] spark pull request: [SPARK-546] Add full outer join to RDD and DSt...

2014-09-08 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/1395#issuecomment-54854991 @staple Could you fix the PEP8 style? ./python/pyspark/join.py:98:1: E302 expected 2 blank lines, found 1 --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54855255 Yup. I played around with this locally with spaces and it also works for me. Merging this into master. Thanks. --- If your project is set up for it, you can reply to

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2229 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread danosipov
Github user danosipov commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54855613 Actually it looks it can work for all cases by getting the credentials out of boto: ``` conn.aws_access_key_id conn.aws_secret_access_key ``` Let me

[GitHub] spark pull request: [SPARK-546] Add full outer join to RDD and DSt...

2014-09-08 Thread staple
Github user staple commented on the pull request: https://github.com/apache/spark/pull/1395#issuecomment-54856229 Sure, I fixed the new python style issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17253072 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Rec

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread danosipov
Github user danosipov commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54857174 Looks like I screwed up a rebase. Should I open a new PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

2014-09-08 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/1106#discussion_r17253575 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -481,13 +481,23 @@ private[spark] class Master( if (state != Rec

[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2087#issuecomment-54859158 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19983/consoleFull) for PR 2087 at commit [`0034292`](https://github.com/ap

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54859224 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19985/consoleFull) for PR 1120 at commit [`07c599c`](https://github.com/ap

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-09-08 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/1360#discussion_r17254222 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -295,28 +295,34 @@ private[spark] class Master( val execOption

[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-54859169 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19984/consoleFull) for PR 1270 at commit [`1843f73`](https://github.com/ap

[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...

2014-09-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54859287 You could open a new PR, but this one is probably salvageable through some Git magic. One thing you could try is to [undo your rebase using git-reflog](http://stackove

[GitHub] spark pull request: [SPARK-3418] Sparse Matrix support (CCS) and a...

2014-09-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2294#issuecomment-54859801 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19982/consoleFull) for PR 2294 at commit [`848406c`](https://github.com/a

  1   2   3   4   >