[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured ra...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13688559 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -431,6 +431,16 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45833299 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2109] Setting SPARK_MEM for bin/pyspark...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1050#issuecomment-45833300 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15701/ --- If your

[GitHub] spark pull request: [SPARK-2014] Make PySpark store RDDs in MEMORY...

2014-06-12 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1051#issuecomment-45833637 I agree !, perhaps part of Storage Levels ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured ra...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45833678 @tgravescs @kayousterhout The PR only work in standalone mode now. But it provide a abstract method isReady() in SchedulerBackend.scala for all backend implementations.

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-12 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/1009#discussion_r13688708 --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala --- @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P : Product2[K, V] : ClassTag](

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1009#discussion_r13688732 --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala --- @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P : Product2[K, V] : ClassTag](

[GitHub] spark pull request: Cleanup on Connection and ConnectionManager

2014-06-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1060#issuecomment-45833767 LGTM. Merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Cleanup on Connection and ConnectionManager

2014-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1060 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45834509 @scrapcodes - do you have any idea on this one? This doesn't make much sense to me at all! @sryza for now you can add an exclude in `project/MimaExcludes` this is

[GitHub] spark pull request: SPARK-2035: Store call stack for stages, displ...

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/981#issuecomment-45834598 Ah cool - great to hear --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread marmbrus
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/1061 [SPARK-1964][SQL] Add timestamp to HiveMetastoreTypes.toMetastoreType You can merge this pull request into a Git repository by running: $ git pull https://github.com/marmbrus/spark timestamp

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45834995 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45835003 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...

2014-06-12 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45835239 @coderxiang Could you merge the current master? This is not mergeable due to a recent commit. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: Made splits deprecated in JavaRDDLike

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1062#issuecomment-45835262 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/490#issuecomment-45835283 @mengxr Do you think it's in good shape now? This is the only issue blocking us using vanilla spark. Thanks. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: Made splits deprecated in JavaRDDLike

2014-06-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1062#discussion_r13689356 --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala --- @@ -43,8 +43,11 @@ trait JavaRDDLike[T, This : JavaRDDLike[T, This]] extends

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread marmbrus
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/1063 [WIP][SPARK-2097][SQL] UDF Support Remaining: - [ ] Python Support - [ ] Test cases - [ ] Input coercion You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836136 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45836147 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45836138 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45836171 @tgravescs @mridulm @kayousterhout I add a commit which submit stage after configured number executor are registered. # submit stage only after successfully

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836204 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15703/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13689598 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -179,8 +178,17 @@ object Client {

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45836829 This happened because in the byte code there was a method as follows `public final java.lang.Object org$apache$spark$rdd$PairRDDFunctions$$createZero$1(byte[],

[GitHub] spark pull request: [SPARK-2086] Improve output of toDebugString t...

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1039#issuecomment-45836980 Hey @GregOwen sorry to keep changing the spec on you, but I thought a bit more and I think for more complicated graphs we might want to change things a bit. I looked at

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/1064 [SPARK-2124] Move aggregation into shuffle implementations This PR is a sub-task of SPARK-2044 to move the execution of aggregation into shuffle implementations. You can merge this pull request

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45837351 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Cleanup on Connection and ConnectionManager

2014-06-12 Thread hsaputra
Github user hsaputra commented on the pull request: https://github.com/apache/spark/pull/1060#issuecomment-45837768 Thx @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...

2014-06-12 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/999#discussion_r13690133 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala --- @@ -0,0 +1,485 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-12 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1009#discussion_r13690254 --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala --- @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P : Product2[K, V] : ClassTag](

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread jkthompson
GitHub user jkthompson opened a pull request: https://github.com/apache/spark/pull/1065 fixed typo in docstring for min() Hi, I found this typo while learning spark and thought I'd do a pull request. You can merge this pull request into a Git repository by running: $ git

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45838723 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45838746 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45839089 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45839097 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690774 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -18,6 +18,7 @@ package org.apache.spark.deploy.yarn

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690778 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -24,6 +24,7 @@ import java.nio.ByteBuffer import

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690782 --- Diff: yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -188,9 +187,18 @@ object Client { // see

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690775 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -138,11 +139,10 @@ class ClientArguments(val args:

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45840353 Sorry to jump in late on this, but I think spark.executor.minRegisteredNum sounds like an executor property, when this is a property of the driver. --- If your project is

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691081 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +402,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45840607 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691100 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +402,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45840608 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15704/ --- If your

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45840609 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15702/ --- If your

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45840606 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691144 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +401,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691153 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +401,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691305 --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691375 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -498,55 +501,56 @@ class RDDSuite extends FunSuite with SharedSparkContext { }

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691382 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -498,55 +501,56 @@ class RDDSuite extends FunSuite with SharedSparkContext { }

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691388 --- Diff: core/src/test/scala/org/apache/spark/util/random/SamplingUtilsSuite.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691398 --- Diff: core/src/test/scala/org/apache/spark/util/random/SamplingUtilsSuite.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841426 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841433 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691458 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691473 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691499 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691512 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691490 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691520 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: Mend StreamingContext stop() followed by await...

2014-06-12 Thread lallea
Github user lallea commented on the pull request: https://github.com/apache/spark/pull/1001#issuecomment-45841618 You beat me to Jira. Thanks for the merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691539 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691536 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691546 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841770 I just tried extracting createZero out and making it private, since the method is used in two places now. But making it private did not help, the mima warning is still

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/1066 Master supports pluggable clock Convenient for testing, especially in timeout scenario. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45842307 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45842316 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45843614 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843613 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843616 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15705/ --- If your

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45843617 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15706/ --- If your

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45844273 Thanks @sryza How about spark.scheduler.minRegisteredExecutors? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread CrazyJvm
Github user CrazyJvm commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13692635 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -225,6 +232,17 @@ class

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45845324 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45845326 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15707/ --- If your project

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45845628 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45845630 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15708/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13692995 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -225,6 +232,17 @@ class

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
Github user CrazyJvm closed the pull request at: https://github.com/apache/spark/pull/1066 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread rahulsinghaliitd
GitHub user rahulsinghaliitd opened a pull request: https://github.com/apache/spark/pull/1067 SPARK-2127: Use application specific folders to dump metrics via CsvSink Generate an unique app name which is used to create events and metric folders. You can merge this pull request

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1067#issuecomment-45878079 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/894#discussion_r13700124 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala --- @@ -88,6 +88,18 @@ private[yarn] class YarnAllocationHandler(

[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1002#issuecomment-45889538 Yeah the Securitymanager is adding both to handle the case the containers run as one user but is accessing HDFS as whoever specified in SPARK_USER, it also came up in

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
GitHub user CrazyJvm reopened a pull request: https://github.com/apache/spark/pull/1066 Master supports pluggable clock Convenient for testing, especially in timeout scenario. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45890066 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45890081 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1002#issuecomment-45890230 also note that yarn does set the SPARK_USER to UGI in ClientBase env(SPARK_USER) = UserGroupInformation.getCurrentUser().getShortUserName() --- If your

[GitHub] spark pull request: SPARK-1937: fix issue with task locality

2014-06-12 Thread lirui-intel
Github user lirui-intel commented on the pull request: https://github.com/apache/spark/pull/892#issuecomment-45890408 Thanks @mridulm , I've updated the patch accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13700626 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property: Properties,

[GitHub] spark pull request: [WIP] [SPARK-2051]In yarn.ClientBase spark.yar...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45891513 @witgo the [WIP] tag tells me you are still working on this, looking for early comments on the approach, and its not really fully done and ready for review, is that the

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread rahulsinghaliitd
Github user rahulsinghaliitd commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13701712 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property:

[GitHub] spark pull request: [WIP] [SPARK-2051]In yarn.ClientBase spark.yar...

2014-06-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45893165 @tgravescs Still have some questions not discussed clearly. @vanzin doesn't like to do so. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13702112 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -220,10 +220,21 @@ trait ClientBase extends Logging { }

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45894603 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45894604 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15709/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/1068 [SPARK-2118] spark class should complaing if tools jar is missing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ScrapCodes/spark-1

  1   2   3   4   5   >