[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836136 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45836147 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45836138 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45836171 @tgravescs @mridulm @kayousterhout I add a commit which submit stage after configured number executor are registered. # submit stage only after successfully reg

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836145 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836204 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15703/ --- If your project is set up for it, you can

[GitHub] spark pull request: [WIP][SPARK-2097][SQL] UDF Support

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1063#issuecomment-45836203 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13689598 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -179,8 +178,17 @@ object Client { System.setProperty("SPARK_YARN_M

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45836829 This happened because in the byte code there was a method as follows `public final java.lang.Object org$apache$spark$rdd$PairRDDFunctions$$createZero$1(byte[], sca

[GitHub] spark pull request: [SPARK-2086] Improve output of toDebugString t...

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1039#issuecomment-45836980 Hey @GregOwen sorry to keep changing the spec on you, but I thought a bit more and I think for more complicated graphs we might want to change things a bit. I looked at

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/1064 [SPARK-2124] Move aggregation into shuffle implementations This PR is a sub-task of SPARK-2044 to move the execution of aggregation into shuffle implementations. You can merge this pull request

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45837345 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45837351 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Cleanup on Connection and ConnectionManager

2014-06-12 Thread hsaputra
Github user hsaputra commented on the pull request: https://github.com/apache/spark/pull/1060#issuecomment-45837768 Thx @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...

2014-06-12 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/999#discussion_r13690133 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala --- @@ -0,0 +1,485 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-12 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1009#discussion_r13690254 --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala --- @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1009#discussion_r13690317 --- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala --- @@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag]( par

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread jkthompson
GitHub user jkthompson opened a pull request: https://github.com/apache/spark/pull/1065 fixed typo in docstring for min() Hi, I found this typo while learning spark and thought I'd do a pull request. You can merge this pull request into a Git repository by running: $ git pul

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45838723 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45838746 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45839089 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45839097 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690774 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -18,6 +18,7 @@ package org.apache.spark.deploy.yarn

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690778 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -24,6 +24,7 @@ import java.nio.ByteBuffer import scala.collection

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690782 --- Diff: yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -188,9 +187,18 @@ object Client { // see Client#setupLaunchEnv().

[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/490#discussion_r13690775 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala --- @@ -138,11 +139,10 @@ class ClientArguments(val args: Array[String]

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45840353 Sorry to jump in late on this, but I think spark.executor.minRegisteredNum sounds like an executor property, when this is a property of the driver. --- If your project is s

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691081 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +402,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45840607 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691100 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +402,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45840608 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15704/ --- If your project

[GitHub] spark pull request: [SPARK-1964][SQL] Add timestamp to HiveMetasto...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1061#issuecomment-45840609 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15702/ --- If your project

[GitHub] spark pull request: [Minor] Fix style, formatting and naming in Bl...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1058#issuecomment-45840606 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691144 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +401,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691153 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -394,20 +401,22 @@ abstract class RDD[T: ClassTag]( return new Array[T](0)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691305 --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691375 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -498,55 +501,56 @@ class RDDSuite extends FunSuite with SharedSparkContext { }

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691382 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -498,55 +501,56 @@ class RDDSuite extends FunSuite with SharedSparkContext { }

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691388 --- Diff: core/src/test/scala/org/apache/spark/util/random/SamplingUtilsSuite.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691398 --- Diff: core/src/test/scala/org/apache/spark/util/random/SamplingUtilsSuite.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841426 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841433 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691458 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0 t

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691473 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0 t

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691499 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0 t

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691512 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691490 --- Diff: python/pyspark/rdd.py --- @@ -365,27 +366,25 @@ def takeSample(self, withReplacement, num, seed=None): fraction = 0.0 t

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691520 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: Mend StreamingContext stop() followed by await...

2014-06-12 Thread lallea
Github user lallea commented on the pull request: https://github.com/apache/spark/pull/1001#issuecomment-45841618 You beat me to Jira. Thanks for the merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691539 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691536 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-1939 Refactor takeSample method in RDD t...

2014-06-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/916#discussion_r13691546 --- Diff: python/pyspark/rdd.py --- @@ -400,6 +399,18 @@ def takeSample(self, withReplacement, num, seed=None): sampler.shuffle(samples)

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45841770 I just tried extracting createZero out and making it private, since the method is used in two places now. But making it private did not help, the mima warning is still

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/1066 Master supports pluggable clock Convenient for testing, especially in timeout scenario. You can merge this pull request into a Git repository by running: $ git pull https://github.com/CrazyJvm

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45842307 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45842316 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45843614 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843613 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843616 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15705/ --- If your project

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1065#issuecomment-45843617 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15706/ --- If your project

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-45844273 Thanks @sryza How about spark.scheduler.minRegisteredExecutors? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread CrazyJvm
Github user CrazyJvm commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13692635 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -225,6 +232,17 @@ class CoarseGrainedSchedulerBac

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45845324 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45845326 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15707/ --- If your project

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45845628 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45845630 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15708/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1946] Submit stage after (configured nu...

2014-06-12 Thread li-zhihui
Github user li-zhihui commented on a diff in the pull request: https://github.com/apache/spark/pull/900#discussion_r13692995 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -225,6 +232,17 @@ class CoarseGrainedSchedulerBa

[GitHub] spark pull request: SPARK-938 - Openstack Swift object storage sup...

2014-06-12 Thread gilv
Github user gilv commented on the pull request: https://github.com/apache/spark/pull/1010#issuecomment-45846831 Removed all openstack dependencies from pom.xml Added an example of Java job with parameters passed to the Swift driver using SparkContext.hadoopConfiguration --- If yo

[GitHub] spark pull request: SPARK-1668: Add implicit preference as an opti...

2014-06-12 Thread coderh
Github user coderh commented on the pull request: https://github.com/apache/spark/pull/597#issuecomment-45847264 Here is the values I have tried: seed is set to 42 in & out means in sample (training set) out-of-sample (test set) # #factor = 12, lamda = 1, alpha = 1

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
Github user CrazyJvm closed the pull request at: https://github.com/apache/spark/pull/1066 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread rahulsinghaliitd
GitHub user rahulsinghaliitd opened a pull request: https://github.com/apache/spark/pull/1067 SPARK-2127: Use application specific folders to dump metrics via CsvSink Generate an unique app name which is used to create events and metric folders. You can merge this pull request into

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1067#issuecomment-45878079 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/894#discussion_r13700124 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala --- @@ -88,6 +88,18 @@ private[yarn] class YarnAllocationHandler(

[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1002#issuecomment-45889538 Yeah the Securitymanager is adding both to handle the case the containers run as one user but is accessing HDFS as whoever specified in SPARK_USER, it also came up in t

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
GitHub user CrazyJvm reopened a pull request: https://github.com/apache/spark/pull/1066 Master supports pluggable clock Convenient for testing, especially in timeout scenario. You can merge this pull request into a Git repository by running: $ git pull https://github.com/CrazyJ

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45890066 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45890081 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1002#issuecomment-45890230 also note that yarn does set the SPARK_USER to UGI in ClientBase env("SPARK_USER") = UserGroupInformation.getCurrentUser().getShortUserName() --- If your proj

[GitHub] spark pull request: SPARK-1937: fix issue with task locality

2014-06-12 Thread lirui-intel
Github user lirui-intel commented on the pull request: https://github.com/apache/spark/pull/892#issuecomment-45890408 Thanks @mridulm , I've updated the patch accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13700626 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property: Properties, va

[GitHub] spark pull request: [WIP] [SPARK-2051]In yarn.ClientBase spark.yar...

2014-06-12 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45891513 @witgo the [WIP] tag tells me you are still working on this, looking for early comments on the approach, and its not really fully done and ready for review, is that the

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread rahulsinghaliitd
Github user rahulsinghaliitd commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13701712 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property: Properti

[GitHub] spark pull request: [WIP] [SPARK-2051]In yarn.ClientBase spark.yar...

2014-06-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/969#issuecomment-45893165 @tgravescs Still have some questions not discussed clearly. @vanzin doesn't like to do so. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2051]In yarn.ClientBase spark.yarn.dist...

2014-06-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/969#discussion_r13702112 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -220,10 +220,21 @@ trait ClientBase extends Logging { }

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45894603 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45894604 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15709/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/1068 [SPARK-2118] spark class should complaing if tools jar is missing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ScrapCodes/spark-1 SPARK-2

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1068#issuecomment-45895480 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1068#issuecomment-45895465 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-2127: Use application specific folders t...

2014-06-12 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/1067#discussion_r13703371 --- Diff: core/src/main/scala/org/apache/spark/metrics/sink/CsvSink.scala --- @@ -53,11 +53,14 @@ private[spark] class CsvSink(val property: Properties, va

[GitHub] spark pull request: SPARK-1668: Add implicit preference as an opti...

2014-06-12 Thread coderh
Github user coderh commented on the pull request: https://github.com/apache/spark/pull/597#issuecomment-45898782 Ok, I have found the error in my metric. ``` val itemFactors = model.productFeatures.collect() ``` This line is for creating a item-factor matrix, the problem

[GitHub] spark pull request: Master supports pluggable clock

2014-06-12 Thread CrazyJvm
Github user CrazyJvm commented on the pull request: https://github.com/apache/spark/pull/1066#issuecomment-45899433 I cannot figure it out why build failed here since everything is OK on my Mac. --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1068#issuecomment-45901402 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1068#issuecomment-45901403 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15710/ --- If your project

[GitHub] spark pull request: [SPARK-2118] spark class should complaing if t...

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1068#issuecomment-45903552 @ScrapCodes I don't think this is what we want - because the tools jar is not required in the general case. We should only check this if the main class is `GenerateMIMAI

[GitHub] spark pull request: fixed typo in docstring for min()

2014-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1065 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45904462 @scrapcodes. Ah I see - there was a name collision with an inner function that was used elsewhere. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45904624 Thanks Sandy - I'm merging this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/705 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [SPARK-1930] The Container is running beyond p...

2014-06-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/894#discussion_r13706707 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala --- @@ -88,6 +88,18 @@ private[yarn] class YarnAllocationHandler(

  1   2   3   4   5   >