[GitHub] spark pull request: SPARK-4159 [CORE] [WIP] Maven build doesn't ru...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3651#issuecomment-66416040 [Test build #24299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24299/consoleFull) for PR 3651 at commit [`125b0b6`](https://githu

[GitHub] spark pull request: [SPARK-4772] Clear local copies of accumulator...

2014-12-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4772] Clear local copies of accumulator...

2014-12-09 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3570#issuecomment-66415593 Thanks for updating the description. This looks good to me, so I'm going to merge this into `master`, `branch-1.0`, and `branch-1.1` (and I'll tag it for a post-releas

[GitHub] spark pull request: [SPARK-4789] [mllib] Standardize ML Prediction...

2014-12-09 Thread Lewuathe
Github user Lewuathe commented on the pull request: https://github.com/apache/spark/pull/3637#issuecomment-66415498 My initial question was based on the viewpoint of developer api. Simple api seems to restrict the possibility of implementation of new algorithm. As @shivaram mention

[GitHub] spark pull request: Add hadoop-2.5 profile with upgraded jets3t

2014-12-09 Thread ZhangBanger
Github user ZhangBanger commented on the pull request: https://github.com/apache/spark/pull/3654#issuecomment-66415382 Thanks for the response! I'll try out the ```hadoop-2.4``` profile again. For ```jets3t```, I'll take another look to see if it's Spark or Hadoop that needs to do som

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3659 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66415145 I'm merging this into `master` and `branch-1.2`. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-4233] [SQL] WIP:Simplify the UDAF API (...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3247#issuecomment-66415137 [Test build #24297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24297/consoleFull) for PR 3247 at commit [`a9c1544`](https://gith

[GitHub] spark pull request: [SPARK-4233] [SQL] WIP:Simplify the UDAF API (...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3247#issuecomment-66415146 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66414901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66414896 [Test build #24296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24296/consoleFull) for PR 3660 at commit [`fab8658`](https://gith

[GitHub] spark pull request: spark-submit with accept multiple properties-f...

2014-12-09 Thread lvsoft
Github user lvsoft commented on the pull request: https://github.com/apache/spark/pull/3490#issuecomment-66414770 Well, I can't understand what's the complexity of this PR. I've reviewed the SPARK-3779 marked as related and didn't find something related to this patch. And, this pa

[GitHub] spark pull request: [SPARK-4233] [SQL] WIP:Simplify the UDAF API (...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3247#issuecomment-66414361 [Test build #24297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24297/consoleFull) for PR 3247 at commit [`a9c1544`](https://githu

[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66414168 [Test build #24296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24296/consoleFull) for PR 3660 at commit [`fab8658`](https://githu

[GitHub] spark pull request: Fix the initialization issue of 'codegenEnable...

2014-12-09 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3660 Fix the initialization issue of 'codegenEnabled' The problem is `codegenEnabled` is `val`, but it uses a `val` `sqlContext`, which can be override by subclasses. Here is a simple example to show thi

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2014-12-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3505#issuecomment-66411889 Let me explain - Spark SQL is more than SQL. It is SQL + a dsl that will be improved over time. I personally believe over time, majority of Spark users will interact directl

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2014-12-09 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3505#issuecomment-66411639 > the problem is that sparkconf is immutable once created - so in order to toggle this on and off, a user would have to restart Spark. I added this configuration

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2014-12-09 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3505#issuecomment-66411392 > Maybe a better place to do this is in SparkSQL? It depends on if this is a fundamental feature for Spark Core. IMO, I think it's better to have a skewed join in

[GitHub] spark pull request: [SPARK-4797] Replace breezeSquaredDistance

2014-12-09 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/3643#issuecomment-66411144 Thanks. I add the consideration for different cases of SparseVector and DenseVector. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2014-12-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3505#issuecomment-66410973 the problem is that sparkconf is immutable once created - so in order to toggle this on and off, a user would have to restart Spark. Maybe a better place to do this is in Sp

[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join

2014-12-09 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3505#issuecomment-66410552 ping @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: [SPARK-4707][STREAMING] Reliable Kafka Receive...

2014-12-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3655#discussion_r21586197 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala --- @@ -201,12 +201,31 @@ class ReliableKafkaReceiver[

[GitHub] spark pull request: Add hadoop-2.5 profile with upgraded jets3t

2014-12-09 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3654#issuecomment-66409892 This has been discussed a few times - the `hadoop-2.4` profile covers 2.4+. You don't need a new profile. I think the goal is to match `jets3t` distributed with Hadoop. It

[GitHub] spark pull request: SPARK-4159 [CORE] [WIP] Maven build doesn't ru...

2014-12-09 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3651#discussion_r21585931 --- Diff: pom.xml --- @@ -941,19 +950,38 @@ true + org.apache.maven.plug

[GitHub] spark pull request: spark-submit with accept multiple properties-f...

2014-12-09 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3490#issuecomment-66408350 As Patrick said, this will make confiugration more complex than more elegant. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-4795][Core] Redesign the "primitive typ...

2014-12-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/3642#discussion_r21585352 --- Diff: graphx/src/test/scala/org/apache/spark/graphx/lib/ShortestPathsSuite.scala --- @@ -40,7 +40,7 @@ class ShortestPathsSuite extends FunSuite with Loc

[GitHub] spark pull request: [SPARK-4741] Do not destroy FileInputStream an...

2014-12-09 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/3600#issuecomment-66407791 -1 This is broken change for multiple reasons - finalize of out of scope variable can trigger close of underlying fd, potential state issue with vars not being null when

[GitHub] spark pull request: spark-submit with accept multiple properties-f...

2014-12-09 Thread lvsoft
Github user lvsoft commented on the pull request: https://github.com/apache/spark/pull/3490#issuecomment-66405387 Well, that's called separated property files, not *common* properties. It'll be hard to adjust common properties and easy to make mistakes. Delete tmp files is

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66405196 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66405191 [Test build #24295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24295/consoleFull) for PR 3659 at commit [`bd72899`](https://gith

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

2014-12-09 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-66404905 Use spark.yarn.am.* and describe the scope of configuration items in docs is better. In cluster mode we should ignore the unused configs, probably also add a warn

[GitHub] spark pull request: SPARK-3779. yarn spark.yarn.applicationMaster....

2014-12-09 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3471#issuecomment-66404718 @andrewor14, makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: spark-submit with accept multiple properties-f...

2014-12-09 Thread WangTaoTheTonic
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3490#issuecomment-66404521 In your case, why don't just add common properties into private config and set a seperate propertiy file for each workload? Why would the tmp conf file be dele

[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.

2014-12-09 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66404298 BTW, credit where credit is due, I got this idea from @arahuja --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as we

[GitHub] spark pull request: spark-submit with accept multiple properties-f...

2014-12-09 Thread lvsoft
Github user lvsoft commented on the pull request: https://github.com/apache/spark/pull/3490#issuecomment-66404194 Sorry for late reply. I'll explain the use cases for multiple properties files. Currently I'm working on a benchmark utility for spark. It'll be nature to adjust

[GitHub] spark pull request: [SPARK-4789] [mllib] Standardize ML Prediction...

2014-12-09 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/3637#issuecomment-66403895 @jkbradley Apologies for the delay - I just read your design doc and am catching up on this discussion. Sorry if I missed something, but could you clarify the use ca

[GitHub] spark pull request: [SPARK-4806] Streaming doc update for 1.2

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3653#issuecomment-66403488 [Test build #24294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24294/consoleFull) for PR 3653 at commit [`17b99fb`](https://gith

[GitHub] spark pull request: [SPARK-4806] Streaming doc update for 1.2

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3653#issuecomment-66403490 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21583881 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-4759] Fix driver hanging from coalescin...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3633#issuecomment-66403290 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4759] Fix driver hanging from coalescin...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3633#issuecomment-66403289 [Test build #24292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24292/consoleFull) for PR 3633 at commit [`e520d6b`](https://gith

[GitHub] spark pull request: SPARK-3779. yarn spark.yarn.applicationMaster....

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3471#issuecomment-66402995 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: SPARK-3779. yarn spark.yarn.applicationMaster....

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3471#issuecomment-66402988 [Test build #24293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24293/consoleFull) for PR 3471 at commit [`20b9887`](https://gith

[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.

2014-12-09 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66402583 cc @mccheah --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: [SPARK-4161]Spark shell class path is not corr...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3050#issuecomment-66402461 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4161]Spark shell class path is not corr...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3050#issuecomment-66402458 [Test build #24291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24291/consoleFull) for PR 3050 at commit [`abb6fa4`](https://gith

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21583427 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-4329][WebUI] HistoryPage pagenation

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3194#issuecomment-66400801 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4329][WebUI] HistoryPage pagenation

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3194#issuecomment-66400794 [Test build #24290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24290/consoleFull) for PR 3194 at commit [`15d3d2d`](https://gith

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582841 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582813 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -149,41 +162,45 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582789 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -149,41 +162,45 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66400109 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66400102 Yeah, I got lazy while grabbing screencaps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-66400106 [Test build #24288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24288/consoleFull) for PR 3607 at commit [`d619996`](https://gith

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582645 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-6633 [Test build #24295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24295/consoleFull) for PR 3659 at commit [`bd72899`](https://githu

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582609 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582597 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66399889 [Test build #24285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24285/consoleFull) for PR 3658 at commit [`4a4ed42`](https://gith

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3603#issuecomment-66399885 @yu-iskw Thanks for the PR! I added some comments but left a question for @mengxr Also, could you please add the [mllib] tag to the PR title? --- If your project

[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66399893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4754] Refactor SparkContext into Execut...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3614#issuecomment-66399823 [Test build #24286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24286/consoleFull) for PR 3614 at commit [`59baf6c`](https://gith

[GitHub] spark pull request: [SPARK-4754] Refactor SparkContext into Execut...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3614#issuecomment-66399829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582550 --- Diff: python/pyspark/mllib/feature.py --- @@ -212,7 +212,7 @@ class IDFModel(JavaVectorTransformer): """ Represents an IDF model that c

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582552 --- Diff: python/pyspark/mllib/feature.py --- @@ -220,12 +220,15 @@ def transform(self, dataset): the terms which occur in fewer than `minDocFre

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582540 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala --- @@ -53,6 +53,19 @@ class IDFSuite extends FunSuite with MLlibTestSparkCont

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582537 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -221,33 +238,87 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582538 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala --- @@ -17,12 +17,10 @@ package org.apache.spark.mllib.feature

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582546 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/IDFSuite.scala --- @@ -86,6 +101,19 @@ class IDFSuite extends FunSuite with MLlibTestSparkCon

[GitHub] spark pull request: [SPARK-4494] IDFModel.transform() add support ...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3603#discussion_r21582536 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala --- @@ -174,37 +174,18 @@ class IDFModel private[mllib] (val idf: Vector) extends Se

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66399713 The new left-hand side gradient is a bit much, but otherwise this LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on G

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3659#issuecomment-66399624 /cc @aarondav, who suggested the `` tag. I think this should go into `branch-1.2`, since that's where this feature was introduced. --- If your project is set up for i

[GitHub] spark pull request: Config updates for the new shuffle transport.

2014-12-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3657 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [Minor] Use tag for help icon in web UI ...

2014-12-09 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/3659 [Minor] Use tag for help icon in web UI page header This small commit makes the `(?)` web UI help link into a superscript, which should address feedback that the current design makes it look like

[GitHub] spark pull request: [SPARK-4772] Clear local copies of accumulator...

2014-12-09 Thread nkronenfeld
Github user nkronenfeld commented on the pull request: https://github.com/apache/spark/pull/3570#issuecomment-66399588 sorry, must have accidentally hit cancel instead of comment the first time. Should be set now. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-4771][Docs] Document standalone cluster...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3627#issuecomment-66399541 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4771][Docs] Document standalone cluster...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3627#issuecomment-66399537 [Test build #24284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24284/consoleFull) for PR 3627 at commit [`9ca0908`](https://gith

[GitHub] spark pull request: [SPARK-4161]Spark shell class path is not corr...

2014-12-09 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/3050#issuecomment-66399472 In my local test, it works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1222#discussion_r21582392 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -149,41 +162,45 @@ private[history] class FsHistoryProvider(co

[GitHub] spark pull request: Config updates for the new shuffle transport.

2014-12-09 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/3657#issuecomment-66399456 Merging this into master and branch-1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-4730][YARN] Warn against deprecated YAR...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3590#issuecomment-66399349 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4730][YARN] Warn against deprecated YAR...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3590#issuecomment-66399346 [Test build #24287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24287/consoleFull) for PR 3590 at commit [`36e0753`](https://gith

[GitHub] spark pull request: Config updates for the new shuffle transport.

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3657#issuecomment-66399245 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: Config updates for the new shuffle transport.

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3657#issuecomment-66399244 [Test build #24281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24281/consoleFull) for PR 3657 at commit [`7370eab`](https://gith

[GitHub] spark pull request: [SPARK-4707][STREAMING] Reliable Kafka Receive...

2014-12-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3655#issuecomment-66399109 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4707][STREAMING] Reliable Kafka Receive...

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3655#issuecomment-66399105 [Test build #24282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24282/consoleFull) for PR 3655 at commit [`5e2e7ad`](https://gith

[GitHub] spark pull request: Update FsHistoryProvider.scala

2014-12-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2546#issuecomment-66399032 @397090770 given that a lot of changes have gone in since this was opened, I would recommend that we close this issue for now until we describe the issue in a JIRA.

[GitHub] spark pull request: [SPARK-3611] Show number of cores for each exe...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2980#issuecomment-66398809 Hey @devldevelopment given that there is no simple way to support this across different cluster managers, I would recommend that we close this issue for now. We can re

[GitHub] spark pull request: [SPARK-4771][Docs] Document standalone cluster...

2014-12-09 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/3627#issuecomment-66398746 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-4771][Docs] Document standalone cluster...

2014-12-09 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/3627#discussion_r21582109 --- Diff: docs/spark-standalone.md --- @@ -272,6 +272,15 @@ should specify them through the `--jars` flag using comma as a delimiter (e.g. ` To control the

[GitHub] spark pull request: [SPARK-4806] Streaming doc update for 1.2

2014-12-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3653#issuecomment-66398673 [Test build #24294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24294/consoleFull) for PR 3653 at commit [`17b99fb`](https://githu

[GitHub] spark pull request: [SPARK-4772] Clear local copies of accumulator...

2014-12-09 Thread nkronenfeld
Github user nkronenfeld commented on the pull request: https://github.com/apache/spark/pull/3570#issuecomment-66398560 I thought I'd done so, it looks like it lost my changes I'll fix that asap --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-4772] Clear local copies of accumulator...

2014-12-09 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3570#issuecomment-66398459 LGTM. Just in case you missed my earlier comment, are you still planning to update the PR description to reflect the actual changes vs. the ones you had planned? ---

[GitHub] spark pull request: SPARK-4749: Allow initializing KMeans clusters...

2014-12-09 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3610#issuecomment-66398376 @nxwhite-str Thanks for the PR! Could you please update the title to start with "[SPARK-4749] [mllib]" to help with automated tagging? --- If your project is set up

[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...

2014-12-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2516#issuecomment-66398367 Hey @tigerquoll usually for large patches like this we require a design doc on the JIRA. Especially because the priority of this is not super important, I would recomm

[GitHub] spark pull request: SPARK-4749: Allow initializing KMeans clusters...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3610#discussion_r21581982 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala --- @@ -43,7 +43,8 @@ class KMeans private ( private var runs: Int,

[GitHub] spark pull request: SPARK-4749: Allow initializing KMeans clusters...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3610#discussion_r21581986 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala --- @@ -353,6 +359,31 @@ object KMeans { } /** + * T

[GitHub] spark pull request: SPARK-4749: Allow initializing KMeans clusters...

2014-12-09 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3610#discussion_r21581990 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala --- @@ -90,6 +90,27 @@ class KMeansSuite extends FunSuite with MLlibTest

[GitHub] spark pull request: [SPARK-4707][STREAMING] Reliable Kafka Receive...

2014-12-09 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/3655#issuecomment-66398284 Thanks Hari, seems this is a simple solution. BTW should we make `count = 3` as a configurable parameter? For others LGTM. Original thoughts of introducing pen

  1   2   3   4   5   >