[GitHub] spark pull request: [GraphX] Modify option name according to examp...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2888#issuecomment-60043984 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: Fix for sampling error in NumPy v1.9 [SPARK-39...

2014-10-21 Thread freeman-lab
GitHub user freeman-lab opened a pull request: https://github.com/apache/spark/pull/2889 Fix for sampling error in NumPy v1.9 [SPARK-3995][PYSPARK] Change maximum value for default seed during RDD sampling so that it is strictly less than 2 ** 32. This prevents a bug in the most rec

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-21 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-60043845 I'll just publish an empty jar in maven central - we can use it forever. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4037][SQL] Removes the SessionState ins...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2887#issuecomment-60043829 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22024/consoleFull) for PR 2887 at commit [`04b374d`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60043790 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22023/consoleFull) for PR 2851 at commit [`1eff69e`](https://github.com/a

[GitHub] spark pull request: [GraphX] Modify option name according to examp...

2014-10-21 Thread GraceH
GitHub user GraceH opened a pull request: https://github.com/apache/spark/pull/2888 [GraphX] Modify option name according to example doc in SynthBenchmark Now graphx.SynthBenchmark example has an option of iteration number named as "niter". However, in its document, it is named as "

[GitHub] spark pull request: [SPARK-4037][SQL] Removes the SessionState ins...

2014-10-21 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/2887 [SPARK-4037][SQL] Removes the SessionState instance created in HiveThriftServer2 `HiveThriftServer2` creates a global singleton `SessionState` instance and overrides `HiveContext` to inject the `

[GitHub] spark pull request: [SPARK-4033][Examples]Input of the SparkPi too...

2014-10-21 Thread advancedxy
Github user advancedxy commented on a diff in the pull request: https://github.com/apache/spark/pull/2874#discussion_r19196652 --- Diff: examples/src/main/scala/org/apache/spark/examples/SparkPi.scala --- @@ -27,7 +27,7 @@ object SparkPi { val conf = new SparkConf().setApp

[GitHub] spark pull request: [SPARK-4033][Examples]Input of the SparkPi too...

2014-10-21 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2874#issuecomment-60043115 @SaintBacchus sure, maybe this is used as a load test in some situations. What if n map tasks computed 10 iterations each (or some smaller quantum)? rather than 10

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60042976 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60042970 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22022/consoleFull) for PR 2851 at commit [`c07c306`](https://github.com/a

[GitHub] spark pull request: [SPARK-4033][Examples]Input of the SparkPi too...

2014-10-21 Thread advancedxy
Github user advancedxy commented on a diff in the pull request: https://github.com/apache/spark/pull/2874#discussion_r19196544 --- Diff: examples/src/main/scala/org/apache/spark/examples/SparkPi.scala --- @@ -27,7 +27,7 @@ object SparkPi { val conf = new SparkConf().setApp

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60041575 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22023/consoleFull) for PR 2851 at commit [`1eff69e`](https://github.com/ap

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60041418 YARN provides (2). After an app makes a request for X executors, YARN keeps track of them until the app goes away. In fact, the API to YARN isn't "I want X more executors

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-21 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-60041268 Not sure which is better ! Creating an empty project with just a pom file in it. Or depending on random jar from maven central. ? I prefer first approach. --- If your

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60041142 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-4016] Allow user to show/hide UI metric...

2014-10-21 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/2867#issuecomment-60041059 @ash211 that's a great idea, I'll add that and incorporate @andrewor14's suggestion about a drop-down menu. Thanks all! --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-3957]: show broadcast variable resource...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60040650 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22022/consoleFull) for PR 2851 at commit [`c07c306`](https://github.com/ap

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60040691 Hey Andrew - I looked at this and I had some small suggestions around naming. However, there is a big open question here relating to the semantics of requesting

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60040484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60040481 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22020/consoleFull) for PR 2746 at commit [`8bc0e9d`](https://github.com/a

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2882#issuecomment-60040386 This is nice work overall. I like the thorough tests, especially the decoupling of the writer / reader tests so that you can test the components separately and as part

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195671 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala --- @@ -553,7 +589,26 @@ object DecisionTree extends Serializable with Logging

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2871#issuecomment-60040376 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2871#issuecomment-60040371 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22018/consoleFull) for PR 2871 at commit [`9cec507`](https://github.com/a

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195658 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala --- @@ -0,0 +1,298 @@ +/* + * Licensed to the Apache So

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195610 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala --- @@ -515,6 +523,34 @@ object DecisionTree extends Serializable with Logging

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195598 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala --- @@ -629,6 +699,10 @@ object DecisionTree extends Serializable with Logging

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-60040201 Thanks for all the comments guys. I'll address them and resubmit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195595 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala --- @@ -584,6 +642,9 @@ object DecisionTree extends Serializable with Logging {

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195587 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195551 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195548 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195544 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195515 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195483 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala --- @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195486 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [WIP]SPARK-3957: show broadcast variable resou...

2014-10-21 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2851#issuecomment-60039787 ![image](https://cloud.githubusercontent.com/assets/678008/4731666/589ce496-59af-11e4-99fd-01e4b37d7fef.png) ![image](https://cloud.githubusercontent.com/

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-21 Thread codedeft
Github user codedeft commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19195461 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195389 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala --- @@ -0,0 +1,298 @@ +/* + * Licensed to the Apache So

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195386 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala --- @@ -0,0 +1,298 @@ +/* + * Licensed to the Apa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195364 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the A

[GitHub] spark pull request: SPARK-1813. Add a utility to SparkConf that ma...

2014-10-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/789 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19195340 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195339 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala --- @@ -0,0 +1,298 @@ +/* + * Licensed to the Apache So

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2840#issuecomment-60039381 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2840#issuecomment-60039380 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22019/consoleFull) for PR 2840 at commit [`db4a679`](https://github.com/a

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195316 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogWriter.scala --- @@ -0,0 +1,101 @@ +/* + * Licensed to the Ap

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19195273 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195267 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogWriter.scala --- @@ -0,0 +1,101 @@ +/* + * Licensed to the Ap

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19195258 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195244 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195251 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogReader.scala --- @@ -0,0 +1,82 @@ +/* + * Licensed to the Apa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195209 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Softw

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19195109 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19195094 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogWriter.scala --- @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19195056 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19194672 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19194607 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19194567 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2746#discussion_r19194532 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...

2014-10-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-60036985 Hold this PR, we may don't need it anymore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2840#issuecomment-60036612 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22019/consoleFull) for PR 2840 at commit [`db4a679`](https://github.com/ap

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60036614 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22020/consoleFull) for PR 2746 at commit [`8bc0e9d`](https://github.com/ap

[GitHub] spark pull request: SPARK-1813. Add a utility to SparkConf that ma...

2014-10-21 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/789#issuecomment-60036532 Thanks Sandy - I've merged this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

2014-10-21 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60036315 The latest changes reflect most of the design I have outlined in my bullet point post above. I intend to refactor it a little more by accessing the task information th

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-21 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2673#discussion_r19194279 --- Diff: pom.xml --- @@ -994,6 +1006,34 @@ org.apache.maven.plugins +maven-shade-plugin + +

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-60036048 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2871#issuecomment-60036043 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22018/consoleFull) for PR 2871 at commit [`9cec507`](https://github.com/ap

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-60036044 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22017/consoleFull)** for PR 2833 at commit [`d523fbf`](https://github.com/apac

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/2871#issuecomment-60035820 I also added a test case to check if blocks are read on use --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19194043 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogWriter.scala --- @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...

2014-10-21 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-60035199 @yhuai Thanks for pointing. Actually we cannot still move it out from ignore because 92233720368547758061.2 is BigDecimal type but num_str + 1.2 is Double type so compari

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193881 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogReader.scala --- @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache So

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193795 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193790 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193702 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193688 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193668 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19193639 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/2871#discussion_r19192935 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag](

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-60029263 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22017/consoleFull) for PR 2833 at commit [`d523fbf`](https://github.com/ap

[GitHub] spark pull request: [SPARK-4016] Allow user to show/hide UI metric...

2014-10-21 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2867#issuecomment-60028659 The first thing I can imagine doing if I start looking at these is to just check them all. Thoughts to making that operation easier, with say a check-all button? --- If

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2014-10-21 Thread viper-kun
Github user viper-kun commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-60027733 @vanzin @andrewor14. is it ok to go? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...

2014-10-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-60027697 @sarutak Can you also move [this related test](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala#L421) out of `i

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19191068 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19190985 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: [Spark-4041][SQL]attributes names in table sca...

2014-10-21 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/2884#issuecomment-60025997 Can you add a unit test? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-10-21 Thread li-zhihui
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/1616#issuecomment-60025907 @andrewor14 more comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19190926 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19190537 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: [SPARK-4033][Examples]Input of the SparkPi too...

2014-10-21 Thread SaintBacchus
Github user SaintBacchus commented on the pull request: https://github.com/apache/spark/pull/2874#issuecomment-60025137 @srowen I had considered that just judging the number of the input to be a small int will be a easy way. But I assume user just want his spark app running, not to

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-21 Thread tianyi
Github user tianyi commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-60025019 Is there any admin could review this PR ASAP? We got some PR blocked by this one. like #2542 @marmbrus @rxin --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-4026][Streaming] Write ahead log manage...

2014-10-21 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/2882#discussion_r19190251 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/FileSegment.scala --- @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] spark pull request: [SPARK-4006] In long running contexts, we enco...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2886#issuecomment-60024273 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22016/consoleFull) for PR 2886 at commit [`f48bce9`](https://github.com/a

[GitHub] spark pull request: [SPARK-4006] In long running contexts, we enco...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2886#issuecomment-60024276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-4033][Examples]Input of the SparkPi too...

2014-10-21 Thread SaintBacchus
Github user SaintBacchus commented on the pull request: https://github.com/apache/spark/pull/2874#issuecomment-60024075 @JoshRosen the n inside "0 until n " must be an integer so we can't make input as a long --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-60023054 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-60023050 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22014/consoleFull)** for PR 2673 at commit [`6fff1ce`](https://github.com/apac

[GitHub] spark pull request: [WIP] [SPARK-4031] Make torrent broadcast read...

2014-10-21 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2871#discussion_r19188778 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -63,12 +63,22 @@ private[spark] class TorrentBroadcast[T: ClassTag](

  1   2   3   4   5   >