[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread mvj101
Github user mvj101 commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67124150 Fixing style issues now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67123940 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67123939 [Test build #24486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24486/consoleFull) for PR 2872 at commit [`4dc6756`](https://gith

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67123879 [Test build #24486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24486/consoleFull) for PR 2872 at commit [`4dc6756`](https://githu

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67123620 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67123069 Thanks for these latest changes. This looks pretty good to me, but I think we can go one step further in removing the last traces of the WholeTextFile-specific

[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...

2014-12-15 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/3319#discussion_r21881678 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala --- @@ -197,6 +295,171 @@ class SparseMatrix( } override def co

[GitHub] spark pull request: [SPARK-4818][Core] Add 'iterator' to reduce me...

2014-12-15 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3671#issuecomment-67122065 @pwendell Is it OK to put this patch into branch 1.2, or it's too late? --- If your project is set up for it, you can reply to this email and have your reply appear on Gi

[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...

2014-12-15 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/3319#discussion_r21881510 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala --- @@ -197,6 +295,171 @@ class SparseMatrix( } override def co

[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...

2014-12-15 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/3319#discussion_r21881450 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala --- @@ -313,15 +593,145 @@ object Matrices { * @return Square `DenseMatrix`

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67120867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67120862 [Test build #24483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24483/consoleFull) for PR 3672 at commit [`526315d`](https://gith

[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67120498 [Test build #24484 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24484/consoleFull) for PR 3710 at commit [`d09066e`](https://gith

[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67120502 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67120306 [Test build #24485 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24485/consoleFull) for PR 3301 at commit [`95d13eb`](https://githu

[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67120182 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-4841] fix zip with textFile()

2014-12-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3706 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4841] fix zip with textFile()

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3706#issuecomment-67120024 Thanks for the update. I've looked over this again and tried it out with a few more hand-written test cases and I've been unable to find any more bugs, so this

[GitHub] spark pull request: [SPARK-4855][mllib] testing the Chi-squared hy...

2014-12-15 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3679#issuecomment-67119469 @jbencook Thanks for the updates! (Your comment about checking for exceptions makes me wonder if you were right before to throw a more meaningful exception than a Py4

[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67118930 This looks like a spurious test failure that might have been caused by a disk failure in one of the Jenkins workers. Let's try this again Jenkins, retest t

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67118614 Even though we don't have Jenkins tests for the EC2 scripts, I'm just going to have Jenkins run this so that I can avoid an inadvertent build break. Jenkins, th

[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...

2014-12-15 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67118160 @maropu Thank you for work in this. I'm just wondering what's the user case that his argument will be useful? Because progress bar is disabled if logging level is DEBUG or

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3635 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3635#issuecomment-67117844 Alright, this seems fine to me. I'm going to merge this into `master`. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-4547 [MLLIB] [WIP] OOM when making bins ...

2014-12-15 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3702#issuecomment-67117735 @srowen Trying to guarantee exactly the requested number of points does seem like more trouble than it is worth. It might require collecting the # of points in each p

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3635#discussion_r21879969 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -67,11 +67,13 @@ private[spark] class DiskBlockManager(blockManager: B

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3635#discussion_r21879937 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -67,11 +67,13 @@ private[spark] class DiskBlockManager(blockManager: B

[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...

2014-12-15 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/3709#discussion_r21879847 --- Diff: bin/utils.sh --- @@ -32,10 +32,11 @@ function gatherSparkSubmitOpts() { APPLICATION_OPTS=() while (($#)); do case "$1" in -

[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67117052 /cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request: [SPARK-4501][Core] - Create build/mvn to autom...

2014-12-15 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/3707#issuecomment-67115834 Hi @brennonyork I was trying this patch out. Seemed good overall. I felt it would be good to print some info messages that indicates what is happening for exam

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread mvj101
Github user mvj101 commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67115653 Thanks, I believe I've updated the code according to your comments. Mike --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67115316 @srowen I checked through more carefully this time, and there were a couple of changes for which I could not find associated warnings. --- If your project is set up f

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878976 --- Diff: core/pom.xml --- @@ -352,9 +352,9 @@ - + --- End diff -- How d

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878944 --- Diff: sql/core/src/main/java/org/apache/spark/sql/api/java/UserDefinedType.java --- @@ -35,6 +35,7 @@ protected UserDefinedType() { } public bool

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878958 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala --- @@ -32,7 +32,7 @@ import org.apache.hadoop.io.Writable

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878950 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala --- @@ -24,6 +24,8 @@ import org.apache.spark.sql.catalyst.expressions._ imp

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878955 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala --- @@ -450,7 +452,9 @@ class ParquetQuerySuite extends QueryTest wit

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878961 --- Diff: sql/hive/src/test/java/org/apache/spark/sql/hive/execution/UDFListListInt.java --- @@ -23,25 +23,21 @@ public class UDFListListInt ext

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878964 --- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java --- @@ -57,7 +57,7 @@ public void equalIterable(Iterable a, Iterable b) {

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878942 --- Diff: mllib/src/test/java/org/apache/spark/mllib/feature/JavaTfIdfSuite.java --- @@ -49,6 +49,7 @@ public void tearDown() { public void tfIdf() {

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878946 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala --- @@ -454,7 +454,7 @@ private[parquet] object ParquetTypesConverter exte

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878938 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -739,7 +739,7 @@ class DAGSchedulerSuite extends TestKit(ActorSyste

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878948 --- Diff: sql/core/src/test/java/org/apache/spark/sql/api/java/JavaRowSuite.java --- @@ -141,6 +141,7 @@ public void constructComplexRow() { doub

[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67115194 [Test build #24484 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24484/consoleFull) for PR 3710 at commit [`d09066e`](https://githu

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878899 --- Diff: core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala --- @@ -24,14 +24,14 @@ import org.apache.spark.deploy.SparkHadoopUtil

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878897 --- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java --- @@ -1556,7 +1558,7 @@ public void testGuavaOptional() { @Test public void t

[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...

2014-12-15 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/3710 [SPARK-4859][Streaming] Improve StreamingListenerBus * Fix the race condition of `queueFullErrorMessageLogged`. * Log the error from listener rather than crashing `listenerThread`. You can merge

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878883 --- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java --- @@ -491,6 +492,7 @@ public Integer call(Integer a, Integer b) { Assert.assertEqu

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878614 --- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java --- @@ -184,6 +184,7 @@ public void sortByKey() { Assert.assertEquals(new Tuple2(3,

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

2014-12-15 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21878594 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala --- @@ -19,6 +19,7 @@ package org.apache.spark.scheduler import j

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3635#issuecomment-67114251 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3635#issuecomment-67114246 [Test build #24482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24482/consoleFull) for PR 3635 at commit [`dd1c66d`](https://gith

[GitHub] spark pull request: [WIP][SPARK-3860][SQL] Improved dimension join...

2014-12-15 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3270#issuecomment-67113744 A bunch of irrelevant changes are found in the diff, seems that this PR was opened against a wrong base branch? --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67113736 [Test build #24483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24483/consoleFull) for PR 3672 at commit [`526315d`](https://githu

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67113444 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-4856] [SQL] NullType instead of StringT...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3708#issuecomment-67111052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4856] [SQL] NullType instead of StringT...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3708#issuecomment-67111050 [Test build #24481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24481/consoleFull) for PR 3708 at commit [`853de51`](https://gith

[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67110752 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...

2014-12-15 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/3709 [SPARK-4858] Add an option to turn off a progress bar in spark-shell Add an '--no-progress-bar' option to easily turn off a progress bar in spark-shell for users who'd like to look into debug logs or

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3635#issuecomment-67109651 [Test build #24482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24482/consoleFull) for PR 3635 at commit [`dd1c66d`](https://githu

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-15 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-67109137 @avulanov If the `MLP` contains multi-layer. Usually don't need to l2 regularization(see: [Dropout: A simple way to prevent neural networks from overfitting](http://w

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-15 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-67108573 @avulanov The label a one-hot vector. See the convert code in `MinstDatasetSuite`. And the latest code has fixed this bug. --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread XuTingjun
Github user XuTingjun commented on a diff in the pull request: https://github.com/apache/spark/pull/3635#discussion_r21876377 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -67,11 +67,13 @@ private[spark] class DiskBlockManager(blockManager: B

[GitHub] spark pull request: [SPARK-4855][mllib] testing the Chi-squared hy...

2014-12-15 Thread jbencook
Github user jbencook commented on the pull request: https://github.com/apache/spark/pull/3679#issuecomment-67108325 OK, should be good to go now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-4855][mllib] testing the Chi-squared hy...

2014-12-15 Thread jbencook
Github user jbencook commented on the pull request: https://github.com/apache/spark/pull/3679#issuecomment-67107800 Thanks for the comments @jkbradley. I revised the JIRA tag to reference the new ticket you added, updated the URL by adding the "/" (oops), and removed the pytho

[GitHub] spark pull request: [SPARK-4087] use broadcast for task only when ...

2014-12-15 Thread davies
Github user davies closed the pull request at: https://github.com/apache/spark/pull/2933 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4087] use broadcast for task only when ...

2014-12-15 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2933#issuecomment-67107062 Close this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: SPARK-4547 [MLLIB] [WIP] OOM when making bins ...

2014-12-15 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3702#issuecomment-67106963 @jkbradley Sure, well my thinking was that there is a nice straightforward approach based on sampling every Nth point, so the natural thing is to add a parameter for this

[GitHub] spark pull request: [SPARK-4856] [SQL] NullType instead of StringT...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3708#issuecomment-67106729 [Test build #24481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24481/consoleFull) for PR 3708 at commit [`853de51`](https://githu

[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67106502 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67106493 [Test build #24480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24480/consoleFull) for PR 3561 at commit [`14daa20`](https://gith

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2848#issuecomment-67106447 Disclaimer: for iterative debugging, I use `sbt` to build Spark, not Maven. Spark 1.2.0 has docs on building with SBT. If possible, I'd switch to using that workflow.

[GitHub] spark pull request: [SPARK-4856] [SQL] NullType instead of StringT...

2014-12-15 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/3708#issuecomment-67106388 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3635#discussion_r21875371 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -67,11 +67,13 @@ private[spark] class DiskBlockManager(blockManager: Bloc

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67105422 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67105417 [Test build #24477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24477/consoleFull) for PR 3672 at commit [`526315d`](https://gith

[GitHub] spark pull request: [SPARK-4839][YARN] Adding a documentation abou...

2014-12-15 Thread oza
Github user oza commented on the pull request: https://github.com/apache/spark/pull/3689#issuecomment-67105316 Thanks for your comment, I'm updating it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4792] Add error message when making loc...

2014-12-15 Thread XuTingjun
Github user XuTingjun commented on the pull request: https://github.com/apache/spark/pull/3635#issuecomment-67105200 @JoshRosen, If users share the cluster, and one unconsciously delete the dir, it will affect the application. So I think add exists() is more secure. --- If your proje

[GitHub] spark pull request: [SPARK-2325] Utils.getLocalDir had better chec...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1281#issuecomment-67103624 (I think 'close this issue' is the magic that the script needs) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-927] detect numpy at time of use

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2313#issuecomment-67103364 Can you close this issue? Thanks! (I think "close this issue" was the magic phrase I needed for the script to pick it up, but let's see). --- If your project

[GitHub] spark pull request: [SPARK-2980][mllib] testing the Chi-squared hy...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3679#issuecomment-67103266 [Test build #546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/546/consoleFull) for PR 3679 at commit [`a17ee84`](https://githu

[GitHub] spark pull request: [SPARK-2325] Utils.getLocalDir had better chec...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1281#issuecomment-67103247 It looks like the [JIRA referenced from this PR](https://issues.apache.org/jira/browse/SPARK-2325) was resolved as a duplicate of an issue which was fixed in #2002. Th

[GitHub] spark pull request: [SPARK-4087] use broadcast for task only when ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2933#issuecomment-67102974 What's the status on this PR / JIRA? As far as I know, it seems that TorrentBroadcast has been more stable lately, so if the only motivation here was stability then I

[GitHub] spark pull request: [SPARK-2301] add ability to submit multiple ja...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-67102624 @andrewor14 @lianhuiwang Is this now "Won't Fix"? If so, we should close this and resolve the JIRA. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-67102568 [Test build #24474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24474/consoleFull) for PR 3694 at commit [`f21bfd4`](https://gith

[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...

2014-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-67102573 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24

[GitHub] spark pull request: Add unit test to spark_ec2 script

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/134#issuecomment-67102486 /bump. Let's see if we can get the auto-close script to pick up this JIRA. close this issue close this pr close this please --- If your proje

[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2895#issuecomment-67102429 If you don't mind, could you close this PR since it has been subsumed by another commit? If we want to track the progress / backport status of a different fix, then we

[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...

2014-12-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67102441 [Test build #24480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24480/consoleFull) for PR 3561 at commit [`14daa20`](https://githu

[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67102056 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67101983 Overall, this looks good to me. I left a couple of nitpicky comments, but besides that + documentation, I'd be happy to merge this. To address a question asked

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2872#discussion_r21873877 --- Diff: ec2/spark_ec2.py --- @@ -162,6 +162,10 @@ def parse_args(): parser.add_option( "--copy-aws-credentials", action="store_true",

[GitHub] spark pull request: [SPARK-4006] In long running contexts, we enco...

2014-12-15 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/2914#issuecomment-67101629 jenkins, test this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2872#discussion_r21873865 --- Diff: ec2/spark_ec2.py --- @@ -162,6 +162,10 @@ def parse_args(): parser.add_option( "--copy-aws-credentials", action="store_true",

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67101443 Since this is an often-requested feature, we should mention this in the EC2 documentation page: https://github.com/apache/spark/blob/master/docs/ec2-scripts.md --- If

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2872#discussion_r21873430 --- Diff: ec2/spark_ec2.py --- @@ -341,11 +355,11 @@ def launch_cluster(conn, opts, cluster_name): if opts.ami is None: opts.ami = get_

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

2014-12-15 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2872#discussion_r21873371 --- Diff: ec2/spark_ec2.py --- @@ -303,12 +307,17 @@ def launch_cluster(conn, opts, cluster_name): user_data_content = user_data_file.read()

[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...

2014-12-15 Thread jkbradley
Github user jkbradley closed the pull request at: https://github.com/apache/spark/pull/3427 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...

2014-12-15 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3427#issuecomment-67099848 I'm closing this since I've begun breaking it into smaller PRs. I copied the PR description to the JIRA and will leave my WIP branch intact. --- If your project is se

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

2014-12-15 Thread avulanov
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/3222#issuecomment-67099517 @witgo I am interested to test you MLP implementation. I have cloned https://github.com/witgo/spark/tree/rbm. When I train MLP with SGD or LBFGS on mnist, it always prod

  1   2   3   4   >