[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76138000 [Test build #27989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27989/consoleFull) for PR 4754 at commit [`c73aabe`](https://gith

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76138008 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76138619 [Test build #27990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27990/consoleFull) for PR 4754 at commit [`7f958c1`](https://gith

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76138628 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-6030][CORE] Don't alignSize the compute...

2015-02-26 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/4783#issuecomment-76138824 @advancedxy -- Thanks for the detailed writeup and the gist. I can see the problem but I am not sure this is the complete fix. You are right that shellSizes don't need

[GitHub] spark pull request: [SPARK-6030][CORE] Don't alignSize the compute...

2015-02-26 Thread advancedxy
Github user advancedxy commented on the pull request: https://github.com/apache/spark/pull/4783#issuecomment-76139869 Ok, Thanks for the update. I will look into the jol and the example when I get some spare time. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-6030][CORE] Don't alignSize the compute...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4783#issuecomment-76140139 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-6030][CORE] Don't alignSize the compute...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4783#issuecomment-76140128 [Test build #27992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27992/consoleFull) for PR 4783 at commit [`8a127dc`](https://gith

[GitHub] spark pull request: [SPARK-4902][CORE] gap-sampling performance op...

2015-02-26 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/3744#issuecomment-76143785 ```scala test("bernoulli sampling benchmark") { class BernoulliSamplerBenchmark(val fraction: Double, items: () => Iterator[Int]) extends scala.testing.Benchmark

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread shenh062326
Github user shenh062326 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25413198 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76145260 Sorry for late, I will change it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146091 [Test build #27993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27993/consoleFull) for PR 4363 at commit [`2dc456e`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146315 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146314 [Test build #27993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27993/consoleFull) for PR 4363 at commit [`2dc456e`](https://gith

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-76147005 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-76147388 [Test build #27994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27994/consoleFull) for PR 4764 at commit [`edee0d2`](https://githu

[GitHub] spark pull request: SPARK-6036 avoid race condition between eventl...

2015-02-26 Thread liyezhang556520
GitHub user liyezhang556520 opened a pull request: https://github.com/apache/spark/pull/4785 SPARK-6036 avoid race condition between eventlogListener and akka actor system For detail description, pls refer to [SPARK-6036](https://issues.apache.org/jira/browse/SPARK-6036). You can

[GitHub] spark pull request: [SPARK-6029] Stop excluding fastutil package

2015-02-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4780#discussion_r25414302 --- Diff: pom.xml --- @@ -471,13 +471,6 @@ com.clearspring.analytics stream 2.7.0 - - -

[GitHub] spark pull request: [SPARK-6036][CORE] avoid race condition betwee...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4785#issuecomment-76148112 [Test build #27995 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27995/consoleFull) for PR 4785 at commit [`79b15b3`](https://githu

[GitHub] spark pull request: Modify default value description for spark.sch...

2015-02-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4781#discussion_r25414439 --- Diff: docs/configuration.md --- @@ -1018,7 +1018,7 @@ Apart from these, the following properties are also available, and may be useful s

[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...

2015-02-26 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/4758#discussion_r25414445 --- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala --- @@ -315,7 +315,14 @@ abstract class

[GitHub] spark pull request: [SPARK-6037][SQL] Avoiding duplicate Parquet s...

2015-02-26 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/4786 [SPARK-6037][SQL] Avoiding duplicate Parquet schema merging `FilteringParquetRowInputFormat` manually merges Parquet schemas before computing splits. However, it is duplicate because the schemas are

[GitHub] spark pull request: [SPARK-6037][SQL] Avoiding duplicate Parquet s...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4786#issuecomment-76148807 [Test build #27996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27996/consoleFull) for PR 4786 at commit [`ef78a5a`](https://githu

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-76149037 [Test build #27994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27994/consoleFull) for PR 4764 at commit [`edee0d2`](https://gith

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-76149044 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5259][CORE]Make sure mapStage.pendingta...

2015-02-26 Thread suyanNone
Github user suyanNone commented on a diff in the pull request: https://github.com/apache/spark/pull/4055#discussion_r25414968 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -483,8 +483,9 @@ private[spark] class TaskSetManager( //

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76151469 [Test build #27997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27997/consoleFull) for PR 4363 at commit [`1a042ff`](https://githu

[GitHub] spark pull request: [SPARK-5914] to run spark-submit requiring onl...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4742#issuecomment-76152977 LGTM. I'll fix the space before the brace on merge. This seems like a clean, simple fix. --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: SPARK-2168 [Spark core] Use relative URIs for ...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4778#issuecomment-76153038 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark pull request: SPARK-2168 [Spark core] Use relative URIs for ...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4778#issuecomment-76153635 [Test build #27998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27998/consoleFull) for PR 4778 at commit [`6d7866d`](https://githu

[GitHub] spark pull request: [SPARK-5259][CORE]Make sure mapStage.pendingta...

2015-02-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4055#discussion_r25416698 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -483,8 +483,9 @@ private[spark] class TaskSetManager( // a

[GitHub] spark pull request: [SPARK-6036][CORE] avoid race condition betwee...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4785#issuecomment-76155748 [Test build #27995 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27995/consoleFull) for PR 4785 at commit [`79b15b3`](https://gith

[GitHub] spark pull request: [SPARK-6036][CORE] avoid race condition betwee...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4785#issuecomment-76155759 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-6036][CORE] avoid race condition betwee...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4785#issuecomment-76156191 Looks like a legit test failure in `EventLoggingListenerSuite`, hm. I think there may be a similar JIRA / PR for this, let me look. --- If your project is set up for it,

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/4787 SPARK-4300 [CORE] Race condition during SparkWorker shutdown Close appender saving stdout/stderr before destroying process to avoid exception on reading closed input stream. (This also removes a

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76159001 [Test build #27999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27999/consoleFull) for PR 4787 at commit [`e0cdabf`](https://githu

[GitHub] spark pull request: [SPARK-6037][SQL] Avoiding duplicate Parquet s...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4786#issuecomment-76159126 [Test build #27996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27996/consoleFull) for PR 4786 at commit [`ef78a5a`](https://gith

[GitHub] spark pull request: [SPARK-6037][SQL] Avoiding duplicate Parquet s...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4786#issuecomment-76159133 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4771#discussion_r25419499 --- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala --- @@ -101,13 +104,16 @@ private[hive] object S

[GitHub] spark pull request: SPARK-5983 [WEBUI] Don't respond to HTTP TRACE...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4765#issuecomment-76161066 @kayousterhout can I get your second opinion on this? I know you've worked on a lot of the UI code. It's a tiny bit of extra code for a quite theoretical low-priority s

[GitHub] spark pull request: [SPARK-5914] to run spark-submit requiring onl...

2015-02-26 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4742 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76161857 [Test build #27997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27997/consoleFull) for PR 4363 at commit [`1a042ff`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76161867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: SPARK-1216. Add a OneHotEncoder for handling c...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/304#issuecomment-76162378 SPARK-5888 tracks the same addition but for the new API. Let's close this one and reopen a PR against that for the new impl. --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-2168 [Spark core] Use relative URIs for ...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4778#issuecomment-76163934 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: SPARK-2168 [Spark core] Use relative URIs for ...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4778#issuecomment-76163920 [Test build #27998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27998/consoleFull) for PR 4778 at commit [`6d7866d`](https://gith

[GitHub] spark pull request: SPARK-2168 [Spark core] Use relative URIs for ...

2015-02-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4778#discussion_r25421061 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala --- @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] spark pull request: SPARK-4704 [CORE] SparkSubmitDriverBootstrap d...

2015-02-26 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/4788 SPARK-4704 [CORE] SparkSubmitDriverBootstrap doesn't flush output Join on output threads to make sure any lingering output from process reaches stdout, stderr before exiting CC @andrewor14 s

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76165482 [Test build #27999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27999/consoleFull) for PR 4787 at commit [`e0cdabf`](https://gith

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76165494 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: SPARK-4704 [CORE] SparkSubmitDriverBootstrap d...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4788#issuecomment-76165690 [Test build #28000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28000/consoleFull) for PR 4788 at commit [`ad7114e`](https://githu

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76165744 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...

2015-02-26 Thread suyanNone
Github user suyanNone commented on a diff in the pull request: https://github.com/apache/spark/pull/3629#discussion_r25421661 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -328,6 +330,7 @@ private[spark] class MemoryStore(blockManager: BlockManager

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76166356 [Test build #28001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28001/consoleFull) for PR 4787 at commit [`e0cdabf`](https://githu

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/4588#discussion_r25422335 --- Diff: core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala --- @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/4588#discussion_r25422350 --- Diff: core/src/main/scala/org/apache/spark/rpc/ActionScheduler.scala --- @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/4588#discussion_r25422378 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala --- @@ -45,30 +42,37 @@ private[spark] class WorkerWatcher(workerUrl: String)

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4588#issuecomment-76168122 [Test build #28002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28002/consoleFull) for PR 4588 at commit [`04a106e`](https://githu

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/4588#discussion_r25422602 --- Diff: core/src/main/scala/org/apache/spark/rpc/ActionScheduler.scala --- @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/4588#discussion_r25423137 --- Diff: core/src/main/scala/org/apache/spark/rpc/akka/AkkaRpcEnv.scala --- @@ -0,0 +1,287 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/4588#issuecomment-76169822 I will update this PR as per discussion in https://issues.apache.org/jira/browse/SPARK-5124 --- If your project is set up for it, you can reply to this email and have yo

[GitHub] spark pull request: [SPARK-6040][SQL] Fix the percent bug in table...

2015-02-26 Thread watermen
GitHub user watermen opened a pull request: https://github.com/apache/spark/pull/4789 [SPARK-6040][SQL] Fix the percent bug in tablesample HiveQL expression like `select count(1) from src tablesample(1 percent);` means take 1% sample to select. But it means 100% in the current versi

[GitHub] spark pull request: [SPARK-6040][SQL] Fix the percent bug in table...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4789#issuecomment-76172235 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: SPARK-4704 [CORE] SparkSubmitDriverBootstrap d...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4788#issuecomment-76175633 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-4704 [CORE] SparkSubmitDriverBootstrap d...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4788#issuecomment-76175625 [Test build #28000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28000/consoleFull) for PR 4788 at commit [`ad7114e`](https://gith

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76176233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: SPARK-4300 [CORE] Race condition during SparkW...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4787#issuecomment-76176221 [Test build #28001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28001/consoleFull) for PR 4787 at commit [`e0cdabf`](https://gith

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4588#issuecomment-76178126 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-5124][Core] A standard RPC interface an...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4588#issuecomment-76178118 [Test build #28002 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28002/consoleFull) for PR 4588 at commit [`04a106e`](https://gith

[GitHub] spark pull request: [SPARK-6007][SQL] Add numRows param in DataFra...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4767#issuecomment-76180179 [Test build #28003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28003/consoleFull) for PR 4767 at commit [`a0e0f4b`](https://githu

[GitHub] spark pull request: [SPARK-6041][GraphX] Compute shortest path for...

2015-02-26 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/4790 [SPARK-6041][GraphX] Compute shortest path for graph with edge distances Add the function to compute shortest path for graph with edge distances. You can merge this pull request into a Git repository

[GitHub] spark pull request: [SPARK-6041][GraphX] Compute shortest path for...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4790#issuecomment-76186543 [Test build #28004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28004/consoleFull) for PR 4790 at commit [`b8439d6`](https://githu

[GitHub] spark pull request: [SPARK-6006][SQL]: Optimize count distinct for...

2015-02-26 Thread saucam
Github user saucam commented on the pull request: https://github.com/apache/spark/pull/4764#issuecomment-76184234 Fixed the null count test failure. Optimization works only in case of single count distinct in select clause --- If your project is set up for it, you can reply to this e

[GitHub] spark pull request: [SPARK-4924] Add a library for launching Spark...

2015-02-26 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3916#issuecomment-76188802 Sorry I haven't had time to look at this in detail, one thing I was looking for though was some documentation for the high level api. I realize java doc will document

[GitHub] spark pull request: [SPARK-6023][SQL] ParquetConversions fails to ...

2015-02-26 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4782 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-6030][CORE] Don't alignSize the compute...

2015-02-26 Thread advancedxy
Github user advancedxy commented on the pull request: https://github.com/apache/spark/pull/4783#issuecomment-76190779 @shivaram -- I followed the [JDK-8024912](https://bugs.openjdk.java.net/browse/JDK-8024912), [JDK-8024913](https://bugs.openjdk.java.net/browse/JDK-8024913) and [JDK-

[GitHub] spark pull request: [SPARK-6016][SQL] Cannot read the parquet tabl...

2015-02-26 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4775#issuecomment-76192307 This LGTM, please help rebasing it, then I can merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-4545 [STREAMING] [WIP] If first Spark St...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4791#issuecomment-76191852 [Test build #28005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28005/consoleFull) for PR 4791 at commit [`e3b4f51`](https://githu

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

2015-02-26 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/4688#issuecomment-76193892 So actually one other option would be to build our own mechanism for pushing new tgt's and updating the UGI. Storm actually has this functionality, you could look ther

[GitHub] spark pull request: SPARK-4545 [STREAMING] [WIP] If first Spark St...

2015-02-26 Thread srowen
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/4791 SPARK-4545 [STREAMING] [WIP] If first Spark Streaming batch fails, it waits 10x batch duration before stopping Consider failed jobs completed too, to avoid excessive waiting dur...ing shutdown for j

[GitHub] spark pull request: [SPARK-6007][SQL] Add numRows param in DataFra...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4767#issuecomment-76198015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6016][SQL] Cannot read the parquet tabl...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4775#issuecomment-76198314 [Test build #28006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28006/consoleFull) for PR 4775 at commit [`78787b1`](https://githu

[GitHub] spark pull request: [SPARK-6007][SQL] Add numRows param in DataFra...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4767#issuecomment-76198005 [Test build #28003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28003/consoleFull) for PR 4767 at commit [`a0e0f4b`](https://gith

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76199342 [Test build #28007 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28007/consoleFull) for PR 4754 at commit [`b7a9e93`](https://githu

[GitHub] spark pull request: [SPARK-6041][GraphX] Compute shortest path for...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4790#issuecomment-76200063 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6041][GraphX] Compute shortest path for...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4790#issuecomment-76200053 [Test build #28004 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28004/consoleFull) for PR 4790 at commit [`b8439d6`](https://gith

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread piaozhexiu
Github user piaozhexiu commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76203984 Thank you Sean for the review! To answer your question, HiveThriftServer2.scala seems to have the same issue because its shutdown hook is basically identical t

[GitHub] spark pull request: SPARK-4545 [STREAMING] [WIP] If first Spark St...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4791#issuecomment-76207169 [Test build #28005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28005/consoleFull) for PR 4791 at commit [`e3b4f51`](https://gith

[GitHub] spark pull request: SPARK-4545 [STREAMING] [WIP] If first Spark St...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4791#issuecomment-76207186 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6029] Stop excluding fastutil package

2015-02-26 Thread jkleckner
Github user jkleckner commented on a diff in the pull request: https://github.com/apache/spark/pull/4780#discussion_r25440596 --- Diff: pom.xml --- @@ -471,13 +471,6 @@ com.clearspring.analytics stream 2.7.0 - - -

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76209607 [Test build #28009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28009/consoleFull) for PR 4771 at commit [`46e73b3`](https://githu

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread piaozhexiu
Github user piaozhexiu commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76209601 Done. Thanks again! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76204252 I think it would be good to try to give those the same treatment while we're at it, yes. I think you're welcome to add that. --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76212311 [Test build #28007 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28007/consoleFull) for PR 4754 at commit [`b7a9e93`](https://gith

[GitHub] spark pull request: [SPARK-6014] [Core] java.io.IOException: Files...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4771#issuecomment-76200389 [Test build #28008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28008/consoleFull) for PR 4771 at commit [`86d1baa`](https://githu

[GitHub] spark pull request: [SPARK-5979][SPARK-6031][SPARK-6032] Refactori...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4754#issuecomment-76212327 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6029] Stop excluding fastutil package

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4780#issuecomment-76214312 Since the PR is the implementation of an issue resolution, if the discussion is about the implementation it can happen here on the PR. Shading isn't the issue in

[GitHub] spark pull request: [SPARK-6029] Stop excluding fastutil package

2015-02-26 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4780#issuecomment-76214905 (PS good news, I see that `parquet-column`'s shading is set to only include classes that are used from `fastutil`. That's great; there are only tens of classes added. That

[GitHub] spark pull request: [SPARK-6016][SQL] Cannot read the parquet tabl...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4775#issuecomment-76215068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-6016][SQL] Cannot read the parquet tabl...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4775#issuecomment-76215053 [Test build #28006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28006/consoleFull) for PR 4775 at commit [`78787b1`](https://gith

  1   2   3   4   5   6   >