[GitHub] spark pull request: [WIP][SQL][SPARK-3839] Reimplement Left/Right ...

2014-10-28 Thread Ishiihara
Github user Ishiihara commented on the pull request: https://github.com/apache/spark/pull/2723#issuecomment-60881119 @marmbrus All test failures have the same pattern select * from a right outer join b on condition1 join c on condition2 With the extra join oper

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60880960 [Test build #22437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22437/consoleFull) for PR 2753 at commit [`cadfd28`](https://githu

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60880706 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60880665 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60880661 [Test build #22428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22428/consoleFull) for PR 2942 at commit [`9f7aea9`](https://gith

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2615#issuecomment-60880659 [Test build #22436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22436/consoleFull) for PR 2615 at commit [`95c2e8e`](https://githu

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60880622 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark pull request: SPARK-3968 Use parquet-mr filter2 api in spark...

2014-10-28 Thread saucam
Github user saucam commented on the pull request: https://github.com/apache/spark/pull/2841#issuecomment-60880619 Hi @mateiz , thanks for the suggestions, just a few points 1. Need to know which strategy to be kept as default (currently we use a different one than the default one

[GitHub] spark pull request: SPARK-3968 Use parquet-mr filter2 api in spark...

2014-10-28 Thread saucam
Github user saucam commented on a diff in the pull request: https://github.com/apache/spark/pull/2841#discussion_r19521851 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala --- @@ -460,29 +515,85 @@ private[parquet] class FilteringParquetR

[GitHub] spark pull request: [SPARK-4122][STREAMING] Add a library that can...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2994#issuecomment-60879456 [Test build #22435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22435/consoleFull) for PR 2994 at commit [`372c749`](https://githu

[GitHub] spark pull request: [SPARK-4122][STREAMING] Add a library that can...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2994#issuecomment-60879163 [Test build #22434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22434/consoleFull) for PR 2994 at commit [`f61d82f`](https://githu

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60878901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60878899 [Test build #22433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22433/consoleFull) for PR 2944 at commit [`bc1e675`](https://gith

[GitHub] spark pull request: [SPARK-4062][Streaming]Add ReliableKafkaReceiv...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2991#issuecomment-60878843 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-4062][Streaming]Add ReliableKafkaReceiv...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2991#issuecomment-60878841 [Test build #22425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22425/consoleFull) for PR 2991 at commit [`5cc4cb1`](https://gith

[GitHub] spark pull request: [SPARK-4122][STREAMING] Add a library that can...

2014-10-28 Thread harishreedharan
GitHub user harishreedharan opened a pull request: https://github.com/apache/spark/pull/2994 [SPARK-4122][STREAMING] Add a library that can write data back to Kafka ... ...from Spark Streaming. This adds a library that can writes dstreams to Kafka. An implicit also has been

[GitHub] spark pull request: [SPARK-4109][CORE] Correctly deserialize Task....

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2971#discussion_r19521513 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -128,7 +128,7 @@ private[spark] class ShuffleMapTask( }

[GitHub] spark pull request: [SPARK-4109][CORE] Correctly deserialize Task....

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2971#discussion_r19521525 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -128,7 +128,7 @@ private[spark] class ShuffleMapTask( }

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60878689 [Test build #22433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22433/consoleFull) for PR 2944 at commit [`bc1e675`](https://githu

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60878629 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60878626 [Test build #22427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22427/consoleFull) for PR 2753 at commit [`cadfd28`](https://gith

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2944#discussion_r19521449 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ThreadDumpPage.scala --- @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-3968 Use parquet-mr filter2 api in spark...

2014-10-28 Thread saucam
Github user saucam commented on a diff in the pull request: https://github.com/apache/spark/pull/2841#discussion_r19521440 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala --- @@ -423,10 +436,8 @@ private[parquet] class FilteringParquetRo

[GitHub] spark pull request: [SPARK-4129][MLlib] Performance tuning in Mult...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2992#issuecomment-60878201 [Test build #22432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22432/consoleFull) for PR 2992 at commit [`2b5e882`](https://githu

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2944#discussion_r19521370 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -412,6 +415,17 @@ class BlockManagerMasterActor(val isLocal: Boo

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60878083 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [Build] Uploads HiveCompatibilitySuite logs

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2993#issuecomment-60878064 [Test build #22431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22431/consoleFull) for PR 2993 at commit [`8e6247f`](https://githu

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60878081 [Test build #22430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22430/consoleFull) for PR 2944 at commit [`f4ac1c1`](https://gith

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60877993 I've rewritten this patch so that thread dumps are triggered on-demand using a new driver -> executor RPC channel. There are a few hacks involved in setting this up, m

[GitHub] spark pull request: [SPARK-611] [WIP] Display executor thread dump...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2944#issuecomment-60877916 [Test build #22430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22430/consoleFull) for PR 2944 at commit [`f4ac1c1`](https://githu

[GitHub] spark pull request: [Build] Uploads HiveCompatibilitySuite logs

2014-10-28 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/2993 [Build] Uploads HiveCompatibilitySuite logs In addition to unit-tests.log files, also upload failure output files generated by `HiveCompatibilitySuite` to Jenkins master. These files can be very

[GitHub] spark pull request: [Spark-4060] [MLlib] exposing special rdd func...

2014-10-28 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/2907#issuecomment-60877866 Yes - treeAggregate is very useful -- In fact I was going to suggest moving it to the core RDD API. Any reasons to not do that ? --- If your project is set up for it, y

[GitHub] spark pull request: [SPARK-4108][SQL] Fixed usage of deprecated in...

2014-10-28 Thread anantasty
Github user anantasty commented on the pull request: https://github.com/apache/spark/pull/2970#issuecomment-60877768 @marmbrus just did that. Thanks for that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-4028][Streaming] ReceivedBlockHandler i...

2014-10-28 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2940#issuecomment-60877622 I mentioned this in the earlier in the original thread. This is a tradeoff between generality and type checking. I want the code in ReceiverSupervisorImpl, and ReceivedBlock

[GitHub] spark pull request: [SPARK-4129][MLlib] Performance tuning in Mult...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2992#issuecomment-60877361 [Test build #22429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22429/consoleFull) for PR 2992 at commit [`ebe3e74`](https://githu

[GitHub] spark pull request: [SPARK-4129][MLlib] Performance tuning in Mult...

2014-10-28 Thread dbtsai
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/2992 [SPARK-4129][MLlib] Performance tuning in MultivariateOnlineSummarizer In MultivariateOnlineSummarizer, breeze's activeIterator is used to loop through the nonZero elements in the vector. However, ac

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60876198 [Test build #22428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22428/consoleFull) for PR 2942 at commit [`9f7aea9`](https://githu

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60875897 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60875890 [Test build #22423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22423/consoleFull) for PR 2746 at commit [`a6a30f2`](https://gith

[GitHub] spark pull request: [Spark-4060] [MLlib] exposing special rdd func...

2014-10-28 Thread javadba
Github user javadba commented on the pull request: https://github.com/apache/spark/pull/2907#issuecomment-60875650 RE: use case. We are considering to use the treeAggregate function within the implementation of SpectralClustering. In addition it was noted that the EigenvalueDecompos

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60875589 [Test build #22427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22427/consoleFull) for PR 2753 at commit [`cadfd28`](https://githu

[GitHub] spark pull request: [SPARK-4062][Streaming]Add ReliableKafkaReceiv...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2991#issuecomment-60875582 [Test build #22425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22425/consoleFull) for PR 2991 at commit [`5cc4cb1`](https://githu

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60875507 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60875506 [Test build #22426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22426/consoleFull) for PR 2942 at commit [`374a706`](https://gith

[GitHub] spark pull request: [SPARK-4113] [SQL] [PySpark] type conversionfo...

2014-10-28 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2973#issuecomment-60875464 thanks, update the link --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4062][Streaming]Add ReliableKafkaReceiv...

2014-10-28 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2991#issuecomment-60875453 Hi @tdas , would you mind taking a look at this? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60875441 [Test build #22426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22426/consoleFull) for PR 2942 at commit [`374a706`](https://githu

[GitHub] spark pull request: [SPARK-4062][Streaming]Add ReliableKafkaReceiv...

2014-10-28 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/2991 [SPARK-4062][Streaming]Add ReliableKafkaReceiver in Spark Streaming Kafka connector Add ReliableKafkaReceiver in Kafka connector to prevent data loss if WAL in Spark Streaming is enabled. Details

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60875179 That is a pass with netty turned on. Now I am turning it off for preparation to merge. --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60874822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60874816 [Test build #22422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22422/consoleFull) for PR 2746 at commit [`c60ec33`](https://gith

[GitHub] spark pull request: [SPARK-4094][CORE] checkpoint should still be ...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2956#issuecomment-60874726 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-4094][CORE] checkpoint should still be ...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2956#issuecomment-60874724 **[Test build #22419 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22419/consoleFull)** for PR 2956 at commit [`b8633c7`](https://git

[GitHub] spark pull request: [EC2] Don't change working dir on user

2014-10-28 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2988#issuecomment-60874638 Phantom new classes _again_? :angry: I'll look into this tomorrow. Thought it was a resolved issue... --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2990#issuecomment-60874554 The failed test from streaming is a known flaky test. @tdas I've merged this one into master (because it will speed up Jenkins builds). --- If your project is se

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2990 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2990#issuecomment-60874351 [Test build #22424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22424/consoleFull) for PR 2990 at commit [`c58f070`](https://gith

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2990#issuecomment-60874354 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60874155 [Test build #22421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22421/consoleFull) for PR 2753 at commit [`d7be11b`](https://gith

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60874159 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]

2014-10-28 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-60873448 Had an offline discussion with @freeman-lab . We decided to introduce the concept of `timeUnit` to describe decay. A `timeUnit` (like a second) could be either a `batch` o

[GitHub] spark pull request: [EC2] Don't change working dir on user

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2988#issuecomment-60873361 [Test build #22420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22420/consoleFull) for PR 2988 at commit [`ce071fc`](https://gith

[GitHub] spark pull request: [EC2] Don't change working dir on user

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2988#issuecomment-60873366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3826][SQL]enable hive-thriftserver to s...

2014-10-28 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2685#issuecomment-60872647 @scwf How about Spark dev related environment variable settings on your host? I unset `SPARK_PREPEND_CLASSES` and set `SPARK_TESTING` to 1 before running `core/test` (w

[GitHub] spark pull request: [SPARK-3826][SQL]enable hive-thriftserver to s...

2014-10-28 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2685#issuecomment-60871776 Local testing each test suite individually doesn't fail, guess another case related to test suite execution order. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2542#issuecomment-60871225 [Test build #22417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22417/consoleFull) for PR 2542 at commit [`efc2df3`](https://gith

[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2542#issuecomment-60871231 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2990#issuecomment-60871216 [Test build #22424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22424/consoleFull) for PR 2990 at commit [`c58f070`](https://githu

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60870919 [Test build #22423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22423/consoleFull) for PR 2746 at commit [`a6a30f2`](https://githu

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2990#issuecomment-60870874 LGTM. This patch decreases build time by roughly 10 minutes. Good job! --- If your project is set up for it, you can reply to this email and have your reply appear on Gi

[GitHub] spark pull request: [FIX] disable benchmark code

2014-10-28 Thread mengxr
GitHub user mengxr opened a pull request: https://github.com/apache/spark/pull/2990 [FIX] disable benchmark code I forgot to disable the benchmark code in #2937, which increased the Jenkins build time by couple minutes. @aarondav You can merge this pull request into a Git

[GitHub] spark pull request: delete jetty 6.1.26 form spark package

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2989#issuecomment-60870455 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

2014-10-28 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2840#issuecomment-60870350 I believe `missing` can't be negative, because that would imply `maxExecutors < numPendingAllocate + numExecutorsRunning`. We only decrease `maxExecutors` from kill ex

[GitHub] spark pull request: delete jetty 6.1.26 form spark package

2014-10-28 Thread KaiXinXiaoLei
GitHub user KaiXinXiaoLei opened a pull request: https://github.com/apache/spark/pull/2989 delete jetty 6.1.26 form spark package In spark code, jetty version is 8.1.14.v20131031 (see ./pom.xml). But when building, the jetty-6.1.26 and jetty-util-6.1.26 is packaged in spark-assembl

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60870010 [Test build #22422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22422/consoleFull) for PR 2746 at commit [`c60ec33`](https://githu

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

2014-10-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2840#discussion_r19519070 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala --- @@ -66,7 +66,16 @@ private[spark] object CoarseGrai

[GitHub] spark pull request: [SPARK-3826][SQL]enable hive-thriftserver to s...

2014-10-28 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2685#issuecomment-60869228 @liancheng, master branch passed in your local machine? i tested with ``` sbt/sbt -Phive,hadoop-2.3 assembly sbt/sbt -Phive,hadoop-2.3 core/test ``` a

[GitHub] spark pull request: [SPARK-4028][Streaming] ReceivedBlockHandler i...

2014-10-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2940#issuecomment-60869205 The way this now we have to do runtime type checking in a bunch of places... I think it could be avoided with a fairly simple change. --- If your project is set up for

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60869230 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3795] Heuristics for dynamically scalin...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-60869227 [Test build #22416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22416/consoleFull) for PR 2746 at commit [`b00b680`](https://gith

[GitHub] spark pull request: [SPARK-4028][Streaming] ReceivedBlockHandler i...

2014-10-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2940#issuecomment-60869054 I took another pass - the main thing blocking this for me is cleaning up the type signature to not have `Option[Any]`. I made a proposal earlier and I stick by that one.

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-28 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19518754 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...

2014-10-28 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2868#discussion_r19518756 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/NodeIdCache.scala --- @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundat

[GitHub] spark pull request: [SPARK-4028][Streaming] ReceivedBlockHandler i...

2014-10-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2940#discussion_r19518752 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala --- @@ -0,0 +1,144 @@ +package org.apache.spark.stream

[GitHub] spark pull request: [SPARK-3826][SQL]enable hive-thriftserver to s...

2014-10-28 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2685#issuecomment-60868885 I'm looking into this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60868809 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3930] [SPARK-3933] Support fixed-precis...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2983#issuecomment-60868804 [Test build #22418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22418/consoleFull) for PR 2983 at commit [`4ca62cd`](https://gith

[GitHub] spark pull request: [EC2] Don't change working dir on user

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2988#issuecomment-60868345 [Test build #22420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22420/consoleFull) for PR 2988 at commit [`ce071fc`](https://githu

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60868156 [Test build #22421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22421/consoleFull) for PR 2753 at commit [`d7be11b`](https://githu

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60867974 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [EC2] Don't change working dir on user

2014-10-28 Thread nchammas
GitHub user nchammas opened a pull request: https://github.com/apache/spark/pull/2988 [EC2] Don't change working dir on user The issue was uncovered after [this discussion](https://issues.apache.org/jira/browse/SPARK-3398?focusedCommentId=14187471&page=com.atlassian.jira.plugin.syst

[GitHub] spark pull request: [SPARK-4094][CORE] checkpoint should still be ...

2014-10-28 Thread liyezhang556520
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/2956#issuecomment-60867864 RDD checkpoint should also support like this: `rdd0 = sc.makeRDD(...)` `rdd1 = rdd0.flatmap(...)` `rdd1.collect()` `rdd0.checkpoint()` `rdd1.cou

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60867860 [Test build #22415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22415/consoleFull) for PR 2753 at commit [`d5d123f`](https://gith

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60867867 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-3826][SQL]enable hive-thriftserver to s...

2014-10-28 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2685#issuecomment-60867822 I tested the master branch locally, it also failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your p

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60867311 **[Test build #22405 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22405/consoleFull)** for PR 2753 at commit [`4a204b8`](https://git

[GitHub] spark pull request: [SPARK-4094][CORE] checkpoint should still be ...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2956#issuecomment-60867321 [Test build #22419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22419/consoleFull) for PR 2956 at commit [`b8633c7`](https://githu

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-60867314 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22

[GitHub] spark pull request: [SPARK-4120][SQL] Join of multiple tables with...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2987#issuecomment-60867101 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-4120][SQL] Join of multiple tables with...

2014-10-28 Thread ravipesala
GitHub user ravipesala opened a pull request: https://github.com/apache/spark/pull/2987 [SPARK-4120][SQL] Join of multiple tables with syntax like SELECT .. FROM T1,T2,T3.. does not work in SparkSQL Right now it works for only 2 tables like below query. sql("SELECT * FROM record

  1   2   3   4   5   6   7   8   >