[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/9000#discussion_r41562570 --- Diff: core/src/test/scala/org/apache/spark/StaticMemoryManagerSuite.scala --- @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-10250][CORE] External group by to handl...

2015-10-08 Thread mccheah
Github user mccheah commented on the pull request: https://github.com/apache/spark/pull/8438#issuecomment-146674050 I get the feeling that the spilling tests I wrote are really really slow, so I may need to adjust them. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/9000#discussion_r41562951 --- Diff: core/src/test/scala/org/apache/spark/StaticMemoryManagerSuite.scala --- @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-10250][CORE] External group by to handl...

2015-10-08 Thread mccheah
Github user mccheah commented on the pull request: https://github.com/apache/spark/pull/8438#issuecomment-146673974 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10993] [SQL] Inital code generated enco...

2015-10-08 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/9019#discussion_r41565489 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala --- @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-8654][SQL] Fix Analysis exception when ...

2015-10-08 Thread dilipbiswal
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/8983#issuecomment-146682265 @marmbrus .. sorry about it. Is there a way i can look at the list of failures ? I had run : build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146691810 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-8848] [SQL] Refactors Parquet write pat...

2015-10-08 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8988#issuecomment-146691882 Since we already have an option for being compatible with Hive (the legacy mode), then we should not worry that (do not need to change anything in this PR).

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146691960 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146693831 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146693992 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1267][PYSPARK] Adds pip installer for p...

2015-10-08 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/8318#discussion_r41572924 --- Diff: python/pyspark/__init__.py --- @@ -36,6 +36,31 @@ Finer-grained cache persistence levels. """ +import os +import sys

[GitHub] spark pull request: [SPARK-9478] [ml] Add class weights to Random ...

2015-10-08 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/9008#discussion_r41573997 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala --- @@ -1211,4 +1212,34 @@ private[ml] object RandomForest extends Logging {

[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...

2015-10-08 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/8909#issuecomment-146699044 @marmbrus @yhuai @andrewor14 I should have addressed all of your comments (except lack of more docs), could you take another pass? The failed test is not related to

[GitHub] spark pull request: [SPARK-10991][ML] logistic regression training...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9037#issuecomment-146701173 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146705900 [Test build #43435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43435/consoleFull) for PR 9036 at commit

[GitHub] spark pull request: [SPARK-8848] [SQL] Refactors Parquet write pat...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8988#issuecomment-146708650 [Test build #43427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43427/console) for PR 8988 at commit

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9000#issuecomment-146713347 [Test build #43437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43437/consoleFull) for PR 9000 at commit

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41581491 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoConf.scala --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41583189 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoInputStream.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41583245 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoInputStream.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8909#issuecomment-146715405 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41584178 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoInputStream.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-11007] [SQL] Adds dictionary aware Parq...

2015-10-08 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/9040 [SPARK-11007] [SQL] Adds dictionary aware Parquet decimal converters For Parquet decimal columns that are encoded using plain-dictionary encoding, we can make the upper level converter aware of

[GitHub] spark pull request: [SPARK-10990] [SQL] improve unrolling of compl...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9016#discussion_r41584197 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala --- @@ -34,7 +34,8 @@ import org.apache.spark.unsafe.types.UTF8String

[GitHub] spark pull request: [SPARK-10599] Lower communication for block ma...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8757#issuecomment-146717274 [Test build #1864 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1864/consoleFull) for PR 8757 at commit

[GitHub] spark pull request: [SPARK-10250][CORE] External group by to handl...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8438#issuecomment-146675573 [Test build #43422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43422/consoleFull) for PR 8438 at commit

[GitHub] spark pull request: [SPARK-10412][SQL] report memory usage for tun...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/8931#issuecomment-146675441 This is blocked by https://issues.apache.org/jira/browse/SPARK-11013 Will work on that first. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-8654][SQL] Fix Analysis exception when ...

2015-10-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/8983#issuecomment-146679919 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8170][Python]Add signal handler to trap...

2015-10-08 Thread ashwinshankar77
Github user ashwinshankar77 commented on the pull request: https://github.com/apache/spark/pull/9033#issuecomment-146679774 @davies @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-8848] [SQL] Refactors Parquet write pat...

2015-10-08 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8988#issuecomment-146681921 The last build failure was caused by #8983, which broke master and has just been reverted. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARKR] [SPARK-10981] SparkR Join improvement...

2015-10-08 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/9029#issuecomment-146681852 @mfliu I think thats just a flaky test, but irrespective of that your changes should be against the current `master` branch. Right now it looks like there are a lot

[GitHub] spark pull request: [SPARK-8848] [SQL] Refactors Parquet write pat...

2015-10-08 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8988#issuecomment-146681939 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10990] [SQL] improve unrolling of compl...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9016#issuecomment-146683593 [Test build #43426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43426/consoleFull) for PR 9016 at commit

[GitHub] spark pull request: [SPARK-8654][SQL] Fix Analysis exception when ...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/8983#issuecomment-146688197 open a new one is also OK. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10988] [SQL] Reduce duplication in Aggr...

2015-10-08 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/9015#discussion_r41571056 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/utils.scala --- @@ -204,10 +144,13 @@ object Utils { } def

[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() to get...

2015-10-08 Thread olarayej
Github user olarayej commented on the pull request: https://github.com/apache/spark/pull/8984#issuecomment-146693478 @shivaram Could you share the best practices to merge the changes from the master branch into the PR branch? This looks like a very common thing and the team

[GitHub] spark pull request: [SPARK-10988] [SQL] Reduce duplication in Aggr...

2015-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9015 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146697687 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8962#issuecomment-146701389 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10991][ML] logistic regression training...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9037#issuecomment-146701153 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146701214 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146701216 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8962#issuecomment-146701289 [Test build #43431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43431/console) for PR 8962 at commit

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8962#issuecomment-146701388 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146703956 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9036#issuecomment-146704000 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41579905 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoCodec.scala --- @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41580011 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoCodec.scala --- @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-10855] [SQL] Add a JDBC dialect for Apa...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8982#issuecomment-146710467 [Test build #43436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43436/consoleFull) for PR 8982 at commit

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41581062 --- Diff: core/src/main/scala/org/apache/spark/crypto/CommonConfigurationKeys.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-11017] Support ImperativeAggregates in ...

2015-10-08 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9038#issuecomment-146712508 /cc @yhuai for review. There are a few specific questions that I'd like to clarify and I will comment on them inline. --- If your project is set up for it, you can

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41581826 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoConf.scala --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41581767 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoConf.scala --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-10993] [SQL] Inital code generated enco...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9019#discussion_r41583796 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala --- @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/9039 [SPARK-10665][SQL] Query planner infra for local physical operators - WIP You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark local-iter

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9039#issuecomment-146718422 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41585744 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoCodec.scala --- @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-10599] Lower communication for block ma...

2015-10-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/8757#issuecomment-146723162 Minor comments only. Other than that, it looks fine to me. @avulanov In your "Block-diagonal matrix multiplication" tests, do you know if data were shuffled

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9000#issuecomment-146725522 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9000#issuecomment-146725531 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9000#discussion_r41587742 --- Diff: core/src/test/scala/org/apache/spark/StaticMemoryManagerSuite.scala --- @@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9000#discussion_r41588487 --- Diff: core/src/main/scala/org/apache/spark/MemoryManager.scala --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-11017] Support ImperativeAggregates in ...

2015-10-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9038#issuecomment-146729145 YAY --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9039#issuecomment-146729855 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9039#issuecomment-146729845 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146729797 Why is this needed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/9039 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730487 [Test build #43452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43452/consoleFull) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730942 LGTM. I'd rephrase the description to say "up to a minute" since I was a little scared when reading the original one. "OMG all my apps are going to wait a minute before

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9041#discussion_r41589229 --- Diff: external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala --- @@ -93,7 +93,11 @@ private[streaming] class

[GitHub] spark pull request: [SPARK-10515] When killing executor, the pendi...

2015-10-08 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8945#issuecomment-146730978 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10990] [SPARK-11018] [SQL] improve unro...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9016#issuecomment-146732511 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-8386] [SQL]add write.mode for insertInt...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9042#issuecomment-146732536 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-8386] [SQL]add write.mode for insertInt...

2015-10-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9042#issuecomment-146732607 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

2015-10-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8962#discussion_r41593057 --- Diff: R/pkg/R/stats.R --- @@ -100,3 +100,30 @@ setMethod("corr", statFunctions <- callJMethod(x@sdf, "stat")

[GitHub] spark pull request: [SPARK-10739][Yarn] Add application attempt wi...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8857#discussion_r41593135 --- Diff: docs/running-on-yarn.md --- @@ -304,6 +304,14 @@ If you need a reference to the proper location to put log files in the YARN so t

[GitHub] spark pull request: [SPARK-10739][Yarn] Add application attempt wi...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8857#discussion_r41593098 --- Diff: docs/running-on-yarn.md --- @@ -304,6 +304,14 @@ If you need a reference to the proper location to put log files in the YARN so t

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

2015-10-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/8962#discussion_r41593053 --- Diff: R/pkg/R/stats.R --- @@ -100,3 +100,30 @@ setMethod("corr", statFunctions <- callJMethod(x@sdf, "stat")

[GitHub] spark pull request: [SPARK-10990] [SPARK-11018] [SQL] improve unro...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9016#issuecomment-146743581 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() to get...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8984#issuecomment-146744455 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() to get...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8984#issuecomment-146744379 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() to get...

2015-10-08 Thread olarayej
Github user olarayej commented on the pull request: https://github.com/apache/spark/pull/8984#issuecomment-146743755 @shivaram Yes, that was helpful. Thank you! I have done the merge already. Jenkins, could you run tests? --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-8654] [SQL] Analysis exception when usi...

2015-10-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9036#discussion_r41595318 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -304,7 +304,10 @@ object HiveTypeCoercion {

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41596018 --- Diff: core/src/main/scala/org/apache/spark/crypto/CommonConfigurationKeys.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41596083 --- Diff: core/src/main/scala/org/apache/spark/crypto/CommonConfigurationKeys.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-10665][SQL] Query planner infra for loc...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9039#issuecomment-146718423 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41584604 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoOutputStream.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41585923 --- Diff: core/src/main/scala/org/apache/spark/crypto/CryptoInputStream.scala --- @@ -0,0 +1,439 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
GitHub user harishreedharan opened a pull request: https://github.com/apache/spark/pull/9041 [SPARK-11019][streaming][flume] Gracefully shutdown Flume receiver th… …reads. Wait for a minute for the receiver threads to shutdown before interrupting them. You can merge

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41586007 --- Diff: core/src/main/scala/org/apache/spark/crypto/Decryptor.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [SPARK-10914] UnsafeRow serialization breaks w...

2015-10-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9030#issuecomment-146724781 Merging this in master & branch-1.5. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41586546 --- Diff: core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala --- @@ -48,14 +53,29 @@ private[spark] class BlockStoreShuffleReader[K,

[GitHub] spark pull request: [SPARK-10914] UnsafeRow serialization breaks w...

2015-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9030 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-10599] Lower communication for block ma...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8757#issuecomment-146724888 [Test build #1864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1864/console) for PR 8757 at commit

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8880#discussion_r41586581 --- Diff: core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala --- @@ -48,14 +53,29 @@ private[spark] class BlockStoreShuffleReader[K,

[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...

2015-10-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8880#issuecomment-146726420 Hi @winningsix, I need to read the buffer handling code again, but it could use some cleanup to make it easier to follow. Also, just pointing out style

[GitHub] spark pull request: [SPARK-10767][PYSPARK] Make pyspark shared par...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9017#issuecomment-146726586 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10990] [SQL] improve unrolling of compl...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9016#issuecomment-146726579 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10956] Common MemoryManager interface f...

2015-10-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9000#discussion_r41587949 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -409,40 +397,42 @@ private[spark] class MemoryStore(blockManager:

<    1   2   3   4   5   6   7   8   9   10   >