[GitHub] spark pull request: [SPARK-1495][SQL]add support for left semi joi...

2014-06-07 Thread adrian-wang
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/837#discussion_r13522951 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -119,6 +119,11 @@ object HashFilteredJoin extends Loggi

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45430013 Hi Matei, since we already have `ShuffleWriter` and `ShuffleManager`, do we still need to keep `ShuffleBlockManager`, I think two functionalities of `ShuffleBlockManage

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429980 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1495][SQL]add support for left semi joi...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/837#issuecomment-45429982 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429981 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15537/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1495][SQL]add support for left semi joi...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/837#issuecomment-45429745 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429744 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429746 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429709 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429514 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429515 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15538/ --- If your project is set up for it, you can

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429513 QA results for PR 1008:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see https://amplab.cs.berke

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429499 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429497 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429482 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429424 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429427 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429416 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429386 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429387 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15536/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429378 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1009#issuecomment-45429375 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2044] Pluggable interface for shuffles

2014-06-07 Thread mateiz
GitHub user mateiz opened a pull request: https://github.com/apache/spark/pull/1009 [SPARK-2044] Pluggable interface for shuffles This is a first cut at moving shuffle logic behind a pluggable interface, as described at https://issues.apache.org/jira/browse/SPARK-2044, to let us mo

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429325 ```QA results for PR 1008:- This patch merges cleanly- This patch adds the following public classes (experimental):class ThisIsAnNewClass extends Logging {```For mor

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429314 For more information see --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429302 QA results for PR 1008:- This patch merges cleanly- This patch adds the following public classes (experimental):class ThisIsAnNewClass extends Logging {For more info

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429293 QA results for PR 1008:- This patch did not merge cleanlyFor more information see --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429301 QA results for PR 1008:- This patch merges cleanly- This patch adds no public classesFor more information see --- If your project is set up for it, you can reply t

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429299 QA results for PR 1008:- This patch did not merge cleanlyFor more information see --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429286 QA results for PR 1008:- This patch did not merge cleanlyFor more information see --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429187 QA results for PR 1008:- This patch merges cleanly- This patch adds the following public classes (experimental):class ThisIsAnNewClass extends Logging {For more info

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429190 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15535/ --- If your project is set up for it, you can

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429189 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429172 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429176 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429125 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15534/ --- If your project is set up for it, you can

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429124 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429109 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429111 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45429074 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428919 QA results for PR :- This patch did not merge cleanly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428922 QA results for PR :- This patch merges cleanly- This patch adds the following public classes:class ThisIsAnNewClass extends Logging { --- If your project is set up

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428882 QA results for PR .- This patch merges cleanly-This patch adds the following public classes:class ThisIsAnNewClass extends Logging { --- If your project is set up f

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell-api
Github user pwendell-api commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428754 This isa comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428606 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15533/ --- If your project is set up for it, you can

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428605 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428591 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428587 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428577 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...

2014-06-07 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/999#discussion_r13522696 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala --- @@ -0,0 +1,364 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428535 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15532/ --- If your project is set up for it, you can

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428534 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428524 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428522 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1008#issuecomment-45428501 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Testing something

2014-06-07 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/1008 Testing something This is to test a QA automation feature. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pwendell/spark better-qa-messages Al

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-07 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45425913 Sorry, have been traveling this week. Not sure why it passed locally for me, but I'll post a new patch early next week. > On Jun 7, 2014, at 7:33 PM, Patrick Wendell

[GitHub] spark pull request: SPARK-554. Add aggregateByKey.

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/705#issuecomment-45424002 @sryza any interest in fixing up the test? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-2056 Set RDD name to input path

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/992#issuecomment-45423853 I went ahead and merged this to test the new script. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: SPARK-2056 Set RDD name to input path

2014-06-07 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/992 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1007 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK

2014-06-07 Thread rezazadeh
Github user rezazadeh commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-45423737 @vrilleup I think the binary compatibility issue is because of the change in method signature. Even though you have a default argument it changes the interface. Try addi

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1007#issuecomment-45422907 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15531/ --- If your project

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1007#issuecomment-45422906 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1007#issuecomment-45422070 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1007#issuecomment-45422071 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1007#issuecomment-45422011 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and w

[GitHub] spark pull request: HOTFIX: Support empty body in merge script

2014-06-07 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/1007 HOTFIX: Support empty body in merge script Discovered in #992 You can merge this pull request into a Git repository by running: $ git pull https://github.com/pwendell/spark hotfix Alternative

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-06-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/455#issuecomment-45421855 @mateiz I see - if this is required for supporting arbitrary input types, I guess it's the only way. I'm hoping in the future though most people don't have to write their

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-06-07 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/455#discussion_r13521516 --- Diff: docs/programming-guide.md --- @@ -378,11 +378,88 @@ Some notes on reading files with Spark: * The `textFile` method also takes an optional

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45421535 One reason we had to add @transient lazy val is due to the lack of an init method on each partition for operators. I think there are benefits of adding that - it makes clear

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1004 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45421524 Agreed, those should all be private[spark] so won't show up. On Jun 7, 2014 2:16 PM, "Reynold Xin" wrote: > I think it could go both ways for the setters. How

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45421512 I'm going to merge this. YOu can test this easily by looking at the log. Spark tells you the size of the task closure and how long it takes to serialize each of them in the

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45421520 Merged in master & branch-1.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-06-07 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/455#discussion_r13521493 --- Diff: docs/programming-guide.md --- @@ -378,11 +378,88 @@ Some notes on reading files with Spark: * The `textFile` method also takes an optional

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45421494 Is there an easy way to measure that? Either way it was wrong before and I don't think making it possible to plan on the slaves is worth the effort. Doing so

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45421436 I think it could go both ways for the setters. However, we should not add all the accessors (e.g. numPartitions, codeGen) to SQLContext, since that is just polluting the name

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45421226 How big does the closure size increase by? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-06-07 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/455#issuecomment-45420441 (Also, once Spark SQL supports a given interface, or even before, we can simply leave out patches that add new Converters. There's no need to provide a huge collection of t

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-06-07 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/455#issuecomment-45420334 I personally think Converter is good to keep because it's the only way to expose arbitrary InputFormats to Python. In Java and Scala, you can always call SparkContext.hadoo

[GitHub] spark pull request: [SPARK-1994][SQL] Weird data corruption bug wh...

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1004#issuecomment-45419626 @rxin I added https://issues.apache.org/jira/browse/SPARK-2068 to track other places where we need to fix this, but we should probably just merge this one right away.

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13521110 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -101,7 +101,7 @@ abstract class LeafNode extends L

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45419468 I agree that it would be silly to expose only the setters but not the other methods. However, the extra syntactic overhead of typing `.sqlConf.` seems unnecessary since

[GitHub] spark pull request: [SQL] Simple framework for debugging query exe...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1005#issuecomment-45419375 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15530/ --- If your project

[GitHub] spark pull request: [SQL] Simple framework for debugging query exe...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1005#issuecomment-45419374 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45419306 Regarding this particular question, I had a brief discussion with @rxin before. We agreed that it seemed superfluous to only expose a setter without exposing

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45419171 This is looking pretty good. Thanks for working on it! I think the only remaining public interface question that I have is if we should expose the sqlConf to the

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13521048 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala --- @@ -138,6 +138,7 @@ abstract class HiveComparisonTest

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13521020 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-07 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13521013 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -101,7 +101,7 @@ abstract class LeafNode extends L

[GitHub] spark pull request: [SPARK-1495][SQL]add support for left semi joi...

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/837#issuecomment-45418856 (I deleted my earlier comment because I found a mistake) I think this is looking pretty good, but we should at least add one test for the Broadcast Nested Loop ve

[GitHub] spark pull request: [SPARK-2067] use relative path for Spark logo ...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1006#issuecomment-45418679 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-2067] use relative path for Spark logo ...

2014-06-07 Thread nevillelyh
GitHub user nevillelyh opened a pull request: https://github.com/apache/spark/pull/1006 [SPARK-2067] use relative path for Spark logo in UI You can merge this pull request into a Git repository by running: $ git pull https://github.com/nevillelyh/spark gh/SPARK-2067 Alternati

[GitHub] spark pull request: [SPARK-1495][SQL]add support for left semi joi...

2014-06-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/837#issuecomment-45418467 I think this is looking pretty good. One problem is that there are no tests for the nested loop version. I tried adding this to SQLQuerySuite: ```scala te

[GitHub] spark pull request: [SPARK-1704][SQL] Fully support EXPLAIN comman...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1003#issuecomment-45418220 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1704][SQL] Fully support EXPLAIN comman...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1003#issuecomment-45418221 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15528/ --- If your project

[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...

2014-06-07 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/999#discussion_r13520869 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonTable.scala --- @@ -0,0 +1,364 @@ +/* + * Licensed to the Apache Software Foundatio

[GitHub] spark pull request: [SQL] Simple framework for debugging query exe...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1005#issuecomment-45417634 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SQL] Simple framework for debugging query exe...

2014-06-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1005#issuecomment-45417630 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

  1   2   >