[GitHub] spark pull request: [SPARK-2325] Utils.getLocalDir had better chec...

2014-07-05 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/1281#issuecomment-48102838 Hi @YanTangZhai, with the merge of https://github.com/apache/spark/pull/1274 is this change still needed? --- If your project is set up for it, you can reply to this emai

[GitHub] spark pull request: [SPARK-2373]RDD add span function (split an RD...

2014-07-05 Thread YanjieGao
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1306#issuecomment-48100374 Thanks ,I optimize the code so it only evaluates the function once .Other comments are on JIRA --- If your project is set up for it, you can reply to this email and ha

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread YanjieGao
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1150#issuecomment-48099853 Thanks a lot, I have reformat the code style. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: [SQL] Prototype implementation of size estimat...

2014-07-05 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r14573147 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -26,6 +26,17 @@ import org.apache.spark.sq

[GitHub] spark pull request: [SQL] Prototype implementation of size estimat...

2014-07-05 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r14573140 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -61,8 +61,6 @@ private[sql] abstract class SparkStrategi

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48098033 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16356/ --- If your project

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48098032 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48097436 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48097437 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16355/ --- If your project

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48096080 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48096085 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1308#discussion_r14572630 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala --- @@ -448,9 +448,9 @@ class JsonSuite extends QueryTest { }

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1150#issuecomment-48095551 > Thanks ,I add three blank after that line ,Now there aren't red lines. If there is trailing whitespace we should remove it. It is okay if it is not code you a

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572610 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -385,6 +385,17 @@ class SQLQuerySuite extends QueryTest { sql("S

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572609 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -385,6 +385,17 @@ class SQLQuerySuite extends QueryTest { sql("S

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572607 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala --- @@ -220,3 +220,16 @@ case class Except(left: SparkPlan, right: Spar

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572602 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala --- @@ -140,6 +141,7 @@ class SqlParser extends StandardTokenParsers with

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572606 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -282,6 +282,8 @@ private[sql] abstract class SparkStrategies ex

[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...

2014-07-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1150#discussion_r14572604 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -282,6 +282,8 @@ private[sql] abstract class SparkStrategies ex

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48095474 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1308#issuecomment-48095478 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2375][SQL] JSON schema inference may no...

2014-07-05 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/1308 [SPARK-2375][SQL] JSON schema inference may not resolve type conflicts correctly for a field inside an array of structs For example, for ``` {"array": [{"field":214748364700}, {"field":1}]}

[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...

2014-07-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-48093749 Thanks! I've merge this into master and we'll back port it into 1.0 after the 1.0.1 release. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...

2014-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-2366] [SQL] Add column pruning for the ...

2014-07-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1301#issuecomment-48093617 Thanks! Merged into master and 1.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2366] [SQL] Add column pruning for the ...

2014-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1301 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: SPARK-1216. Add a OneHotEncoder for handling c...

2014-07-05 Thread karlhigley
Github user karlhigley commented on the pull request: https://github.com/apache/spark/pull/304#issuecomment-48093398 This looked useful, so I tried it out. It works as expected so long as the feature array is an `Array[Any]`.If the features are all categorical, then the feature a

[GitHub] spark pull request: [SPARK-2334] fix rdd.id() Attribute Error

2014-07-05 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1276#issuecomment-48090143 The cause seems to be that when you do operations like map() followed by map(), you get a PipelinedRDD, which does not necessarily have an underlying Java RDD until you ac

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083786 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16354/ --- If your project

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083785 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083120 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083121 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16353/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083048 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48083043 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48082949 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1307#issuecomment-48082948 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2374] Path pattern matching for GraphX

2014-07-05 Thread ankurdave
GitHub user ankurdave opened a pull request: https://github.com/apache/spark/pull/1307 [SPARK-2374] Path pattern matching for GraphX Based on a [request](http://apache-spark-user-list.1001560.n3.nabble.com/Graphx-traversal-and-merge-interesting-edges-td8788.html) on the mailing lis

[GitHub] spark pull request: [SPARK-2373]RDD add span function (split an RD...

2014-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1306#issuecomment-48080698 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-2373]RDD add span function (split an RD...

2014-07-05 Thread YanjieGao
GitHub user YanjieGao opened a pull request: https://github.com/apache/spark/pull/1306 [SPARK-2373]RDD add span function (split an RDD to two RDD based on user's function)] def span(p: T => Boolean): (RDD[T], RDD[T]) Splits this RDD into a prefix/suffix pair according to a pred