[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-05-21 Thread l15k4
Github user l15k4 commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-220782023 Guys do you have a rough guess about when bucketing is to be implemented for `org.apache.spark.sql.DataFrameWriter#save` ? --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49034956 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelation.scala --- @@ -125,7 +125,7 @@ private[sql] case class

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10498 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169512619 I'm going to merge this. @cloud-fan can you create a follow-up pr to address some of the comments above? --- If your project is set up for it, you can reply to this emai

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49030540 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONRelation.scala --- @@ -34,13 +34,13 @@ import org.apache.spark.rdd.R

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49030037 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelation.scala --- @@ -125,7 +125,7 @@ private[sql] case cl

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49026005 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,121 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49025627 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONRelation.scala --- @@ -34,13 +34,13 @@ import org.apache.spark.rdd.RDD

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169495337 @cloud-fan Can we write bucketed table without partitions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49023295 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49023042 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPartitio

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49022978 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPartitio

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49022763 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPart

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49021259 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169444645 This looks good to me to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49003254 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -342,22 +461,34 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49002988 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,121 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49002810 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,121 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49002660 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,117 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r49002565 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,121 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48996768 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala --- @@ -161,6 +161,20 @@ trait HadoopFsRelationProvider { dataSchema:

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48996772 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala --- @@ -351,7 +365,18 @@ abstract class OutputWriterFactory extends Serializabl

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169338192 **[Test build #48856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48856/consoleFull)** for PR 10498 at commit [`1afd3ee`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169317899 **[Test build #48856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48856/consoleFull)** for PR 10498 at commit [`1afd3ee`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169317848 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169317846 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48953481 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,121 @@ private[sql] class DynamicPar

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169313888 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169313893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169313791 **[Test build #48854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48854/consoleFull)** for PR 10498 at commit [`d3200cf`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169309910 **[Test build #48854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48854/consoleFull)** for PR 10498 at commit [`d3200cf`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48951242 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala --- @@ -351,7 +365,18 @@ abstract class OutputWriterFactory extends Serial

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48951109 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -335,6 +339,117 @@ private[sql] class DynamicPar

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48924160 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala --- @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48921611 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala --- @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Soft

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48895396 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala --- @@ -87,7 +91,8 @@ private[sql] class ParquetOu

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48889876 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala --- @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48889504 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -451,3 +457,147 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48889469 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -451,3 +457,147 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r4699 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -451,3 +457,147 @@ private[sql] class DynamicPartit

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r4112 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -130,6 +130,34 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread nongli
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48887994 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -130,6 +130,34 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169021962 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169021958 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169021584 **[Test build #48770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48770/consoleFull)** for PR 10498 at commit [`74bd524`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169002391 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169002388 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-169002057 **[Test build #48765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48765/consoleFull)** for PR 10498 at commit [`3ff968b`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168995524 **[Test build #48770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48770/consoleFull)** for PR 10498 at commit [`74bd524`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48842647 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,47 @@ final class DataFrameWriter private[sql](df: DataFr

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48842483 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -240,6 +241,25 @@ private[hive] class HiveMetastoreCatalog(va

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168982645 **[Test build #48765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48765/consoleFull)** for PR 10498 at commit [`3ff968b`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48823466 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -240,6 +241,23 @@ private[hive] class HiveMetastoreCatalog(va

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48822540 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFr

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48821710 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -240,6 +241,23 @@ private[hive] class HiveMetastoreCatalog(val cli

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48821401 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -240,6 +241,23 @@ private[hive] class HiveMetastoreCatalog(va

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48820931 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48820679 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFr

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48820313 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48820170 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFr

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168925986 Would be great if @davies can take a look at this too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48819162 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala --- @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48819110 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala --- @@ -239,12 +240,21 @@ object ResolvedDataSource exten

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48819115 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -247,7 +252,7 @@ private[sql] class DefaultWriterCont

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48819093 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48819062 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/bucket.scala --- @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818992 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala --- @@ -219,16 +231,22 @@ class CommitFailureTestRelation( ove

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818983 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala --- @@ -151,6 +151,7 @@ case class CreateMetastoreDataSource( ta

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818977 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -596,6 +614,7 @@ private[hive] class HiveMetastoreCatalog(val clie

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818942 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -240,6 +241,23 @@ private[hive] class HiveMetastoreCatalog(val cli

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818904 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala --- @@ -33,7 +33,7 @@ import org.apache.spark.sql.catalyst.{CatalystTypeConvert

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818865 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala --- @@ -192,7 +192,14 @@ private[sql] case class PreWriteCheck(catalog

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168924177 If I'm looking at it correctly, writeRows function is almost 500 loc line. Can we break it into different logical parts and create functions for each of them? --- If yo

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818640 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala --- @@ -312,14 +318,35 @@ private[sql] class DynamicPartitio

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818408 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -189,13 +220,43 @@ final class DataFrameWriter private[sql](df: DataFrame)

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818349 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -117,7 +117,7 @@ final class DataFrameWriter private[sql](df: DataFrame) {

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48818260 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DDLParser.scala --- @@ -109,6 +109,7 @@ class DDLParser(parseQuery: String => Logi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168921838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168921832 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168921702 **[Test build #48737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48737/consoleFull)** for PR 10498 at commit [`3df61dc`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168907148 **[Test build #48737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48737/consoleFull)** for PR 10498 at commit [`3df61dc`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168878213 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168878214 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168877974 **[Test build #48701 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48701/consoleFull)** for PR 10498 at commit [`6e3c1c0`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168859783 **[Test build #48701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48701/consoleFull)** for PR 10498 at commit [`6e3c1c0`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/10498#discussion_r48747698 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala --- @@ -177,3 +179,36 @@ case class Crc32(child: Expression) ex

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168717379 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168717374 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168716880 **[Test build #48663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48663/consoleFull)** for PR 10498 at commit [`70ebd69`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168702656 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168703688 The failed test cases can be passed locally, seems it's because `TestHiveContext.warehousePath` has some problem at jenkins machines. cc @JoshRosen @yhuai --- If y

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168702652 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168702433 **[Test build #48662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48662/consoleFull)** for PR 10498 at commit [`e3c3728`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168696115 **[Test build #48663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48663/consoleFull)** for PR 10498 at commit [`70ebd69`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168685889 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168685716 **[Test build #48659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48659/consoleFull)** for PR 10498 at commit [`21e0c48`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168685886 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168683054 **[Test build #48662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48662/consoleFull)** for PR 10498 at commit [`e3c3728`](https://gi

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168675138 **[Test build #48658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48658/consoleFull)** for PR 10498 at commit [`ba23292`](https://g

[GitHub] spark pull request: [SPARK-12539][SQL] support writing bucketed ta...

2016-01-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10498#issuecomment-168675283 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

  1   2   >