spark git commit: [SPARK-15792][SQL] Allows operator to change the verbosity in explain output

2016-06-06 Thread lian
68 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5f731d68 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5f731d68 Branch: refs/heads/master Commit: 5f731d6859c4516941e5f90c99c966ef76268864 Parents: 0e0904a Author: Sean Zhong <seanzh...@databricks.com> Authored: Mon Jun 6

spark git commit: [SPARK-15632][SQL] Typed Filter should NOT change the Dataset schema

2016-06-06 Thread lian
15632. (cherry picked from commit 0e0904a2fce3c4447c24f1752307b6d01ffbd0ad) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a5bec5b8 Tree: http://git-wip-us.apache.org/repos/a

spark git commit: [SPARK-15632][SQL] Typed Filter should NOT change the Dataset schema

2016-06-06 Thread lian
h: refs/heads/master Commit: 0e0904a2fce3c4447c24f1752307b6d01ffbd0ad Parents: c409e23 Author: Sean Zhong <seanzh...@databricks.com> Authored: Mon Jun 6 22:40:21 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Mon Jun 6 22:40:21 2016 -0700 --

spark git commit: [SPARK-15657][SQL] RowEncoder should validate the data type of input object

2016-06-05 Thread lian
t;wenc...@databricks.com> Closes #13401 from cloud-fan/bug. (cherry picked from commit 30c4774f33fed63b7d400d220d710fb432f599a8) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/38a6

spark git commit: [SPARK-15657][SQL] RowEncoder should validate the data type of input object

2016-06-05 Thread lian
/30c4774f Branch: refs/heads/master Commit: 30c4774f33fed63b7d400d220d710fb432f599a8 Parents: 8a91105 Author: Wenchen Fan <wenc...@databricks.com> Authored: Sun Jun 5 15:59:52 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Sun Jun

spark git commit: [SPARK-15140][SQL] make the semantics of null input object for encoder clear

2016-06-03 Thread lian
ry picked from commit 11c83f83d5172167cb64513d5311b4178797d40e) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7315acf8 Tree: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-15140][SQL] make the semantics of null input object for encoder clear

2016-06-03 Thread lian
er Commit: 11c83f83d5172167cb64513d5311b4178797d40e Parents: 28ad0f7 Author: Wenchen Fan <wenc...@databricks.com> Authored: Fri Jun 3 14:28:19 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Jun 3 14:28:19 2016 -0700 ---

spark git commit: [SPARK-15547][SQL] nested case class in encoder can have different number of fields from the real schema

2016-06-03 Thread lian
pache.org/repos/asf/spark/tree/61b80d55 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/61b80d55 Branch: refs/heads/master Commit: 61b80d552aafb262b5f817f7bc9c0acd0328715b Parents: eb10b48 Author: Wenchen Fan <wenc...@databricks.com> Authored: Fri Jun 3 14:26:24 2016 -0700 Committer

spark git commit: [SPARK-15494][SQL] encoder code cleanup

2016-06-03 Thread lian
nal` for this purpose. (serializer expression still use `BoundReference`, we can replace it with `GetColumnByOrdinal` in follow-ups) ## How was this patch tested? existing test Author: Wenchen Fan <wenc...@databricks.com> Author: Cheng Lian <l...@databricks.com> Closes #13269 from cloud-fa

spark git commit: [SPARK-15494][SQL] encoder code cleanup

2016-06-03 Thread lian
nal` for this purpose. (serializer expression still use `BoundReference`, we can replace it with `GetColumnByOrdinal` in follow-ups) ## How was this patch tested? existing test Author: Wenchen Fan <wenc...@databricks.com> Author: Cheng Lian <l...@databricks.com> Closes #13269 from cloud-fa

spark git commit: [SPARK-14959][SQL] handle partitioned table directories in distributed filesystem

2016-06-02 Thread lian
park/diff/76aa45d3 Branch: refs/heads/master Commit: 76aa45d359d034e9ccaac64b36738d47e1e42f2c Parents: 6dde274 Author: Xin Wu <xi...@us.ibm.com> Authored: Thu Jun 2 22:49:17 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Jun 2 22:49:17 2016 -0700

spark git commit: [SPARK-14959][SQL] handle partitioned table directories in distributed filesystem

2016-06-02 Thread lian
13463 from xwu0226/SPARK-14959. (cherry picked from commit 76aa45d359d034e9ccaac64b36738d47e1e42f2c) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1e13d09c Tree: http://git-

spark git commit: [SPARK-15733][SQL] Makes the explain output less verbose by hiding some verbose output like None, null, empty List, and etc.

2016-06-02 Thread lian
and ``` ## How was this patch tested? Manual test. Author: Sean Zhong <seanzh...@databricks.com> Closes #13470 from clockfly/verbose_breakdown_4. (cherry picked from commit 6dde27404cb3d921d75dd6afca4b383f9df5976a) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.a

spark git commit: [SPARK-15732][SQL] better error message when use java reserved keyword as field name

2016-06-02 Thread lian
hor: Wenchen Fan <wenc...@databricks.com> Closes #13485 from cloud-fan/java. (cherry picked from commit 6323e4bd763eafe23c619a89552c00e1244e4690) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [SPARK-15732][SQL] better error message when use java reserved keyword as field name

2016-06-02 Thread lian
org/repos/asf/spark/diff/6323e4bd Branch: refs/heads/master Commit: 6323e4bd763eafe23c619a89552c00e1244e4690 Parents: d1c1fbc Author: Wenchen Fan <wenc...@databricks.com> Authored: Thu Jun 2 18:13:04 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Jun

spark git commit: [SPARK-15734][SQL] Avoids printing internal row in explain output

2016-06-02 Thread lian
Sean Zhong <seanzh...@databricks.com> Closes #13471 from clockfly/verbose_breakdown_5. (cherry picked from commit 985d532812cf176d0e12b799c723f917282b6813) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip

spark git commit: [SPARK-15734][SQL] Avoids printing internal row in explain output

2016-06-02 Thread lian
git-wip-us.apache.org/repos/asf/spark/diff/985d5328 Branch: refs/heads/master Commit: 985d532812cf176d0e12b799c723f917282b6813 Parents: 4315427 Author: Sean Zhong <seanzh...@databricks.com> Authored: Thu Jun 2 16:21:33 2016 -0700 Committer: Cheng Lian <l

spark git commit: [SPARK-15719][SQL] Disables writing Parquet summary files by default

2016-06-02 Thread lian
Author: Cheng Lian <l...@databricks.com> Closes #13455 from liancheng/spark-15719-disable-parquet-summary-files. (cherry picked from commit 431542765785304edb76a19885fbc5f9b8ae7d64) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARK-15441][SQL] support null object in Dataset outer-join

2016-06-01 Thread lian
om commit 8640cdb836b4964e4af891d9959af64a2e1f304e) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/91812226 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/91812226 Diff: http://git-wip-us.apache.org/

spark git commit: [SPARK-15441][SQL] support null object in Dataset outer-join

2016-06-01 Thread lian
: Wenchen Fan <wenc...@databricks.com> Authored: Wed Jun 1 16:16:54 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Wed Jun 1 16:16:54 2016 -0700 -- .../catalyst/encoders/ExpressionEncoder.scala

spark git commit: [SPARK-9876] [BRANCH-2.0] Revert "[SPARK-9876][SQL] Update Parquet to 1.8.1."

2016-06-01 Thread lian
epos/asf/spark/diff/8cdc0d4d Branch: refs/heads/branch-2.0 Commit: 8cdc0d4da6324f48e1a25496dc4bc7635d9472ef Parents: e033fd5 Author: Yin Huai <yh...@databricks.com> Authored: Wed Jun 1 16:05:13 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed

spark git commit: [SPARK-15269][SQL] Removes unexpected empty table directories created while creating external Spark SQL data sourcet tables.

2016-06-01 Thread lian
408=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15297408 ## How was this patch tested? 1. A new test case is added in `HiveQuerySuite` for this case 2. Updated `ShowCreateTableSuite` to use the same table name in all test cases. (This is how I hit this issue at the first place.) Author: Cheng Lian <l...@databric

spark git commit: [SPARK-15269][SQL] Removes unexpected empty table directories created while creating external Spark SQL data sourcet tables.

2016-06-01 Thread lian
ent-tabpanel#comment-15297408 ## How was this patch tested? 1. A new test case is added in `HiveQuerySuite` for this case 2. Updated `ShowCreateTableSuite` to use the same table name in all test cases. (This is how I hit this issue at the first place.) Author: Cheng Lian <l...@databric

spark git commit: [SPARK-14343][SQL] Proper column pruning for text data source

2016-06-01 Thread lian
umn is not requested. This may happen when only partitioning column(s) are requested for a partitioned text table. ## How was this patch tested? New test case added in `TextSuite`. Author: Cheng Lian <l...@databricks.com> Closes #13431 from liancheng/spark-14343-partitioned-text-table.

spark git commit: [SPARK-14343][SQL] Proper column pruning for text data source

2016-06-01 Thread lian
umn is not requested. This may happen when only partitioning column(s) are requested for a partitioned text table. ## How was this patch tested? New test case added in `TextSuite`. Author: Cheng Lian <l...@databricks.com> Closes #13431 from liancheng/spark-14343-partitioned-text-table. Proj

spark git commit: [SPARK-15112][SQL] Disables EmbedSerializerInFilter for plan fragments that change schema

2016-05-30 Thread lian
tps://issues.apache.org/jira/browse/SPARK-15632 Author: Cheng Lian <l...@databricks.com> Closes #13362 from liancheng/spark-15112-corrupted-filter. (cherry picked from commit 1360a6d636dd812a27955fc85df8e0255db60dfa) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.a

spark git commit: [SPARK-15112][SQL] Disables EmbedSerializerInFilter for plan fragments that change schema

2016-05-30 Thread lian
tps://issues.apache.org/jira/browse/SPARK-15632 Author: Cheng Lian <l...@databricks.com> Closes #13362 from liancheng/spark-15112-corrupted-filter. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1360a6d6 Tree: http://git-wip-us.a

spark git commit: [SPARK-9876][SQL] Update Parquet to 1.8.1.

2016-05-27 Thread lian
tch tested? This uses the existing Parquet tests. Author: Ryan Blue <b...@apache.org> Closes #13280 from rdblue/SPARK-9876-update-parquet. (cherry picked from commit 776d183c82b424ef7c3cae30537d8afe9b9eee83) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.a

spark git commit: [SPARK-9876][SQL] Update Parquet to 1.8.1.

2016-05-27 Thread lian
org/repos/asf/spark/tree/776d183c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/776d183c Branch: refs/heads/master Commit: 776d183c82b424ef7c3cae30537d8afe9b9eee83 Parents: 019afd9 Author: Ryan Blue <b...@apache.org> Authored: Fri May 27 16:59:38 2016 -0700 Committer

spark git commit: [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows

2016-05-26 Thread lian
ing methods like `Dataset.col`. So it probably makes more sense to show full contents of the query plan. ## How was this patch tested? Two new test cases are added in `DatasetSuite` to check `.showString()` output. Author: Cheng Lian <l...@databricks.com> Closes #13331 from liancheng/sp

spark git commit: [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows

2016-05-26 Thread lian
ing methods like `Dataset.col`. So it probably makes more sense to show full contents of the query plan. ## How was this patch tested? Two new test cases are added in `DatasetSuite` to check `.showString()` output. Author: Cheng Lian <l...@databricks.com> Closes #13331 from liancheng/spark

spark git commit: [SPARK-15498][TESTS] fix slow tests

2016-05-24 Thread lian
picked from commit 50b660d725269dc0c11e0d350ddd7fc8b19539a0) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fb7b90f6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fb7b90f6 Diff: htt

spark git commit: [SPARK-15498][TESTS] fix slow tests

2016-05-24 Thread lian
a0 Parents: 4acabab Author: Wenchen Fan <wenc...@databricks.com> Authored: Tue May 24 21:23:39 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Tue May 24 21:23:39 2016 -0700 -- .../expressions/codege

spark git commit: [SPARK-15431][SQL] Support LIST FILE(s)|JAR(s) command natively

2016-05-23 Thread lian
ode path. Author: Xin Wu <xi...@us.ibm.com> Author: xin Wu <xi...@us.ibm.com> Closes #13212 from xwu0226/list_command. (cherry picked from commit 01659bc50cd3d53815d205d005c3678e714c08e0) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.or

spark git commit: [SPARK-15431][SQL] Support LIST FILE(s)|JAR(s) command natively

2016-05-23 Thread lian
pos/asf/spark/tree/01659bc5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/01659bc5 Branch: refs/heads/master Commit: 01659bc50cd3d53815d205d005c3678e714c08e0 Parents: a8e97d1 Author: Xin Wu <xi...@us.ibm.com> Authored: Mon

spark git commit: [SPARK-14031][SQL] speedup CSV writer

2016-05-23 Thread lian
3229 from davies/csv_writer. (cherry picked from commit 80091b8a6840b562cf76341926e5b828d4def7e2) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6eb8ec6f Tree: http://git-wi

spark git commit: [SPARK-14031][SQL] speedup CSV writer

2016-05-23 Thread lian
ds/master Commit: 80091b8a6840b562cf76341926e5b828d4def7e2 Parents: dafcb05 Author: Davies Liu <dav...@databricks.com> Authored: Mon May 23 10:48:25 2016 -0700 Committer: Cheng Lian <l...@databricks.com> Committed: Mon May 2

spark git commit: [SPARK-15334][SQL][HOTFIX] Fixes compilation error for Scala 2.10

2016-05-18 Thread lian
How was this patch tested? Jenkins build. Author: Cheng Lian <l...@databricks.com> Closes #13166 from liancheng/hotfix-for-scala-2.10. (cherry picked from commit c4a45fd855252f0f7130cdcc08486591fadab7df) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-15334][SQL][HOTFIX] Fixes compilation error for Scala 2.10

2016-05-18 Thread lian
tch tested? Jenkins build. Author: Cheng Lian <l...@databricks.com> Closes #13166 from liancheng/hotfix-for-scala-2.10. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c4a45fd8 Tree: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-15307][SQL] speed up listing files for data source

2016-05-18 Thread lian
ime are spent in merging schema now). Author: Davies Liu <dav...@databricks.com> Closes #13094 from davies/listing. (cherry picked from commit 33814f887aea339c99e14ce7f14ca6fcc6875015) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARK-15307][SQL] speed up listing files for data source

2016-05-18 Thread lian
/33814f88 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/33814f88 Branch: refs/heads/master Commit: 33814f887aea339c99e14ce7f14ca6fcc6875015 Parents: 6e02aec Author: Davies Liu <dav...@databricks.com> Authored: Wed May 18 18:46:57 2016 +0800 Committer: Cheng Lian <l...@databricks.c

[3/3] spark git commit: [SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable

2016-05-17 Thread lian
d from commit 25b315e6cad7c27b62dcaa2c194293c1115fdfb3) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5f5270ea Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5f5270e

[2/3] spark git commit: [SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable

2016-05-17 Thread lian
http://git-wip-us.apache.org/repos/asf/spark/blob/5f5270ea/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala -- diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala

[2/3] spark git commit: [SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable

2016-05-17 Thread lian
http://git-wip-us.apache.org/repos/asf/spark/blob/25b315e6/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala -- diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala

[3/3] spark git commit: [SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable

2016-05-17 Thread lian
Commit: 25b315e6cad7c27b62dcaa2c194293c1115fdfb3 Parents: b674e67 Author: Sean Zhong <seanzh...@databricks.com> Authored: Wed May 18 09:01:59 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Wed May 18 09:01:59 2016 +0800 -- do

[1/3] spark git commit: [SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable

2016-05-17 Thread lian
Repository: spark Updated Branches: refs/heads/master b674e67c2 -> 25b315e6c http://git-wip-us.apache.org/repos/asf/spark/blob/25b315e6/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala -- diff

spark git commit: [SPARK-15171][SQL] Deprecate registerTempTable and add dataset.createTempView

2016-05-12 Thread lian
iew, dataset.createOrReplaceTempView. ## How was this patch tested? Unit tests. Author: Sean Zhong <seanzh...@databricks.com> Closes #12945 from clockfly/spark-15171. (cherry picked from commit 33c6eb5218ce3c31cc9f632a67fd2c7057569683) Signed-off-by: Cheng Lian <l...@databricks.com> Project:

spark git commit: [SPARK-15171][SQL] Deprecate registerTempTable and add dataset.createTempView

2016-05-12 Thread lian
ommitter: Cheng Lian <l...@databricks.com> Committed: Thu May 12 15:51:53 2016 +0800 -- .../sbt_app_sql/src/main/scala/SqlApp.scala | 4 +- .../apache/spark/examples/sql/JavaSparkSQL.java | 8 +-- .../streaming/JavaSql

spark git commit: [SPARK-14933][SQL] Failed to create view out of a parquet or orc table

2016-05-11 Thread lian
7b72ca25d) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/749c29bc Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/749c29bc Diff: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-14933][SQL] Failed to create view out of a parquet or orc table

2016-05-11 Thread lian
in Wu <xi...@us.ibm.com> Authored: Wed May 11 22:17:59 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Wed May 11 22:17:59 2016 +0800 -- .../spark/sql/hive/HiveMetastoreCatalog.scala | 10

spark git commit: [SPARK-14459][SQL] Detect relation partitioning and adjust the logical plan

2016-05-09 Thread lian
ARK-14459-detect-hive-partitioning. (cherry picked from commit 652bbb1bf62722b08a062c7a2bf72019f85e179e) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8caaaed9 Tree:

spark git commit: [SPARK-14459][SQL] Detect relation partitioning and adjust the logical plan

2016-05-09 Thread lian
efs/heads/master Commit: 652bbb1bf62722b08a062c7a2bf72019f85e179e Parents: 16a503c Author: Ryan Blue <b...@apache.org> Authored: Mon May 9 17:01:23 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Mon May

spark git commit: [SPARK-15211][SQL] Select features column from LibSVMRelation causes failure

2016-05-09 Thread lian
cked from commit 635ef407e11dec41ae9bc428935fb8fdaa482f7e) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/238b7b41 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/238b

spark git commit: [SPARK-15211][SQL] Select features column from LibSVMRelation causes failure

2016-05-09 Thread lian
er Commit: 635ef407e11dec41ae9bc428935fb8fdaa482f7e Parents: a59ab59 Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Authored: Mon May 9 15:05:06 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Mon May 9 15:05:06 2016 +0800 -- .../apache/s

spark git commit: [SPARK-14962][SQL] Do not push down isnotnull/isnull on unsuportted types in ORC

2016-05-06 Thread lian
ySuite` and `OrcFilterSuite` and `sbt scalastyle`. Author: hyukjinkwon <gurwls...@gmail.com> Author: Hyukjin Kwon <gurwls...@gmail.com> Closes #12777 from HyukjinKwon/SPARK-14962. (cherry picked from commit fa928ff9a3c1de5d5aff9d14e6bc1bd03fcca087) Signed-off-by: Cheng Lian <l...@databricks

spark git commit: [SPARK-14962][SQL] Do not push down isnotnull/isnull on unsuportted types in ORC

2016-05-06 Thread lian
ay 7 01:46:45 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Sat May 7 01:46:45 2016 +0800 -- .../apache/spark/sql/test/SQLTestUtils.scala| 2 +- .../apache/spark/sql/hive/orc/OrcFilters.scala | 63

spark git commit: [SPARK-14139][SQL] RowEncoder should preserve schema nullability

2016-05-05 Thread lian
s/master Commit: 55cc1c991a9e39efb14177a948b09b7909e53e25 Parents: 77361a4 Author: Wenchen Fan <wenc...@databricks.com> Authored: Fri May 6 01:08:04 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Fri May 6 01:08:04 2016 +0800 --

spark git commit: [SPARK-14127][SQL] Native "DESC [EXTENDED | FORMATTED] " DDL command

2016-05-04 Thread lian
| |Database: |default | | |Owner: |lian | | |Create Time:|Mon Jan 04 17:06:00 CST 2016 | | |Last Access Time:

spark git commit: [SPARK-14127][SQL] Native "DESC [EXTENDED | FORMATTED] " DDL command

2016-05-04 Thread lian
| |Database: |default | | |Owner: |lian | | |Create Time:|Mon Jan 04 17:06:00 CST 2016 | | |Last Access Time:

spark git commit: [SPARK-14237][SQL] De-duplicate partition value appending logic in various buildReader() implementations

2016-05-04 Thread lian
do the work. Author: Cheng Lian <l...@databricks.com> Closes #12866 from liancheng/spark-14237-simplify-partition-values-appending. (cherry picked from commit bc3760d405cc8c3ffcd957b188afa8b7e3b1f824) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/

spark git commit: [SPARK-14237][SQL] De-duplicate partition value appending logic in various buildReader() implementations

2016-05-04 Thread lian
ork. Author: Cheng Lian <l...@databricks.com> Closes #12866 from liancheng/spark-14237-simplify-partition-values-appending. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc3760d4 Tree: http://git-wip-us.apache.org/

spark git commit: [SPARK-14875][SQL] Makes OutputWriterFactory.newInstance public

2016-04-25 Thread lian
ain, since 3rd party data sources like spark-avro depend on it. ## How was this patch tested? N/A Author: Cheng Lian <l...@databricks.com> Closes #12652 from liancheng/spark-14875. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spa

spark git commit: [SPARK-14843][ML] Fix encoding error in LibSVMRelation

2016-04-22 Thread lian
asf/spark/tree/8098f158 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8098f158 Branch: refs/heads/master Commit: 8098f158576b07343f74e2061d217b106c71b62d Parents: c089c6f Author: Liang-Chi Hsieh <sim...@tw.ibm.com> Authored: Sat Apr 23 01:11:36 2016 +0800 Committer: Cheng Lian <l

spark git commit: [SPARK-14488][SPARK-14493][SQL] "CREATE TEMPORARY TABLE ... USING ... AS SELECT" shouldn't create persisted table

2016-04-12 Thread lian
that we were `CreateMetastoreDataSourceAsSelect` uses default Hive warehouse location when `PATH` data source option is absent. ## How was this patch tested? Added a test case to create a temporary table using the target syntax and check whether it's indeed a temporary table. Author: Cheng Lian <l..

spark git commit: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 compilation failure

2016-04-01 Thread lian
tch tested? Compilation. Author: Cheng Lian <l...@databricks.com> Closes #12107 from liancheng/spark-14295-hotfix. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3715ecdf Tree: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-14114][SQL] implement buildReader for text data source

2016-03-30 Thread lian
org/repos/asf/spark/tree/816f359c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/816f359c Branch: refs/heads/master Commit: 816f359cf043ef719a0bc7df0506a3a830fff70d Parents: 7320f9b Author: Wenchen Fan <wenc...@databricks.com> Authored: Wed Mar 30 17:32:53 2016 +0800 Committer

spark git commit: [SPARK-14208][SQL] Renames spark.sql.parquet.fileScan

2016-03-29 Thread lian
ces are being migrated to `FileScanRDD` code path. ## How was this patch tested? None. Author: Cheng Lian <l...@databricks.com> Closes #12003 from liancheng/spark-14208-option-renaming. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spa

spark git commit: [SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL

2016-03-25 Thread lian
Fan <wenc...@databricks.com> Authored: Fri Mar 25 20:19:04 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Mar 25 20:19:04 2016 +0800 -- .../scala/org/apache/spark/repl/ReplSuite.scala | 15 .

spark git commit: [SPARK-13549][SQL] Refactor the Optimizer Rule CollapseProject

2016-03-23 Thread lian
/6ce008ba Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6ce008ba Branch: refs/heads/master Commit: 6ce008ba46aa1fc8a5c222ce0f25a6d81f53588e Parents: cde086c Author: gatorsmile <gatorsm...@gmail.com> Authored: Thu Mar 24 00:51:31 2016 +0800 Committer: Cheng Lian <l...@databricks.c

spark git commit: [SPARK-13817][SQL][MINOR] Renames Dataset.newDataFrame to Dataset.ofRows

2016-03-23 Thread lian
How was this patch tested? Existing tests. [1]: https://github.com/apache/spark/commit/6d37e1eb90054cdb6323b75fb202f78ece604b15#commitcomment-16654694 Author: Cheng Lian <l...@databricks.com> Closes #11889 from liancheng/spark-13817-follow-up. Project: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-13772][SQL] Fix data type mismatch for decimal

2016-03-22 Thread lian
.6 Commit: 3243d46ab8c3f76c1a7ee2cc5588f08ec7b51cfe Parents: f9221ad Author: cenyuhai <cenyu...@didichuxing.com> Authored: Tue Mar 22 20:53:11 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Tue Mar 22 20:53:18 2016 +0800 --

spark git commit: [SPARK-13774][SQL] - Improve error message for non-existent paths and add tests

2016-03-22 Thread lian
-us.apache.org/repos/asf/spark/tree/0ce01635 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0ce01635 Branch: refs/heads/master Commit: 0ce01635cc66ca5f9d8962235054335b16f7507e Parents: 4e09a0d Author: Sunitha Kambhampati <skam...@us.ibm.com> Authored: Tue Mar 22 20:47:57 2016 +0800 Committe

spark git commit: [SPARK-13473][SQL] Simplifies PushPredicateThroughProject

2016-03-22 Thread lian
ting test cases should do the work. Author: Cheng Lian <l...@databricks.com> Closes #11864 from liancheng/spark-13473-cleanup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f2e855fb Tree: http://git-wip-us.apache.org

spark git commit: [SPARK-14000][SQL] case class with a tuple field can't work in Dataset

2016-03-21 Thread lian
tabricks.com> Authored: Mon Mar 21 22:22:15 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Mon Mar 21 22:22:15 2016 +0800 -- .../apache/spark/sql/catalyst/analysis/Analyzer.scala | 5 - .../

spark git commit: [SPARK-14004][SQL][MINOR] AttributeReference and Alias should only use the first qualifier to generate SQL strings

2016-03-19 Thread lian
ing the first qualifiers. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Existing tests should be enough. Author: Cheng Lian <l...@databricks.com> Closes #11820 from liancheng/spark-14004-single-qualifier. Proj

spark git commit: [SPARK-14001][SQL] support multi-children Union in SQLBuilder

2016-03-18 Thread lian
+0800 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Mar 18 19:42:33 2016 +0800 -- .../scala/org/apache/spark/sql/hive/SQLBuilder.scala | 14 ++ .../apache/spark/sql/hive/LogicalPlanToSQLSuite.scala

spark git commit: [SPARK-13972][SQ] hive tests should fail if SQL generation failed

2016-03-18 Thread lian
er Commit: 0acb32a3f19484d3ea3b60fccef596025c8a8f83 Parents: 53f32a2 Author: Wenchen Fan <wenc...@databricks.com> Authored: Fri Mar 18 23:16:14 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Fri Mar 1

spark git commit: [MINOR][DOCS] Replace `DataFrame` with `Dataset` in Javadoc.

2016-03-12 Thread lian
3 12:11:18 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Sun Mar 13 12:11:18 2016 +0800 -- .../org/apache/spark/ml/attribute/package-info.java | 2 +- .../scala/org/apache/spark/ml/feature/p

spark git commit: [SPARK-13817][BUILD][SQL] Re-enable MiMA and removes object DataFrame

2016-03-11 Thread lian
we can simply put a `MissingClassProblem` filter in `MimaExcludes.scala` for most DataFrame API changes. ## How was this patch tested? Tested by MiMA check triggered by Jenkins. Author: Cheng Lian <l...@databricks.com> Closes #11656 from liancheng/re-enable-mima. Project: http:

spark git commit: Revert "[SPARK-13616][SQL] Let SQLBuilder convert logical plan without a project on top of it"

2016-03-05 Thread lian
ert PR #11466 for safe. Author: Cheng Lian <l...@databricks.com> Closes #11539 from liancheng/revert-pr-11466. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8ff88094 Tree: http://git-wip-us.apache.org/repos/asf/spark

spark git commit: [SPARK-12720][SQL] SQL Generation Support for Cube, Rollup, and Grouping Sets

2016-03-05 Thread lian
git-wip-us.apache.org/repos/asf/spark/tree/adce5ee7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/adce5ee7 Branch: refs/heads/master Commit: adce5ee721c6a844ff21dfcd8515859458fe611d Parents: f19228e Author: gatorsmile <gatorsm...@gmail.com> Authored: Sat Mar 5 19:25:03 2016 +08

spark git commit: [SPARK-13540][SQL] Supports using nested classes within Scala objects as Dataset element type

2016-02-29 Thread lian
ase is added to `DatasetSuite`. It checks contents of a Dataset whose element type is a nested class declared in a Scala object. Author: Cheng Lian <l...@databricks.com> Closes #11421 from liancheng/spark-13540-object-as-outer-scope. Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARK-13457][SQL] Removes DataFrame RDD operations

2016-02-26 Thread lian
ons with `withNewExecutionId` to track Spark jobs. But they are removed in #11323. ## How was the this patch tested? No extra tests are added. Existing tests should do the work. Author: Cheng Lian <l...@databricks.com> Closes #11388 from liancheng/remove-df-rdd-ops. Project: http://git-wip-us.apache.

spark git commit: [SPARK-13457][SQL] Removes DataFrame RDD operations

2016-02-25 Thread lian
ods of `DataFrame.rdd`. ## How was the this patch tested? No extra tests are added. Existing tests should do the work. Author: Cheng Lian <l...@databricks.com> Closes #11323 from liancheng/remove-df-rdd-ops. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

[3/3] spark git commit: [SPARK-13486][SQL] Move SQLConf into an internal package

2016-02-25 Thread lian
gt; Authored: Thu Feb 25 17:49:50 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Feb 25 17:49:50 2016 +0800 -- project/MimaExcludes.scala | 6 + .../scala/org/apache/spark/s

[1/3] spark git commit: [SPARK-13486][SQL] Move SQLConf into an internal package

2016-02-25 Thread lian
Repository: spark Updated Branches: refs/heads/master 07f92ef1f -> 2b2c8c332 http://git-wip-us.apache.org/repos/asf/spark/blob/2b2c8c33/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala -- diff --git

spark git commit: [SPARK-13263][SQL] SQL Generation Support for Tablesample

2016-02-23 Thread lian
k-Pro.local> This patch had conflicts when merged, resolved by Committer: Cheng Lian <l...@databricks.com> Closes #11148 from gatorsmile/tablesplitsample. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/87250580 Tre

[1/2] spark git commit: [SPARK-12799] Simplify various string output for expressions

2016-02-21 Thread lian
Repository: spark Updated Branches: refs/heads/master d806ed343 -> d9efe63ec http://git-wip-us.apache.org/repos/asf/spark/blob/d9efe63e/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/CatalystSchemaConverter.scala

spark git commit: [SPARK-13205][SQL] SQL Generation Support for Self Join

2016-02-10 Thread lian
8:21 2016 +0800 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Feb 11 11:08:21 2016 +0800 -- .../spark/sql/execution/datasources/LogicalRelation.scala | 6 +- .../main/scala/org/apache/spark/sql/hive/

spark git commit: [SPARK-12231][SQL] create a combineFilters' projection when we call buildPartitionedTableScan

2016-01-31 Thread lian
the otherwise failure cases. Author: Kevin Yu <q...@us.ibm.com> Closes #10388 from kevinyu98/spark-12231. (cherry picked from commit fd50df413fbb3b7528cdff311cc040a6212340b9) Signed-off-by: Cheng Lian <l...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Com

spark git commit: [SPARK-12818] Polishes spark-sketch module

2016-01-29 Thread lian
Repository: spark Updated Branches: refs/heads/master 5f686cc8b -> 2b027e9a3 [SPARK-12818] Polishes spark-sketch module Fixes various minor code and Javadoc styling issues. Author: Cheng Lian <l...@databricks.com> Closes #10985 from liancheng/sketch-polishing. Project: http:

spark git commit: [SPARK-13050][BUILD] Scalatest tags fail build with the addition of the sketch module

2016-01-28 Thread lian
4:50 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Jan 28 23:34:50 2016 -0800 -- common/sketch/pom.xml | 7 +++ 1 file

spark git commit: [SPARK-12401][SQL] Add integration tests for postgres enum types

2016-01-28 Thread lian
//git-wip-us.apache.org/repos/asf/spark/diff/abae889f Branch: refs/heads/master Commit: abae889f08eb412cb897e4e63614ec2c93885ffd Parents: df78a93 Author: Takeshi YAMAMURO <linguin@gmail.com> Authored: Thu Jan 28 15:20:16 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed

spark git commit: [SPARK-11955][SQL] Mark optional fields in merging schema for safely pushdowning filters in Parquet

2016-01-28 Thread lian
s/master Commit: 4637fc08a3733ec313218fb7e4d05064d9a6262d Parents: 3a40c0e Author: Liang-Chi Hsieh <vii...@appier.com> Authored: Thu Jan 28 16:25:21 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Thu Jan 28 16:25:21 2016 -0800 ---

spark git commit: [SQL] Minor Scaladoc format fix

2016-01-26 Thread lian
Repository: spark Updated Branches: refs/heads/master ee74498de -> 83507fea9 [SQL] Minor Scaladoc format fix Otherwise the `^` character is always marked as error in IntelliJ since it represents an unclosed superscript markup tag. Author: Cheng Lian <l...@databricks.com> Closes #1

spark git commit: [SPARK-12934] use try-with-resources for streams

2016-01-25 Thread lian
or: tedyu <yuzhih...@gmail.com> Authored: Mon Jan 25 18:23:47 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Mon Jan 25 18:23:47 2016 -0800 -- .../src/main/java/org/apache/spark/util/sketch/CountMin

spark git commit: [SPARK-12560][SQL] SqlTestUtils.stripSparkFilter needs to copy utf8strings

2016-01-19 Thread lian
: b72e01e Author: Imran Rashid <iras...@cloudera.com> Authored: Tue Jan 19 12:24:21 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Tue Jan 19 12:24:21 2016 -0800 -- .../src/test/scala/org/

spark git commit: [SPARK-12867][SQL] Nullability of Intersect can be stricter

2016-01-19 Thread lian
er Commit: b72e01e82148a908eb19bb3f526f9777bfe27dde Parents: 2388de5 Author: gatorsmile <gatorsm...@gmail.com> Authored: Tue Jan 19 11:35:58 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Tue Jan 19 11:35:58 2016 -0800 -

spark git commit: [SPARK-12724] SQL generation support for persisted data source tables

2016-01-12 Thread lian
ier]` is added to `LogicalRelation`. When a `LogicalRelation` representing a persisted data source relation is created, this field holds the database name and table name of the relation. Author: Cheng Lian <l...@databricks.com> Closes #10712 from liancheng/spark-12724-datasources-sql-gen.

spark git commit: [SPARK-12742][SQL] org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to Table already exists exception

2016-01-11 Thread lian
er Commit: 473907adf6e37855ee31d0703b43d7170e26b4b9 Parents: fe9eb0b Author: wangfei <wangfei_he...@126.com> Authored: Mon Jan 11 18:18:44 2016 -0800 Committer: Cheng Lian <l...@databricks.com> Committed: Mon Jan 11 18:18:44 2016 -0800

<    1   2   3   4   5   >