Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-103106678
@marmbrus This should be ready to go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30535915
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -188,18 +188,20 @@ private[sql] class DDLParser(
private[sql] object
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/6194
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-103177741
Thanks guys! Merged to master and 1.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102889095
In the last a few commits, I added `orc` as a built-in data source name,
so that we can have
```scala
hiveContext.read.format(orc).load(hdfs://...)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102755727
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102755710
[Test build #32926 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32926/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102755729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102755272
LGTM with respect to API change (there isn't any).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461540
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102647157
[Test build #32907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32907/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102647163
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102647161
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102632583
[Test build #32907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32907/consoleFull)
for PR 6194 at commit
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461430
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461401
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -143,6 +145,9 @@ private[sql] class SQLConf extends Serializable with
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461455
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102632505
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102632498
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461402
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/HadoopTypeConverter.scala
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30461406
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102743175
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102743234
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102744323
[Test build #32926 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32926/consoleFull)
for PR 6194 at commit
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30459211
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30466733
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30465348
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -143,6 +145,9 @@ private[sql] class SQLConf extends Serializable with
CatalystConf {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102533199
[Test build #32848 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32848/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102533209
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102533213
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102564782
@zhzhan Thanks for the information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458451
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458591
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102563697
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30457281
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102563704
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102567633
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102567644
[Test build #32881 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32881/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102567627
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458421
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/HadoopTypeConverter.scala
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102569336
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102569337
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102569335
[Test build #32874 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32874/consoleFull)
for PR 6194 at commit
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458478
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458636
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/HadoopTypeConverter.scala
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102572221
[Test build #32881 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32881/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102572223
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-10257
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102563766
[Test build #32874 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32874/consoleFull)
for PR 6194 at commit
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30457511
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458409
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -143,6 +145,9 @@ private[sql] class SQLConf extends Serializable with
CatalystConf {
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30458552
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/package.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102460838
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102460819
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102464098
Some TODO items related to testing:
- [ ] Cleanup current ORC test suites, as most of them are based on old
Parquet test code, which has been deprecated and
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30432257
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102472560
@liancheng Thanks for the followup. For the future work, feel free to
assign to me.
---
If your project is set up for it, you can reply to this email and have your
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102508997
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102509270
[Test build #32848 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32848/consoleFull)
for PR 6194 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102509191
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102509161
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102495862
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102495861
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102495847
[Test build #32839 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32839/consoleFull)
for PR 6194 at commit
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6194#discussion_r30437802
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/6194
[SPARK-2883] [SQL] ORC data source for Spark SQL
This PR is an update of #6135 authored by @zhzhan from Hortonworks.
This PR implements a Spark SQL data source for accessing
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102460973
[Test build #32839 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32839/consoleFull)
for PR 6194 at commit
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102463555
@zhzhan Here is a rough list of my updates:
1. Rebased to PR #6150, which updated the newly introduced partitioning
support API
Made
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/6194#issuecomment-102526125
@liancheng FYI: For schema merging, I checked some orc experts, and
probably it is not supported the filter push down if the column is not in that
specific orc file (I
69 matches
Mail list logo
