[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-21 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199160796 Thanks! Merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11756 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199158029 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199158025 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199157504 **[Test build #53659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53659/consoleFull)** for PR 11756 at commit [`dec3d81`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199143263 LGTM, pending tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199139827 **[Test build #53659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53659/consoleFull)** for PR 11756 at commit [`dec3d81`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199137111 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199137110 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199137105 Last 2 comments, otherwise LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199136840 **[Test build #53652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53652/consoleFull)** for PR 11756 at commit [`3ff900e`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56784237 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +964,53 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56784202 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +964,53 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199116308 **[Test build #53652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53652/consoleFull)** for PR 11756 at commit [`3ff900e`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199116238 @cloud-fan Is this a typo maybe :)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199116245 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-199090207 retest it please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198215631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197667954 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197658346 Filed in https://issues.apache.org/jira/browse/SPARK-13953. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197636051 **[Test build #53384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53384/consoleFull)** for PR 11756 at commit [`551593a`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198215626 **[Test build #53506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53506/consoleFull)** for PR 11756 at commit [`59e7214`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197629010 LGTM, cc @davies for another look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198214354 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197276405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197628087 This makes sense to me. Actually for CSV, when `any row does not have a same schema`, it just means `corrupted format`, as CSV has a very simple format and can always

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56323774 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala --- @@ -40,6 +40,7 @@ private[sql] object InferSchema

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197636844 @cloud-fan Sorry, one more question. Would it be great if we maybe make `spark.sql.columnNameOfCorruptRecord` as an option just like the compression option for othe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198236937 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198217639 **[Test build #53507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53507/consoleFull)** for PR 11756 at commit [`3ff900e`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197652379 **[Test build #53379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53379/consoleFull)** for PR 11756 at commit [`de8d291`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197652643 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197667952 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197665701 **[Test build #53382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53382/consoleFull)** for PR 11756 at commit [`29a8f68`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197276172 **[Test build #53312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53312/consoleFull)** for PR 11756 at commit [`4440a55`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197644079 This is a good point, if we only use this config while dealing with json data. cc @rxin what do you think? --- If your project is set up for it, you can reply to th

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197622125 **[Test build #53379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53379/consoleFull)** for PR 11756 at commit [`de8d291`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198212315 **[Test build #53504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53504/consoleFull)** for PR 11756 at commit [`bfc0405`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197299127 Overall LGTM, thanks for working on it! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198214357 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197634470 **[Test build #53382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53382/consoleFull)** for PR 11756 at commit [`29a8f68`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198236936 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197665906 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197290603 ah, thanks for the detail explanation and examples! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198215423 **[Test build #53506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53506/consoleFull)** for PR 11756 at commit [`59e7214`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198214348 **[Test build #53504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53504/consoleFull)** for PR 11756 at commit [`bfc0405`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197624257 @cloud-fan Actually, I have a question. So, in JSON data source, I thought JSON data format itself can have a flexible schema so it does not necessarily have the

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198215630 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-198236738 **[Test build #53507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53507/consoleFull)** for PR 11756 at commit [`3ff900e`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197645874 Yea I think it should be an option on each read, not a global option. Most global options don't make a lot of sense as global options. --- If your project is set up for

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197645943 But since we had it, i'd say we should keep it to avoid breaking compatibility. We can have the per-read option override the global option. --- If your project is set up

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56325150 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,28 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56323218 --- Diff: python/pyspark/sql/readwriter.py --- @@ -162,6 +162,14 @@ def json(self, path, schema=None): (e.g. 00012) *

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197665908 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197652640 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197667739 **[Test build #53384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53384/consoleFull)** for PR 11756 at commit [`551593a`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197276404 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197246235 **[Test build #53313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53313/consoleFull)** for PR 11756 at commit [`32ae8b2`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197246251 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197246246 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197242349 **[Test build #53313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53313/consoleFull)** for PR 11756 at commit [`32ae8b2`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197239116 **[Test build #53312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53312/consoleFull)** for PR 11756 at commit [`4440a55`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197238803 The commit I submitted includes comment changes and avoiding to add a `_corrupt_record` field when it is `DROPMALFORMED` mode in type inference. --- If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197235349 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197235350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197234856 **[Test build #53303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53303/consoleFull)** for PR 11756 at commit [`3675fae`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56306144 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -288,6 +288,9 @@ class DataFrameReader private[sql](sqlContext: SQLC

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197224540 For example, the data below: ``` 1,2,3,4 3,2,1 ``` will produce the records below: - `PERMISSIVE` ``` Row(1,2,3,4)

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197222768 I'm not familiar with CSV part, what if users set the schema directly before read data and the mode is `PERMISSIVE`? Will we add the extra field? --- If your project

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56300778 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56299745 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56299307 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56299195 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56299031 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56298949 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56298666 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -288,6 +288,9 @@ class DataFrameReader private[sql](sqlContext: SQLCon

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56296202 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -963,6 +963,31 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197208262 **[Test build #53303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53303/consoleFull)** for PR 11756 at commit [`3675fae`](https://gi

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197207970 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197207972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197207753 **[Test build #53286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53286/consoleFull)** for PR 11756 at commit [`4c46f4b`](https://g

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56292393 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -991,6 +999,16 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56291887 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -991,6 +999,16 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56291651 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -969,6 +969,14 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56291636 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -991,6 +999,16 @@ class JsonSuite extends Query

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56291446 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -991,6 +999,16 @@ class JsonSuite extends QueryTe

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56290903 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala --- @@ -49,6 +50,16 @@ private[sql] class JSONOption

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11756#discussion_r56290625 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ParseModes.scala --- @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-13764][SQL] Parse modes in JSON data so...

2016-03-16 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11756#issuecomment-197186617 cc @cloud-fan for review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this