[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Yep, should be doable without too much effort. On Sun, Jun 4, 2017 at 9:54 PM, Xiao Li wrote: > @NathanHowell It sounds like we also > can

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16386 @NathanHowell It sounds like we also can provide multi-line support for JSON too. For example, in a single JSON file ``` {"a": 1, "b": 1.1} {"a": 2, "b": 1.1} {"a": 3, "b": 1.1

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16386 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wi

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73032/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73032/testReport)** for PR 16386 at commit [`b801ab0`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73030/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73030/testReport)** for PR 16386 at commit [`58118f2`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73029/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73029/testReport)** for PR 16386 at commit [`e323317`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73032/testReport)** for PR 16386 at commit [`b801ab0`](https://github.com/apache/spark/commit/b8

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73030/testReport)** for PR 16386 at commit [`58118f2`](https://github.com/apache/spark/commit/58

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #73029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73029/testReport)** for PR 16386 at commit [`e323317`](https://github.com/apache/spark/commit/e3

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 @cloud-fan When implementing tests for the other modes I've uncovered an existing bug in schema inference in `DROPMALFORMED` mode: https://issues.apache.org/jira/browse/SPARK-19641. Since it is

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16386 LGTM if the test can pass. It will be good if you can also address https://github.com/apache/spark/pull/16386#discussion_r100679183 --- If your project is set up for it, you can reply to this ema

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72975/testReport)** for PR 16386 at commit [`7296f7e`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72975/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72975/testReport)** for PR 16386 at commit [`7296f7e`](https://github.com/apache/spark/commit/72

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72729/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72729/testReport)** for PR 16386 at commit [`b0a5cc8`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72726/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72726/testReport)** for PR 16386 at commit [`0f4686d`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 @cloud-fan I just pushed a few more changes to address some of your comments. I'll be back later next week to continue work. --- If your project is set up for it, you can reply to this email a

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72729/testReport)** for PR 16386 at commit [`b0a5cc8`](https://github.com/apache/spark/commit/b0

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72726/testReport)** for PR 16386 at commit [`0f4686d`](https://github.com/apache/spark/commit/0f

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72694/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72694/testReport)** for PR 16386 at commit [`aafe7bd`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72694/testReport)** for PR 16386 at commit [`aafe7bd`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72685/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72685/testReport)** for PR 16386 at commit [`aafe7bd`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72620/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72620/testReport)** for PR 16386 at commit [`f71a465`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72620/testReport)** for PR 16386 at commit [`f71a465`](https://github.com/apache/spark/commit/f7

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72603/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72603/testReport)** for PR 16386 at commit [`30fb509`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72603/testReport)** for PR 16386 at commit [`30fb509`](https://github.com/apache/spark/commit/30

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Rebased again to pickup the build break hotfix in c618ccdbe9ac103dfa3182346e2a14a1e7fca91a --- If your project is set up for it, you can reply to this email and have your reply appear on GitHu

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72601/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72601/testReport)** for PR 16386 at commit [`6f8b0c3`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 I rebased to master and hopefully addressed all of your comments @cloud-fan, please have another look. --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #72601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72601/testReport)** for PR 16386 at commit [`6f8b0c3`](https://github.com/apache/spark/commit/6f

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16386 Hi @NathanHowell , do you have time to work on it? thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16386 Sorry, I missed the ping. Will review it tonight. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16386 can we focus on supporting multiline json in this PR? We can leave the improvements in new PRs, or this PR is kind of hard to review. --- If your project is set up for it, you can reply to this e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/16386 cc @gatorsmile can you please take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Any other comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71147/ Test PASSed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #71147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71147/testReport)** for PR 16386 at commit [`9dc084d`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #71147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71147/testReport)** for PR 16386 at commit [`9dc084d`](https://github.com/apache/spark/commit/9d

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Can someone kick off the tests again? The last failure was in another module (Kafka). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70730/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70730/testReport)** for PR 16386 at commit [`9dc084d`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 @HyukjinKwon I just pushed a change that makes the corrupt record handling consistent: if a corrupt record column is defined it will always get the json text for failed records. If `wholeFile`

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70730/testReport)** for PR 16386 at commit [`9dc084d`](https://github.com/apache/spark/commit/9d

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16386 Only regarding the comment, https://github.com/apache/spark/pull/16386#issuecomment-269386229, I have a similar (rather combined) idea that we provide another option such as corrupt file name o

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 The tests failed for an unrelated reason, looks to be running out of heap space in SBT somewhere. --- If your project is set up for it, you can reply to this email and have your reply appear o

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 @HyukjinKwon I agree that overloading the corrupt record column is undesirable and `F.input_file_name` is a better way to fetch the filename. It would be nice to extend this concept further and

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70644/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70644/testReport)** for PR 16386 at commit [`aa5a6db`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70644/testReport)** for PR 16386 at commit [`aa5a6db`](https://github.com/apache/spark/commit/aa

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 @srowen It is functionally the same as what you're suggesting. The question is how (or if) it should it be first class in the `DataFrameReader` api. If we agree that it should be exposed, eithe

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user srowen commented on the issue: https://github.com/apache/spark/pull/16386 Why do we need this at all? just use `wholeTextFiles` and parse them as JSON. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16386 > the corrupt column will contain the filename instead of the literal JSON if there is a parsing failure I am worried of changing the behaviour. I understand why it had to be here as yo

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70531/ Test FAILed. ---

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16386 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70531/testReport)** for PR 16386 at commit [`7ad5d5b`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user NathanHowell commented on the issue: https://github.com/apache/spark/pull/16386 Hello recent JacksonGenerator.scala commiters, please take a look. cc/ @rxin @hvanhovell @clockfly @hyukjinkwon @cloud-fan --- If your project is set up for it, you can reply to this e

[GitHub] spark issue #16386: [SPARK-18352][SQL] Support parsing multiline json files

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16386 **[Test build #70531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70531/testReport)** for PR 16386 at commit [`7ad5d5b`](https://github.com/apache/spark/commit/7a