Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113670396
Merged. I manually fixed a small conflict for 1.4 branch. Thanks
@NathanHowell !
---
If your project is set up for it, you can reply to this email and have your
reply app
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/6799
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113669845
I am merging it to master and branch 1.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project d
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113669186
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113669113
[Test build #35324 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35324/console)
for PR 6799 at commit
[`76ac3e8`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113656422
[Test build #940 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/940/console)
for PR 6799 at commit
[`76ac3e8`](https://github.c
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113648717
[Test build #35324 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35324/consoleFull)
for PR 6799 at commit
[`76ac3e8`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113648331
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113648307
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113647698
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113647364
**[Test build #938 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/938/console)**
for PR 6799 at commit
[`76ac3e8`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113645291
[Test build #940 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/940/consoleFull)
for PR 6799 at commit
[`76ac3e8`](https://githu
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113606861
LGTM pending Jenkins tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have t
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113606824
Found it is hard to drop those columns in parquet's write path... Let's
check this one in to make JSON has the same behavior with 1.3. I will merge it
to both master and b
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113597100
[Test build #938 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/938/consoleFull)
for PR 6799 at commit
[`76ac3e8`](https://githu
Github user NathanHowell commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113551319
I'm fine with that too.
On Thu, Jun 18, 2015 at 6:23 PM, Yin Huai wrote:
> I feel we can just fix the parquet part and do not need to touch code
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113335243
I feel we can just fix the parquet part and do not need to touch code
related to json.
---
If your project is set up for it, you can reply to this email and have your
rep
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-11118
@NathanHowell Actually, do you think we should just fix it in the parquet
side instead of introducing the flag? Since it is parquet's issue, maybe it is
not worth adding a
Github user NathanHowell commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113321989
Sounds good to me.
On Jun 18, 2015 4:58 PM, "Yin Huai" wrote:
> After second thought, I feel it is better to just drop those empty structs
> and the
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113321836
After second thought, I feel it is better to just drop those empty structs
and their corresponding values when we write data to parquet and log a warning
message. @NathanH
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-113309616
@NathanHowell Yeah, sounds good. In the error message, we can ask user to
drop that column. @liancheng Where will be the good place to add this check?
---
If your project
Github user NathanHowell commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111841630
@yhuai I agree, I think a better default approach might be to fail in the
Parquet writer (instead of writing a fail it cannot read)... and add a flag to
enable this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111767643
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111767641
[Test build #34855 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34855/console)
for PR 6799 at commit
[`76ac3e8`](https://github.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111760720
@NathanHowell Thank you for working on it! I am wondering if we can keep
the new behavior and introduce a flag to let users switch back to the old
behavior? Here is my tho
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111760616
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111760633
[Test build #34855 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34855/consoleFull)
for PR 6799 at commit
[`76ac3e8`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111760619
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111760542
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabl
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6799#issuecomment-111736867
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
GitHub user NathanHowell opened a pull request:
https://github.com/apache/spark/pull/6799
[SPARK-8093] [SQL] Remove empty structs inferred from JSON documents
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NathanHowell/spark spa
31 matches
Mail list logo
