[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425497#comment-15425497
]
Yin Huai commented on SPARK-16320:
--
[~maver1ck] Seems we can close this jira?
> Spark 2.0 slower than
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421731#comment-15421731
]
Sean Owen commented on SPARK-16320:
---
I think that was the problem being solved there though, right?
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421728#comment-15421728
]
Maciej Bryński commented on SPARK-16320:
Maybe we can change SPARK-12384 a little bit and set
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421719#comment-15421719
]
Sean Owen commented on SPARK-16320:
---
I see, I wonder if this deserves a bit of documentation in the
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421699#comment-15421699
]
Maciej Bryński commented on SPARK-16320:
[~srowen], [~michael],
I found the reason why G1GC with
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406430#comment-15406430
]
Michael Allman commented on SPARK-16320:
Thank you for the new benchmarks, [~maver1ck].
The
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406372#comment-15406372
]
Maciej Bryński commented on SPARK-16320:
Yes.
I also added Spark 1.6 with G1GC.
Have in mind that
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406125#comment-15406125
]
Michael Allman commented on SPARK-16320:
Hi [~maver1ck]. These are excellent findings! I'm
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405765#comment-15405765
]
Maciej Bryński commented on SPARK-16320:
Yes. But, it will be default on Java 9.
I think you can
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405682#comment-15405682
]
Sean Owen commented on SPARK-16320:
---
OK is this one "not a problem" then? G1GC is non default right?
>
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405603#comment-15405603
]
Maciej Bryński commented on SPARK-16320:
PS.
When using
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405576#comment-15405576
]
Maciej Bryński commented on SPARK-16320:
You're right.
I think I found solution for SPARK-16320
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405057#comment-15405057
]
Sean Zhong commented on SPARK-16320:
[~maver1ck] Did you use the test case in this jira
{code}
select
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404675#comment-15404675
]
Apache Spark commented on SPARK-16320:
--
User 'maver1ck' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404646#comment-15404646
]
Maciej Bryński commented on SPARK-16320:
[~michael], [~yhuai]
I think this is smallest change
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404620#comment-15404620
]
Maciej Bryński commented on SPARK-16320:
I think that problem is already resolved by
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404619#comment-15404619
]
Maciej Bryński commented on SPARK-16320:
Yes.
That's it.
With this PR Spark 2.0 is faster than
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404369#comment-15404369
]
Yin Huai commented on SPARK-16320:
--
Can you also try https://github.com/apache/spark/pull/13701 and see
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404361#comment-15404361
]
Michael Allman commented on SPARK-16320:
[~maver1ck] I'm having trouble reproducing your problem.
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404114#comment-15404114
]
Maciej Bryński commented on SPARK-16320:
[~clockfly]
I tested your patch. Results are equal to
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403470#comment-15403470
]
Maciej Bryński commented on SPARK-16320:
Yes. I'll do check tomorrow.
> Spark 2.0 slower than
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403238#comment-15403238
]
Sean Zhong commented on SPARK-16320:
[~loziniak] Can you check whether the PR works for you?
>
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402961#comment-15402961
]
Apache Spark commented on SPARK-16320:
--
User 'clockfly' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386586#comment-15386586
]
Michael Allman commented on SPARK-16320:
Okay, so the metastore is not a factor.
> Spark 2.0
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386552#comment-15386552
]
Maciej Bryński commented on SPARK-16320:
[~michael]
Could you check SPARK-16321 ?
I attached
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386395#comment-15386395
]
Michael Allman commented on SPARK-16320:
The code path for reading data from parquet files has
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386333#comment-15386333
]
Michael Allman commented on SPARK-16320:
[~maver1ck] Would it be possible for you to share your
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384465#comment-15384465
]
Maciej Bryński commented on SPARK-16320:
OK. I think we have general problem with Spark 2.0
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361781#comment-15361781
]
Maciej Bryński commented on SPARK-16320:
[~rxin]
I created benchmark script and added results.
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357213#comment-15357213
]
Maciej Bryński commented on SPARK-16320:
OK.
I'll try to confirm this issue on generated data.
[
https://issues.apache.org/jira/browse/SPARK-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1535#comment-1535
]
Reynold Xin commented on SPARK-16320:
-
Can you try just generating a simple file with a nested column
31 matches
Mail list logo
