[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089656#comment-16089656
]
Peng Meng commented on SPARK-21401:
---
Got it, thanks [~srowen]
> add poll function for
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089651#comment-16089651
]
Sean Owen commented on SPARK-21401:
---
That's why using the Scala implementation may be better. It should
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089648#comment-16089648
]
Peng Meng commented on SPARK-21401:
---
Yes, you don't need to do it for the vast majority of elements.
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089641#comment-16089641
]
Sean Owen commented on SPARK-21401:
---
I think it's not worth writing and maintaining a custom
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089638#comment-16089638
]
Peng Meng commented on SPARK-21401:
---
I mean we totally rewrite the BoundedPriorityQueue, not use Java
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089633#comment-16089633
]
Sean Owen commented on SPARK-21401:
---
I'm not sure how you would remove and add an element to the
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089629#comment-16089629
]
Peng Meng commented on SPARK-21401:
---
Thanks @srowen.
I mean for BoundedPriorityQueue, you also can
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089613#comment-16089613
]
Sean Owen commented on SPARK-21401:
---
I also am not clear why the implementation didn't use Scala's
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089606#comment-16089606
]
Peng Meng commented on SPARK-21401:
---
I think the BoundedPriorityQueue should be rewritten. there are
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089589#comment-16089589
]
Sean Owen commented on SPARK-21401:
---
Yes, I agree with you now. I find they're roughly the same, but,
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089586#comment-16089586
]
Peng Meng commented on SPARK-21401:
---
I have tested much about poll and toArray.sorted.
If the queue is
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089581#comment-16089581
]
Sean Owen commented on SPARK-21401:
---
Scratch that, I get the issue now. We can't remove the queue
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089578#comment-16089578
]
Peng Meng commented on SPARK-21401:
---
Hi [~srowen], I got why my original test pq.toArray.sorted is very
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089448#comment-16089448
]
Sean Owen commented on SPARK-21401:
---
What about the time needed to build the queue? insertions are
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089437#comment-16089437
]
Peng Meng commented on SPARK-21401:
---
I benchmarking just change pq.toArray.sorted. and pq.poll.
pq.poll
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089431#comment-16089431
]
Sean Owen commented on SPARK-21401:
---
You're benchmarking a lot of other changes all at once. I don't
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089424#comment-16089424
]
Peng Meng commented on SPARK-21401:
---
Hi [~srowen], for ALS optimization, the difference of using poll
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089415#comment-16089415
]
Sean Owen commented on SPARK-21401:
---
I can't see how polling would be faster. When you poll you have to
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089209#comment-16089209
]
Peng Meng commented on SPARK-21401:
---
Hi [~srowen], here we also want to get a fully sorted list by get
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088516#comment-16088516
]
Sean Owen commented on SPARK-21401:
---
[~peng.m...@intel.com] on re-reading this, where is something
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085874#comment-16085874
]
Peng Meng commented on SPARK-21401:
---
Sure, I will add isEmpty and maybe some other functions, and tests
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085869#comment-16085869
]
Sean Owen commented on SPARK-21401:
---
I don't think this is worth breaking out unless you are going to
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085738#comment-16085738
]
Peng Meng commented on SPARK-21401:
---
Yes, SPARK-21389 used pq.poll. pq.poll is just a small part of
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085727#comment-16085727
]
Sean Owen commented on SPARK-21401:
---
This sounds like a detail of
[
https://issues.apache.org/jira/browse/SPARK-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085651#comment-16085651
]
Apache Spark commented on SPARK-21401:
--
User 'mpjlu' has created a pull request for this issue:
25 matches
Mail list logo
