[GitHub] spark pull request: [SPARK-4999][Streaming] Change storeInBlockMan...
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/3906#issuecomment-68840845 Good catchï¼ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-47196343 @pwendell I will update the codes soon, thanks for reminder. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46537162 you could open a JIRA issue to describe you request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user xiajunluan closed the pull request at: https://github.com/apache/spark/pull/931 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
GitHub user xiajunluan reopened a pull request: https://github.com/apache/spark/pull/931 Fix JIRA-983 and support exteranl sort for sortByKey Change class ExternalAppendOnlyMap and make it support customized comparator function(not only sorted by hashCode). You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiajunluan/spark-1 JIRA-983 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/931.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #931 commit 3092241234e6e85becccbd3d93a2a69b776d71e9 Author: Andrew xia junluan@intel.com Date: 2014-05-31T13:19:35Z Fix JIRA-983 and support exteranl sort for sortByKey --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-45059292 hi @matei 1.I will measure the performance influence after I add the pluggable comparator 2.I agree with you. if we just implement sortByKey, we should not use combiner(it is for combineByKey related API), it will need firstly aggregate values and after sorting, unfold values for same key. In this patch, I would like to reuse current class and fix this bug quickly. for long-term, I think we should write another similar AppendOnlyMap and ExternalAppendOnlyMap class for sortByKey, and ignore functions such as createCombiner, mergeValue, etc. I will try to design these class later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-45078849 Hi @mateiz 1. I will measure the performance influence after I add the pluggable comparator. 2. I agree with you. if we just implement sortByKey, we should not use combiner(it is for combineByKey related API), it will need firstly aggregate values and after sorting, unfold values for same key. In this patch, I would like to reuse current class and fix this bug quickly. for long-term, I think we should write another similar AppendOnlyMap and ExternalAppendOnlyMap class for sortByKey, and ignore functions such as createCombiner, mergeValue, etc. I will try to design these class later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
GitHub user xiajunluan opened a pull request: https://github.com/apache/spark/pull/931 Fix JIRA-983 and support exteranl sort for sortByKey Change class ExternalAppendOnlyMap and make it support customized comparator function(not only sorted by hashCode). You can merge this pull request into a Git repository by running: $ git pull https://github.com/xiajunluan/spark-1 JIRA-983 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/931.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #931 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
