[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 Sorry,I download the master of the branch 2.1.My issue also mentioned to remove the code, and should not be Resolution: Won't Fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17220 Well, I still saw "tungsten-sort" in branch 2.1 and master (https://github.com/apache/spark/blob/branch-2.1/core/src/main/scala/org/apache/spark/SparkEnv.scala#L320). Can you tell which code did you check? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 spark2.0.2 val shortShuffleMgrNames = Map( "sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName, "tungsten-sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName) spark2.1.0 val shortShuffleMgrNames = Map( "sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName) The above is based on the analysis of release the original. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17220 What's the meaning of "has been deleted in Spark 2.1.0"? I think the reason mention above is quite clear. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 @jerryshao@rxin@srowen In spark2.1.0,"tungsten-sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName has been deleted,but you didn't agree with my issue SPARK-19862.why? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17220 Hash-based shuffle has some problems with large number of partitions, and part of hash-based shuffle's feature has already been incorporated into sort-based shuffle. Spark's sort-based shuffle is not pure sort-based like MR, it is actually a mixed pattern depends on the partition numbers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 why HashShuffleManager have been deleted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 thanks.I understand this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17220 @guoxiaolongzte , I think here though "tungsten-sort" is the same as "sort" now, for the configuration back-compatibility we still need to keep it. If somehow user still configured with "tungsten-sort", with your change the application will be failed. I think that's what @rxin mentioned about. Looking at some other configurations, we typically keep backward compatibility unless major release (Spark 1.6 to Spark 2.0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 I think I should delete, update in the document at the same time, so that to ensure the uniqueness of function. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 I don't think you understand this. This value is here so if at some point some user picked tungsten-sort, we won't break it. In recent versions of Spark the default sort manager accomplishes the thing as the old tungsten sort. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 If anything, we should just update the file to add a line of comment to make sure people don't delete this in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 I think the compatibility, the resulting shuffle manager is not I want.Only the parameter values' sort real SortShuffleManager said. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 Is this change even correct? This is here for backward compatibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17220 Ok, I have modified the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
