Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
Ok, I have modified the title.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabl
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17220
Is this change even correct? This is here for backward compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project doe
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
I think the compatibility, the resulting shuffle manager is not I want.Only
the parameter values' sort real SortShuffleManager said.
---
If your project is set up for it, you can reply to th
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17220
I don't think you understand this. This value is here so if at some point
some user picked tungsten-sort, we won't break it. In recent versions of Spark
the default sort manager accomplishes the thing
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17220
If anything, we should just update the file to add a line of comment to
make sure people don't delete this in the future.
---
If your project is set up for it, you can reply to this email and have yo
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
I think I should delete, update in the document at the same time, so that
to ensure the uniqueness of function.
---
If your project is set up for it, you can reply to this email and have you
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17220
@guoxiaolongzte , I think here though "tungsten-sort" is the same as "sort"
now, for the configuration back-compatibility we still need to keep it. If
somehow user still configured with "tungsten-
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
thanks.I understand this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled an
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
why HashShuffleManager have been deleted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17220
Hash-based shuffle has some problems with large number of partitions, and
part of hash-based shuffle's feature has already been incorporated into
sort-based shuffle. Spark's sort-based shuffle is
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
@jerryshao@rxin@srowen
In spark2.1.0,"tungsten-sort" ->
classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName has been
deleted,but you didn't agree with my issue SPARK-19862
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17220
What's the meaning of "has been deleted in Spark 2.1.0"? I think the reason
mention above is quite clear.
---
If your project is set up for it, you can reply to this email and have your
reply app
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
spark2.0.2
val shortShuffleMgrNames = Map(
"sort" ->
classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName,
"tungsten-sort" ->
classOf[org.apache.s
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17220
Well, I still saw "tungsten-sort" in branch 2.1 and master
(https://github.com/apache/spark/blob/branch-2.1/core/src/main/scala/org/apache/spark/SparkEnv.scala#L320).
Can you tell which co
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17220
Sorry,I download the master of the branch 2.1.My issue also mentioned to
remove the code, and should not be Resolution: Won't Fix.
---
If your project is set up for it, you can reply to this
15 matches
Mail list logo
