[jira] [Commented] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups
[
https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139658#comment-15139658
]
Thomas Graves commented on SPARK-11316:
---
So we ran into this again, here is the scenario and what is happening:
UnionRDD is being coalesced. The UnionRDD is made up of mapPartitionRDD with
not preferred locations and a checkpointedRDD with preferred locations.
Its coalescing to a > number of partitions but its not using shuffle so its
going to coalesce to same number of partitions. The UnionRDD has 2 Rdd's, one
with 1020 in MapPartitionsRDD and 960 in CheckPointedRDD, thus its coalescing
from 1980 to 1980. It goes into the setupGroups called to setup 1980 groups,
but since the MapPartitionsRDD doesn't have preferred locations it only has 960
actual preferred locations. It goes through the first while loop and create
partitionsGroups for each of the hosts possible until it hits expectedCoupons2
number. In this has it hits 1661, so it created groups for 1661 of 1980 and a
bunch of those groups got partitions assigned (out of the 960).
It then enters the second while loop to go through the rest of the
1980-1661=319 groups it needs. Here though for each of the 319 iterations it
goes into the inner while loop while (!addPartToPGroup(nxt_part, pgroup) &&
tries < targetLen) trying to add a partition to each group. In this case since
there are less partitions then groups it ends up walking through targetLen
almost all of the times and never adding a partition to the group because all
the partitions are already assigned to groups (because we only have 960
partitions to put into 1980 groups). The entire process of 319 * 1980 tries
takes over 15 minutes (3 seconds per 319 interation).
> isEmpty before coalesce seems to cause huge performance issue in setupGroups
>
>
> Key: SPARK-11316
> URL: https://issues.apache.org/jira/browse/SPARK-11316
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>Priority: Critical
>
> So I haven't fully debugged this yet but reporting what I'm seeing and think
> might be going on.
> I have a graph processing job that is seeing huge slow down in setupGroups in
> the location iterator where its getting the preferred locations for the
> coalesce. They are coalescing from 2400 down to 1200 and its taking 17+
> hours to do the calculation. Killed it at this point so don't know total
> time.
> It appears that the job is doing an isEmpty call, a bunch of other
> transformation, then a coalesce (where it takes so long), other
> transformations, then finally a count to trigger it.
> It appears that there is only one node that its finding in the setupGroup
> call and to get to that node it has to first to through the while loop:
> while (numCreated < targetLen && tries < expectedCoupons2) {
> where expectedCoupons2 is around 19000. It finds very few or none in this
> loop.
> Then it does the second loop:
> while (numCreated < targetLen) { // if we don't have enough partition
> groups, create duplicates
> var (nxt_replica, nxt_part) = rotIt.next()
> val pgroup = PartitionGroup(nxt_replica)
> groupArr += pgroup
> groupHash.getOrElseUpdate(nxt_replica, ArrayBuffer()) += pgroup
> var tries = 0
> while (!addPartToPGroup(nxt_part, pgroup) && tries < targetLen) { //
> ensure at least one part
> nxt_part = rotIt.next()._2
> tries += 1
> }
> numCreated += 1
> }
> Where it has an inner while loop and both of those are going 1200 times.
> 1200*1200 loops. This is taking a very long time.
> The user can work around the issue by adding in a count() call very close to
> after the isEmpty call before the coalesce is called. I also tried putting
> in a take(1) right before the isEmpty call and it seems to work around
> the issue, took 1 hours with the take vs a few minutes with the count().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups
[
https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131196#comment-15131196
]
Apache Spark commented on SPARK-11316:
--
User 'zhuoliu' has created a pull request for this issue:
https://github.com/apache/spark/pull/11060
> isEmpty before coalesce seems to cause huge performance issue in setupGroups
>
>
> Key: SPARK-11316
> URL: https://issues.apache.org/jira/browse/SPARK-11316
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>Priority: Critical
>
> So I haven't fully debugged this yet but reporting what I'm seeing and think
> might be going on.
> I have a graph processing job that is seeing huge slow down in setupGroups in
> the location iterator where its getting the preferred locations for the
> coalesce. They are coalescing from 2400 down to 1200 and its taking 17+
> hours to do the calculation. Killed it at this point so don't know total
> time.
> It appears that the job is doing an isEmpty call, a bunch of other
> transformation, then a coalesce (where it takes so long), other
> transformations, then finally a count to trigger it.
> It appears that there is only one node that its finding in the setupGroup
> call and to get to that node it has to first to through the while loop:
> while (numCreated < targetLen && tries < expectedCoupons2) {
> where expectedCoupons2 is around 19000. It finds very few or none in this
> loop.
> Then it does the second loop:
> while (numCreated < targetLen) { // if we don't have enough partition
> groups, create duplicates
> var (nxt_replica, nxt_part) = rotIt.next()
> val pgroup = PartitionGroup(nxt_replica)
> groupArr += pgroup
> groupHash.getOrElseUpdate(nxt_replica, ArrayBuffer()) += pgroup
> var tries = 0
> while (!addPartToPGroup(nxt_part, pgroup) && tries < targetLen) { //
> ensure at least one part
> nxt_part = rotIt.next()._2
> tries += 1
> }
> numCreated += 1
> }
> Where it has an inner while loop and both of those are going 1200 times.
> 1200*1200 loops. This is taking a very long time.
> The user can work around the issue by adding in a count() call very close to
> after the isEmpty call before the coalesce is called. I also tried putting
> in a take(1) right before the isEmpty call and it seems to work around
> the issue, took 1 hours with the take vs a few minutes with the count().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups
[
https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974236#comment-14974236
]
Thomas Graves commented on SPARK-11316:
---
Note I"m wondering if since the isEmpty call is doing a take(1) if its only
finding 1 locations and thus throwing off the setupGroups call.
> isEmpty before coalesce seems to cause huge performance issue in setupGroups
>
>
> Key: SPARK-11316
> URL: https://issues.apache.org/jira/browse/SPARK-11316
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>Priority: Critical
>
> So I haven't fully debugged this yet but reporting what I'm seeing and think
> might be going on.
> I have a graph processing job that is seeing huge slow down in setupGroups in
> the location iterator where its getting the preferred locations for the
> coalesce. They are coalescing from 2400 down to 1200 and its taking 17+
> hours to do the calculation. Killed it at this point so don't know total
> time.
> It appears that the job is doing an isEmpty call, a bunch of other
> transformation, then a coalesce (where it takes so long), other
> transformations, then finally a count to trigger it.
> It appears that there is only one node that its finding in the setupGroup
> call and to get to that node it has to first to through the while loop:
> while (numCreated < targetLen && tries < expectedCoupons2) {
> where expectedCoupons2 is around 19000. It finds very few or none in this
> loop.
> Then it does the second loop:
> while (numCreated < targetLen) { // if we don't have enough partition
> groups, create duplicates
> var (nxt_replica, nxt_part) = rotIt.next()
> val pgroup = PartitionGroup(nxt_replica)
> groupArr += pgroup
> groupHash.getOrElseUpdate(nxt_replica, ArrayBuffer()) += pgroup
> var tries = 0
> while (!addPartToPGroup(nxt_part, pgroup) && tries < targetLen) { //
> ensure at least one part
> nxt_part = rotIt.next()._2
> tries += 1
> }
> numCreated += 1
> }
> Where it has an inner while loop and both of those are going 1200 times.
> 1200*1200 loops. This is taking a very long time.
> The user can work around the issue by adding in a count() call very close to
> after the isEmpty call before the coalesce is called. I also tried putting
> in a take(1) right before the isEmpty call and it seems to work around
> the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
