[jira] [Comment Edited] (SPARK-11293) Spillable collections leak shuffle memory
[
https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249679#comment-15249679
]
Reynold Xin edited comment on SPARK-11293 at 5/3/16 4:31 AM:
-
I was using Apache spark 1.6 in EMR with spark streaming in yarn and saw memory
leaks in one of the containers. Here are the logs
{code}
16/04/14 13:49:10 INFO executor.CoarseGrainedExecutorBackend: Got assigned task
2942916
16/04/14 13:49:10 INFO executor.Executor: Running task 22.0 in stage 35684.0
(TID 2942915)
16/04/14 13:49:10 INFO executor.Executor: Running task 23.0 in stage 35684.0
(TID 2942916)
16/04/14 13:49:10 INFO storage.ShuffleBlockFetcherIterator: Getting 94
non-empty blocks out of 94 blocks
16/04/14 13:49:10 INFO storage.ShuffleBlockFetcherIterator: Getting 94
non-empty blocks out of 94 blocks
16/04/14 13:49:10 INFO storage.ShuffleBlockFetcherIterator: Started 2 remote
fetches in 1 ms
16/04/14 13:49:10 INFO storage.ShuffleBlockFetcherIterator: Started 2 remote
fetches in 1 ms
16/04/14 13:49:10 INFO storage.MemoryStore: Block input-3-1460583424327 stored
as values in memory (estimated size 244.7 KB, free 19.3 MB)
16/04/14 13:49:10 INFO receiver.BlockGenerator: Pushed block
input-3-1460641750200
16/04/14 13:49:10 INFO storage.MemoryStore: 1 blocks selected for dropping
16/04/14 13:49:10 INFO storage.BlockManager: Dropping block
input-1-1460615659379 from memory
16/04/14 13:49:10 INFO storage.MemoryStore: 1 blocks selected for dropping
16/04/14 13:49:10 INFO storage.BlockManager: Dropping block
input-1-1460615659380 from memory
16/04/14 13:49:10 INFO memory.TaskMemoryManager: Memory used in task 2942915
16/04/14 13:49:10 INFO memory.TaskMemoryManager: Acquired by
org.apache.spark.unsafe.map.BytesToBytesMap@34158d5f: 32.3 MB
16/04/14 13:49:10 INFO memory.TaskMemoryManager: 0 bytes of memory were used by
task 2942915 but are not associated with specific consumers
16/04/14 13:49:10 INFO memory.TaskMemoryManager: 101247172 bytes of memory are
used for execution and 3603881260 bytes of memory are used for storage
16/04/14 13:49:10 WARN memory.TaskMemoryManager: leak 32.3 MB memory from
org.apache.spark.unsafe.map.BytesToBytesMap@34158d5f
16/04/14 13:49:10 ERROR executor.Executor: Managed memory leak detected; size =
33816576 bytes, TID = 2942915
16/04/14 13:49:10 ERROR executor.Executor: Exception in task 22.0 in stage
35684.0 (TID 2942915)
java.lang.OutOfMemoryError: Unable to acquire 262144 bytes of memory, got 220032
at
org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:91)
at
org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:735)
at
org.apache.spark.unsafe.map.BytesToBytesMap.(BytesToBytesMap.java:197)
at
org.apache.spark.unsafe.map.BytesToBytesMap.(BytesToBytesMap.java:212)
at
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.(UnsafeFixedWidthAggregationMap.java:103)
at
org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.(TungstenAggregationIterator.scala:483)
at
org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
at
org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/04/14 13:49:10 INFO executor.Executor: Finished task 23.0 in stage 35684.0
(TID 2942916). 1921 bytes result sent to driver
16/04/14 13:49:10 INFO executor.CoarseGrainedExecutorBackend: Got assigned task
2942927
16/04/14 13:49:10 INFO executor.Executor: Running task 34.0 in stage 35684.0
(TID 2942927)
16/04/14 13:49:10 ERROR util.SparkUncaughtExceptionHand
[jira] [Comment Edited] (SPARK-11293) Spillable collections leak shuffle memory
[
https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212497#comment-15212497
]
Imran Rashid edited comment on SPARK-11293 at 3/25/16 10:27 PM:
I've seen a few people misled by the error msg, so I'd like to try to downgrade
that. I've created a separate ticket [SPARK-141868 |
https://issues.apache.org/jira/browse/SPARK-14168] just for changing the msg,
in case there is some fix in store here.
was (Author: irashid):
I've seen a few people misled by the error msg, so I'd like to try to downgrade
that. I've created a separate ticket just for changing the msg, in case there
is some fix in store here.
> Spillable collections leak shuffle memory
> -
>
> Key: SPARK-11293
> URL: https://issues.apache.org/jira/browse/SPARK-11293
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0, 1.6.1
>Reporter: Josh Rosen
>Assignee: Josh Rosen
>Priority: Critical
>
> I discovered multiple leaks of shuffle memory while working on my memory
> manager consolidation patch, which added the ability to do strict memory leak
> detection for the bookkeeping that used to be performed by the
> ShuffleMemoryManager. This uncovered a handful of places where tasks can
> acquire execution/shuffle memory but never release it, starving themselves of
> memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
> memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
> {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing
> its resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11293) Spillable collections leak shuffle memory
[
https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206635#comment-15206635
]
Yi Zhou edited comment on SPARK-11293 at 3/22/16 3:56 PM:
--
seem to hit the issue with Spark 1.6.1 not sure if this is relative to this..if
yes, it can be fixed in Spark 1.6.2 ?
{code}
16/03/22 23:10:26 INFO memory.TaskMemoryManager: Allocate page number 16
(67108864 bytes)
16/03/22 23:10:26 INFO sort.UnsafeExternalSorter: Thread 221 spilling sort data
of 1472.0 MB to disk (0 time so far)
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Allocate page number 1
(1060044737 bytes)
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Memory used in task 9302
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Acquired by
org.apache.spark.shuffle.sort.ShuffleExternalSorter@8bac554: 32.0 KB
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Acquired by
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7a117b4f:
512.0 MB
16/03/22 23:11:26 INFO memory.TaskMemoryManager: 0 bytes of memory were used by
task 9302 but are not associated with specific consumers
16/03/22 23:11:26 INFO memory.TaskMemoryManager: 14909439433 bytes of memory
are used for execution and 1376877 bytes of memory are used for storage
16/03/22 23:11:26 WARN memory.TaskMemoryManager: leak 32.0 KB memory from
org.apache.spark.shuffle.sort.ShuffleExternalSorter@8bac554
16/03/22 23:11:26 ERROR executor.Executor: Managed memory leak detected; size =
32768 bytes, TID = 9302
16/03/22 23:11:26 ERROR executor.Executor: Exception in task 192.0 in stage
153.0 (TID 9302)
java.lang.OutOfMemoryError: Unable to acquire 1073741824 bytes of memory, got
1060044737
at
org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:91)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.growPointerArrayIfNecessary(UnsafeExternalSorter.java:295)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:330)
at
org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:91)
at
org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:168)
at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90)
at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
was (Author: jameszhouyi):
seem to hit the issue with Spark 1.6.1 not sure if this is relative to this..if
yes, it can be fixed in Spark 1.6.2 ?
16/03/22 23:10:26 INFO memory.TaskMemoryManager: Allocate page number 16
(67108864 bytes)
16/03/22 23:10:26 INFO sort.UnsafeExternalSorter: Thread 221 spilling sort data
of 1472.0 MB to disk (0 time so far)
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Allocate page number 1
(1060044737 bytes)
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Memory used in task 9302
16/03/22 23:11:26 INFO memory.TaskMemoryManager: Acquired by
org.apache.spark.shuf
[jira] [Comment Edited] (SPARK-11293) Spillable collections leak shuffle memory
[
https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204064#comment-15204064
]
Damian Momot edited comment on SPARK-11293 at 3/21/16 11:36 AM:
It looks we are getting same errors on 1.6.0 too - any plans to address it?
was (Author: daimon):
It looks we are getting same errors on 1.6.0 too
> Spillable collections leak shuffle memory
> -
>
> Key: SPARK-11293
> URL: https://issues.apache.org/jira/browse/SPARK-11293
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
>Reporter: Josh Rosen
>Assignee: Josh Rosen
>Priority: Critical
>
> I discovered multiple leaks of shuffle memory while working on my memory
> manager consolidation patch, which added the ability to do strict memory leak
> detection for the bookkeeping that used to be performed by the
> ShuffleMemoryManager. This uncovered a handful of places where tasks can
> acquire execution/shuffle memory but never release it, starving themselves of
> memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
> memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
> {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing
> its resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
