[
https://issues.apache.org/jira/browse/SPARK-49091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-49091:
--------------------------------
Description:
Please apply this patch([^SPARK-49091.patch]) to reproduce this issue. This
issue may cause driver memory leak.
!driver heap.png|thumbnail!
Before SPARK-41914:
{noformat}
[info] BroadcastCleanerSuite:
10:30:16.228 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 0, names:
[info] - Test broadcast cleaner (1 minute, 4 seconds)
10:31:21.552 WARN org.apache.spark.sql.BroadcastCleanerSuite:
{noformat}
After SPARK-41914:
{noformat}
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
[info] *** Test still running after 4 minutes, 58 seconds: suite name:
BroadcastCleanerSuite, test name: Test broadcast cleaner.
...
{noformat}
was:
Please apply this patch([^SPARK-49091.patch]) to reproduce this issue. This
issue may cause driver memory leak.
!driver heap.png|thumbnail!
!driver heap.png!
Before SPARK-41914:
{noformat}
[info] BroadcastCleanerSuite:
10:30:16.228 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 0, names:
[info] - Test broadcast cleaner (1 minute, 4 seconds)
10:31:21.552 WARN org.apache.spark.sql.BroadcastCleanerSuite:
{noformat}
After SPARK-41914:
{noformat}
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
entries size: 2, names: broadcast_1_piece0, broadcast_1
[info] *** Test still running after 4 minutes, 58 seconds: suite name:
BroadcastCleanerSuite, test name: Test broadcast cleaner.
...
{noformat}
> Some broadcasts cannot be cleared from memory storage
> -----------------------------------------------------
>
> Key: SPARK-49091
> URL: https://issues.apache.org/jira/browse/SPARK-49091
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 3.4.0, 3.5.0, 4.0.0
> Reporter: Yuming Wang
> Priority: Major
> Attachments: SPARK-49091.patch, driver heap.png
>
>
> Please apply this patch([^SPARK-49091.patch]) to reproduce this issue. This
> issue may cause driver memory leak.
> !driver heap.png|thumbnail!
> Before SPARK-41914:
> {noformat}
> [info] BroadcastCleanerSuite:
> 10:30:16.228 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 0, names:
> [info] - Test broadcast cleaner (1 minute, 4 seconds)
> 10:31:21.552 WARN org.apache.spark.sql.BroadcastCleanerSuite:
> {noformat}
> After SPARK-41914:
> {noformat}
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 8, names: broadcast_0_piece0, broadcast_0, broadcast_1_piece0,
> broadcast_2_piece0, broadcast_2, broadcast_1, broadcast_3_piece0, broadcast_3
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> entries size: 2, names: broadcast_1_piece0, broadcast_1
> [info] *** Test still running after 4 minutes, 58 seconds: suite name:
> BroadcastCleanerSuite, test name: Test broadcast cleaner.
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
