Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
@hvanhovell, thanks for your suggestion. I will change the PR description
as suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
Please retest!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
@hvanhovell , I have already updated the description and explained how the
PR fixed it. Could you please take another look?
---
If your project is set up for it, you can reply to this email
Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
Here is my understanding: after spilling, it would call reset() to release
the memory. In the reset() function, it deletes all the memory pages, but it
did not release any memory from longArray
Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
This is production query. Sorry, I could not share it. It is doing a join
between two big tables.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user jiexiong commented on a diff in the pull request:
https://github.com/apache/spark/pull/15722#discussion_r86048313
--- Diff:
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ---
@@ -903,11 +906,12 @@ public void reset() {
numKeys = 0
Github user jiexiong commented on the issue:
https://github.com/apache/spark/pull/15722
Here is the query:
Here is the query:
INSERT OVERWRITE TABLE
lookalike_trainer_campaign_conv_users_with_country_shadow
PARTITION(ds='2016-10-19')
SELECT c.source_id
GitHub user jiexiong opened a pull request:
https://github.com/apache/spark/pull/15722
[SPARK-18208] [Shuffle] Executor OOM due to a memory leak in BytesToBytesMap
## What changes were proposed in this pull request?
Fixed the OOM problem in BytesToBytesMap.
## How
