[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:07 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:04 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:04 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:03 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan commented on TEZ-3809:
--
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680| 32 |66,537,624 | 2.88%
| |-
Success: TEZ-3810 PreCommit Build #2597
Jira: https://issues.apache.org/jira/browse/TEZ-3810
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2597/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 339.59 KB...]
[INFO] Tez SUCCESS [ 0.025 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 55:29 min
[INFO] Finished at: 2017-08-02T17:13:37Z
[INFO] Final Memory: 89M/1585M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880063/TEZ-3810-001.patch
against master revision 614937c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2597//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2597//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
ded15c2f828ef98c8d997d6f2e88582eacb7f80d logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3810
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3810) TezCounter for idle time in shuffle phase
[
https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111340#comment-16111340
]
TezQA commented on TEZ-3810:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880063/TEZ-3810-001.patch
against master revision 614937c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2597//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2597//console
This message is automatically generated.
> TezCounter for idle time in shuffle phase
> -
>
> Key: TEZ-3810
> URL: https://issues.apache.org/jira/browse/TEZ-3810
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ashwin Ramesh
> Attachments: TEZ-3810-001.patch
>
>
> A task attempt counter that tracks how much time was spent waiting for
> inputs in the shuffle phase. We can use this to quickly identify jobs that
> are wasting a lot of time on the grid with idle reducer tasks instead of
> shuffling/merging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1661#comment-1661 ] Ashwin Ramesh commented on TEZ-3810: [~kshukla] Thanks for the review, will look at that immediately. > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > Attachments: TEZ-3810-001.patch > > > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. We can use this to quickly identify jobs that > are wasting a lot of time on the grid with idle reducer tasks instead of > shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1642#comment-1642 ] Kuhu Shukla commented on TEZ-3810: -- Thanks [~aramesh2] for the patch! I haven't fully reviewed this patch, but I think we will need an equivalent change in ShuffleScheduler as well (which is the shuffle-r for the ordered case). > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > Attachments: TEZ-3810-001.patch > > > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. We can use this to quickly identify jobs that > are wasting a lot of time on the grid with idle reducer tasks instead of > shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Ramesh updated TEZ-3810: --- Attachment: TEZ-3810-001.patch > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > Attachments: TEZ-3810-001.patch > > > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. We can use this to quickly identify jobs that > are wasting a lot of time on the grid with idle reducer tasks instead of > shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Ramesh updated TEZ-3810: --- Description: A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase. We can use this to quickly identify jobs that are wasting a lot of time on the grid with idle reducer tasks instead of shuffling/merging. was: We can use this to quickly identify jobs that are wasting a lot of time on the grid with idle reducer tasks instead of shuffling/merging. > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. We can use this to quickly identify jobs that > are wasting a lot of time on the grid with idle reducer tasks instead of > shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Ramesh updated TEZ-3810: --- Summary: TezCounter for idle time in shuffle phase (was: A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase. ) > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > > We can use this to quickly identify jobs that are wasting a lot of time on > the grid with idle reducer tasks instead of shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3810) A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase.
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Ramesh updated TEZ-3810: --- Priority: Major (was: Minor) > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh > > We can use this to quickly identify jobs that are wasting a lot of time on > the grid with idle reducer tasks instead of shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3810) A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase.
Ashwin Ramesh created TEZ-3810: -- Summary: A task attempt counter that tracks how much time was spent waiting for inputs in the shuffle phase. Key: TEZ-3810 URL: https://issues.apache.org/jira/browse/TEZ-3810 Project: Apache Tez Issue Type: Improvement Reporter: Ashwin Ramesh Priority: Minor We can use this to quickly identify jobs that are wasting a lot of time on the grid with idle reducer tasks instead of shuffling/merging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
