[jira] [Updated] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3212: - Attachment: tez-3212.004.patch Thanks [~rajesh.balamohan] for the suggestion. I moved the common code for creating the new array to a separate method (instead of the common code for computing the new array length). > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:03 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:04 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:07 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan commented on TEZ-3809:
--
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680| 32 |66,537,624 | 2.88%
| |-
[jira] [Comment Edited] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111794#comment-16111794
]
Muhammad Samir Khan edited comment on TEZ-3809 at 8/2/17 10:04 PM:
---
Took a heap dump on ordered word count before final merge. In the after case,
one of the outputs was written to disk instead of kept in memory and that is
why it has 37 entries.
Before:
Class Name
| Shallow Heap | Retained Heap | Percentage
---
java.lang.Thread @ 0x5d2c473f8 ShuffleAndMergeRunner {Tokenizer} Thread
| 120 | 2,229,207,992 | 96.48%
|- java.util.ArrayList @ 0x73f978f10
| 24 | 2,229,206,760 | 96.48%
| '- java.lang.Object[38] @ 0x73f979130
| 168 | 2,229,206,736 | 96.48%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88898| 32 |68,078,192 | 2.95%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b260| 32 |67,839,520 | 2.94%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a888b8| 32 |67,700,608 | 2.93%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73f9db168| 32 |67,500,816 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab36218| 32 |67,408,704 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deed28| 32 |67,367,424 | 2.92%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x743b86ee0| 32 |67,337,936 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af3a698| 32 |67,300,896 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c5b8| 32 |67,282,464 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab33140| 32 |67,264,304 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5e4a88878| 32 |67,127,368 | 2.91%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0b218| 32 |67,098,216 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c6c8| 32 |67,064,504 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d239a6c8| 32 |67,003,776 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x5d23b7e10| 32 |66,965,296 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631def2b8| 32 |66,928,032 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60ab351d0| 32 |66,916,896 | 2.90%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x74805dfb8| 32 |66,886,272 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39598| 32 |66,718,800 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x73fb0fb78| 32 |66,688,296 | 2.89%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c4b0| 32 |66,656,312 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x60af39578| 32 |66,629,936 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631deec30| 32 |66,584,576 | 2.88%
| |-
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
@ 0x631e0c680|
[jira] [Updated] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3809: - Attachment: TEZ-3809.001.patch Initial patch with the changes mentioned in the description. The changes have not been extensively tested. Putting up the patch to start the discussion. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This can create a > mismatch between the primary input buffers which don't have checksum and the > merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
Muhammad Samir Khan created TEZ-3809: Summary: The buffer size allocated for InMemoryMapOutput can be optimized Key: TEZ-3809 URL: https://issues.apache.org/jira/browse/TEZ-3809 Project: Apache Tez Issue Type: Bug Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This can create a mismatch between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3809: - Description: Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This creates an inconsistency between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. -InMemoryWriter does not account for two EOF_MARKERs written at close() in its accounting so that the getRawLength() method is off by two bytes. was: Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This can create a mismatch between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. -InMemoryWriter does not account for two EOF_MARKERs written at close() in its accounting so that the getRawLength() method is off by two bytes. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3809: - Description: Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This can create a mismatch between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. -InMemoryWriter does not account for two EOF_MARKERs written at close() in its accounting so that the getRawLength() method is off by two bytes. was: Related jiras: TEZ-3752 and TEZ-3732. -When shuffling input to memory, the decompressed length is used to create the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 bytes less (the IFile header). These 4 bytes can optimized and, in an extreme case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers to allocate the new InMemoryMapOutput. However, each input has two EOF_MARKERs while only two are needed at the end. -InMemoryWriter wraps the output BoundedByteArrayOutputStream in IFileOutputStream which will write checksum at close. This can create a mismatch between the primary input buffers which don't have checksum and the merged buffers which do. IFileOutputStream wrap can be removed to save 4 bytes per merged buffers. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This can create a > mismatch between the primary input buffers which don't have checksum and the > merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3813: - Attachment: TEZ-3813.001.patch Replace using BoundedByteArrayOutputStream with a byte array in MemoryFetchedInput. > Reduce Object size of MemoryFetchedInput for large jobs > --- > > Key: TEZ-3813 > URL: https://issues.apache.org/jira/browse/TEZ-3813 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3813.001.patch > > > Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a > BoundedByteArrayOutputStream that is not used (only the underlying byte[] is > used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
Muhammad Samir Khan created TEZ-3813: Summary: Reduce Object size of MemoryFetchedInput for large jobs Key: TEZ-3813 URL: https://issues.apache.org/jira/browse/TEZ-3813 Project: Apache Tez Issue Type: Bug Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a BoundedByteArrayOutputStream that is not used (only the underlying byte[] is used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113366#comment-16113366 ] Muhammad Samir Khan commented on TEZ-3809: -- Also tested with memory-to-memory merger to check if the output is the same. For the unordered case, used filterLinesByWord and compared output before and after. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113451#comment-16113451
]
Muhammad Samir Khan commented on TEZ-3813:
--
*JOL Dump:*
+Before:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7a 12 01 f8 (0010 00010010
0001 1000) (-134147462)
12 4
int FetchedInput.id 0
16 8
long FetchedInput.actualSize 0
24 8
long FetchedInput.compressedSize 0
32 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
36 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type
FetchedInput.type (object)
40 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
44 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.State
FetchedInput.state(object)
48 4
org.apache.hadoop.io.BoundedByteArrayOutputStream MemoryFetchedInput.byteStream
(object)
52 4
(loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
23264 [C
22448 java.lang.String
13232
org.apache.hadoop.io.BoundedByteArrayOutputStream
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
15656
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
9 264 (total)
{code}
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113454#comment-16113454 ] Muhammad Samir Khan commented on TEZ-3813: -- Tested with filterLinesByWord and compared output before and after. > Reduce Object size of MemoryFetchedInput for large jobs > --- > > Key: TEZ-3813 > URL: https://issues.apache.org/jira/browse/TEZ-3813 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3813.001.patch > > > Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a > BoundedByteArrayOutputStream that is not used (only the underlying byte[] is > used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113369#comment-16113369 ] Muhammad Samir Khan commented on TEZ-3809: -- Request for comments [~rajesh.balamohan]/[~sseth] > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3159: - Attachment: TEZ-3159.004.patch Incorporated comments from the review. Removed the alignment to power of two. When growing the first buffer, it is set to the max of 2*existing length and the desired length, similar to ByteArrayOutputStream. Also added FlexibleDataInputBuffer and the relevant logic in IFile.Reader. However, to use it, we will have to change the TezRawKeyValueIterator interface and its implementing classes, which is a bigger change. > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > Attachments: TEZ-3159.001.patch, TEZ-3159.002.patch, > TEZ-3159.003.patch, TEZ-3159.004.patch > > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3159: - Attachment: TEZ-3159.005.patch Fixes to findbugs etc. Added unit test for FlexibleDataInputBuffer. Fixed a couple of bugs with the previous iteration of FlexibleDataInputBuffer. Some changes to unit test for FlexibleDataOutputBuffer. > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > Attachments: TEZ-3159.001.patch, TEZ-3159.002.patch, > TEZ-3159.003.patch, TEZ-3159.004.patch, TEZ-3159.005.patch > > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3159: - Attachment: TEZ-3159.001.patch Added a new class that grows buffers via "doubling" until it hits a threshold and then adds a new buffer to a list. Also modified IFile.Writer to work with it. > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > Attachments: TEZ-3159.001.patch > > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3816) Ability to automatically speculate single-task vertices
[ https://issues.apache.org/jira/browse/TEZ-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3816: - Description: When a single-task vertex is unlucky, it lands on a very slow node. Speculation doesn't currently apply when there are no other tasks to compare with. It would be good to have a configurable timeout after which the tasks automatically speculate. (was: When a single-task vertex is unlucky, it lands on a very slow node. Speculation doesn't currently apply when there are no other tasks to compare with. It would be good to either have a configurable timeout after which the tasks automatically speculate.) > Ability to automatically speculate single-task vertices > --- > > Key: TEZ-3816 > URL: https://issues.apache.org/jira/browse/TEZ-3816 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3816.001.patch > > > When a single-task vertex is unlucky, it lands on a very slow node. > Speculation doesn't currently apply when there are no other tasks to compare > with. It would be good to have a configurable timeout after which the tasks > automatically speculate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3159: - Attachment: TEZ-3159.003.patch Some changes to ensureExtraCapacity to create a separate method for growing the last buffer and fixed a bug with creating new buffers. Also added a "unit test" for comparing runtime against ByteArrayOutputStream. FlexibleByteArrayOutputStream: 110ms ByteArrayOutputStream: 95ms > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > Attachments: TEZ-3159.001.patch, TEZ-3159.002.patch, > TEZ-3159.003.patch > > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3816) Ability to automatically speculate single-task vertices
[ https://issues.apache.org/jira/browse/TEZ-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3816: - Attachment: TEZ-3816.002.patch Thanks [~sseth] for the comments. I updated the patch to put the logic in LegacySpeculator. Also added a unit test. If there is an existing speculation attempt, then speculationValue() will return ALREADY_SPECULATING which will prevent more speculations. Tested the updated patch with mrrsleep. > Ability to automatically speculate single-task vertices > --- > > Key: TEZ-3816 > URL: https://issues.apache.org/jira/browse/TEZ-3816 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3816.001.patch, TEZ-3816.002.patch > > > When a single-task vertex is unlucky, it lands on a very slow node. > Speculation doesn't currently apply when there are no other tasks to compare > with. It would be good to have a configurable timeout after which the tasks > automatically speculate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3816) Ability to automatically speculate single-task vertices
[ https://issues.apache.org/jira/browse/TEZ-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3816: - Attachment: TEZ-3816.003.patch Added a couple of comments and speculationValue returns ON_SCHEDULE if the single task has not timed out yet. > Ability to automatically speculate single-task vertices > --- > > Key: TEZ-3816 > URL: https://issues.apache.org/jira/browse/TEZ-3816 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3816.001.patch, TEZ-3816.002.patch, > TEZ-3816.003.patch > > > When a single-task vertex is unlucky, it lands on a very slow node. > Speculation doesn't currently apply when there are no other tasks to compare > with. It would be good to have a configurable timeout after which the tasks > automatically speculate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3753) Improve error message in IFile for buffer length overflow
[ https://issues.apache.org/jira/browse/TEZ-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3753: - Attachment: tez-3753.002.patch Updated patch to include a unit test and some changes based on feedback from [~jeagles] and [~rohini] > Improve error message in IFile for buffer length overflow > - > > Key: TEZ-3753 > URL: https://issues.apache.org/jira/browse/TEZ-3753 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.1 >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3753.001.patch, tez-3753.002.patch > > > When a record size is too big and the byte array doubling expansion crosses > 2G the array size overflows and becomes negative. It would be good to fail > the code paths saying record is too big so that error is easy to understand > for users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3212: Assignee: Muhammad Samir Khan (was: Jonathan Eagles) > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096906#comment-16096906 ] Muhammad Samir Khan commented on TEZ-3212: -- Added unit test and throw an error message when the length exceeds the max. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3212: - Attachment: tez-3212.002.patch > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reopened TEZ-3212: -- > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3752) Reduce Object size of InMemoryMapOutput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103716#comment-16103716 ] Muhammad Samir Khan commented on TEZ-3752: -- Ran orderedwordcount with -Dtez.shuffle-vertex-manager.enable.auto-parallel=true -Dtez.runtime.io.sort.factor=4 -Dtez.runtime.shuffle.memory-to-memory.enable=true. Sorted the output (via sort) and diff'd against the output from orderedwordcount without the changes. Also turned on the '"writeFile SAME_KEY count=" + count' log line in TezMerger.writeFile to ensure we hit the RLE case with in memory merge: 2017-07-27 18:19:18,128 [INFO] [MemToMemMerger [Tokenizer]] |orderedgrouped.MergeManager|: Tokenizer: Initiating Memory-to-Memory merge with 4 segments of total-size: 22182024 2017-07-27 18:19:18,770 [INFO] [MemToMemMerger [Tokenizer]] |impl.TezMerger|: writeFile SAME_KEY count=1544269 2017-07-27 18:19:18,771 [INFO] [MemToMemMerger [Tokenizer]] |orderedgrouped.MergeManager|: Tokenizer Memory-to-Memory merge of the 4 files in-memory complete with mergeOutputSize=22182024 > Reduce Object size of InMemoryMapOutput for large jobs > -- > > Key: TEZ-3752 > URL: https://issues.apache.org/jira/browse/TEZ-3752 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: TEZ-3752.001.patch > > > Follow-on jira from TEZ-3732. The InMemoryMapOutput has a > BoundedByteArrayOutputStream that is only used in the Merged MapOutput case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3212: - Attachment: tez-3212.003.patch 1) Added MAX_BUFFER_SIZE not final to findbugs-exclude.xml 2) Documented MAX_BUFFER_SIZE directly in the comments and removed the link to stackoverflow 3) Changed the NegativeArraySizeException to IllegalArgumentException > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3752) Reduce Object size of InMemoryMapOutput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103880#comment-16103880 ] Muhammad Samir Khan commented on TEZ-3752: -- However, this test doesn't actually hit the RLE case. InMemoryWriter has RLE turned off since the Writer constructor it calls has rle flag set to false. > Reduce Object size of InMemoryMapOutput for large jobs > -- > > Key: TEZ-3752 > URL: https://issues.apache.org/jira/browse/TEZ-3752 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: TEZ-3752.001.patch > > > Follow-on jira from TEZ-3732. The InMemoryMapOutput has a > BoundedByteArrayOutputStream that is only used in the Merged MapOutput case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3752) Reduce Object size of InMemoryMapOutput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3752: - Attachment: TEZ-3752.001.patch Replace using BoundedByteArrayOutputStream with a byte array in InMemoryMapOutput. > Reduce Object size of InMemoryMapOutput for large jobs > -- > > Key: TEZ-3752 > URL: https://issues.apache.org/jira/browse/TEZ-3752 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3752.001.patch > > > Follow-on jira from TEZ-3732. The InMemoryMapOutput has a > BoundedByteArrayOutputStream that is only used in the Merged MapOutput case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3752) Reduce Object size of InMemoryMapOutput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102046#comment-16102046
]
Muhammad Samir Khan commented on TEZ-3752:
--
JOL dump:
Before:
-internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,boolean,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
object internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01
00 00 00 (0001 ) (1)
4 4
(object header) 00
00 00 00 ( ) (0)
8 4
(object header) 78
12 01 f8 (0000 00010010 0001 1000) (-134147464)
12 4
int MapOutput.id 1
16 1
boolean MapOutput.primaryMapOutputfalse
17 3
(alignment/padding gap)
20 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
MapOutput.attemptIdentifier null
24 4
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped
MapOutput.callbacknull
28 4
org.apache.hadoop.io.BoundedByteArrayOutputStream InMemoryMapOutput.byteStream
(object)
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
-footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,boolean,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput@10bdf5e5d
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
13232
org.apache.hadoop.io.BoundedByteArrayOutputStream
13232
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
3 80 (total)
{code}
After:
-internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput(org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetchedInputAllocatorOrderedGrouped,long,boolean,org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$1)
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput$InMemoryMapOutput
object internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105671#comment-16105671 ] Muhammad Samir Khan commented on TEZ-3212: -- The failing tests pass locally. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3807) InMemoryWriter is not tested with RLE enabled
Muhammad Samir Khan created TEZ-3807: Summary: InMemoryWriter is not tested with RLE enabled Key: TEZ-3807 URL: https://issues.apache.org/jira/browse/TEZ-3807 Project: Apache Tez Issue Type: Test Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan In TestIFile, A couple of test cases are supposed to test InMemoryWriter with RLE enabled but the RLE flag is turned off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3807) InMemoryWriter is not tested with RLE enabled
[ https://issues.apache.org/jira/browse/TEZ-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3807: - Attachment: TEZ-3807.001.patch Fix testInMemoryWriter() and add test cases to testWithRLEMarker() to also test InMemoryWriter. > InMemoryWriter is not tested with RLE enabled > - > > Key: TEZ-3807 > URL: https://issues.apache.org/jira/browse/TEZ-3807 > Project: Apache Tez > Issue Type: Test >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3807.001.patch > > > In TestIFile, A couple of test cases are supposed to test InMemoryWriter with > RLE enabled but the RLE flag is turned off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
[ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3212: - Attachment: tez-3212.005.patch Made createLargerArray() static. > IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB > --- > > Key: TEZ-3212 > URL: https://issues.apache.org/jira/browse/TEZ-3212 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Muhammad Samir Khan > Attachments: tez-3212.002.patch, tez-3212.003.patch, > tez-3212.004.patch, tez-3212.005.patch, TEZ-3212.1.patch > > > This is not a regression with respect to MR, just an issue that was > encountered with a job whose IFile record values (which can be of max size > 2GB) which can be successfully written but not successfully read. > Failure while running task:java.lang.NegativeArraySizeException > at > org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3159: Assignee: Muhammad Samir Khan > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3807) InMemoryWriter is not tested with RLE enabled
[ https://issues.apache.org/jira/browse/TEZ-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108089#comment-16108089 ] Muhammad Samir Khan commented on TEZ-3807: -- Fixed both. > InMemoryWriter is not tested with RLE enabled > - > > Key: TEZ-3807 > URL: https://issues.apache.org/jira/browse/TEZ-3807 > Project: Apache Tez > Issue Type: Test >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3807.001.patch, TEZ-3807.002.patch > > > In TestIFile, A couple of test cases are supposed to test InMemoryWriter with > RLE enabled but the RLE flag is turned off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3807) InMemoryWriter is not tested with RLE enabled
[ https://issues.apache.org/jira/browse/TEZ-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3807: - Attachment: TEZ-3807.002.patch > InMemoryWriter is not tested with RLE enabled > - > > Key: TEZ-3807 > URL: https://issues.apache.org/jira/browse/TEZ-3807 > Project: Apache Tez > Issue Type: Test >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3807.001.patch, TEZ-3807.002.patch > > > In TestIFile, A couple of test cases are supposed to test InMemoryWriter with > RLE enabled but the RLE flag is turned off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3813: - Attachment: TEZ-3813.004.patch Added getBytes() to findbugs exclude file. > Reduce Object size of MemoryFetchedInput for large jobs > --- > > Key: TEZ-3813 > URL: https://issues.apache.org/jira/browse/TEZ-3813 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch, > TEZ-3813.003.patch, TEZ-3813.004.patch > > > Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a > BoundedByteArrayOutputStream that is not used (only the underlying byte[] is > used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.005.patch
Removed int size from MemoryFetchedInput.
*JOL Dump:*
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7d 12 01 f8 (0101 00010010
0001 1000) (-134147459)
12 4
int FetchedInput.id 0
16 1
byte FetchedInput.state0
17 3
(alignment/padding gap)
20 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
24 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
28 4
byte[] MemoryFetchedInput.byteArray []
Instance size: 32 bytes
Space losses: 3 bytes internal + 0 bytes external = 3 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
13232
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
2 48 (total)
{code}
> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch,
> TEZ-3813.003.patch, TEZ-3813.004.patch, TEZ-3813.005.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is
> used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3159) Reduce memory utilization while serializing keys and values
[ https://issues.apache.org/jira/browse/TEZ-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3159: - Attachment: TEZ-3159.002.patch Fixed findbugs and release audit warnings. > Reduce memory utilization while serializing keys and values > --- > > Key: TEZ-3159 > URL: https://issues.apache.org/jira/browse/TEZ-3159 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Muhammad Samir Khan > Attachments: TEZ-3159.001.patch, TEZ-3159.002.patch > > > Currently DataOutputBuffer is used for serializing. The underlying buffer > keeps doubling in size when it reaches capacity. In some of the Pig scripts > which serialize big bags, we end up with OOM in Tez as there is no space to > double the array size. Mapreduce mode runs fine in those cases with 1G heap. > The scenarios are > - When combiner runs in reducer and some of the fields after combining > are still big bags (For eg: distinct). Currently with mapreduce combiner does > not run in reducer - MAPREDUCE-5221. Since input sort buffers hold good > amount of memory at that time it can easily go OOM. >- While serializing output with bags when there are multiple inputs and > outputs and the sort buffers for those take up space. > It is a pain especially after buffer size hits 128MB. Doubling at 128MB will > require 128MB (existing array) +256MB (new array). Any doubling after that > requires even more space. But most of the time the data is probably not going > to fill up that 256MB leading to wastage. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[ https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3813: - Attachment: TEZ-3813.006.patch Fixed > Reduce Object size of MemoryFetchedInput for large jobs > --- > > Key: TEZ-3813 > URL: https://issues.apache.org/jira/browse/TEZ-3813 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch, > TEZ-3813.003.patch, TEZ-3813.004.patch, TEZ-3813.005.patch, TEZ-3813.006.patch > > > Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a > BoundedByteArrayOutputStream that is not used (only the underlying byte[] is > used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3816) Ability to automatically speculate single-task vertices
Muhammad Samir Khan created TEZ-3816: Summary: Ability to automatically speculate single-task vertices Key: TEZ-3816 URL: https://issues.apache.org/jira/browse/TEZ-3816 Project: Apache Tez Issue Type: Improvement Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan When a single-task vertex is unlucky, it lands on a very slow node. Speculation doesn't currently apply when there are no other tasks to compare with. It would be good to either have a configurable timeout after which the tasks automatically speculate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3816) Ability to automatically speculate single-task vertices
[ https://issues.apache.org/jira/browse/TEZ-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3816: - Attachment: TEZ-3816.001.patch Added ability to LegacyTaskRuntimeEstimator to speculate based on timeout for single task vertices. > Ability to automatically speculate single-task vertices > --- > > Key: TEZ-3816 > URL: https://issues.apache.org/jira/browse/TEZ-3816 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3816.001.patch > > > When a single-task vertex is unlucky, it lands on a very slow node. > Speculation doesn't currently apply when there are no other tasks to compare > with. It would be good to either have a configurable timeout after which the > tasks automatically speculate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.002.patch
Some more optimizations related to enums, similar to TEZ-3732.
*JOL Dump:*
+Before:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7a 12 01 f8 (0010 00010010
0001 1000) (-134147462)
12 4
int FetchedInput.id 0
16 8
long FetchedInput.actualSize 0
24 8
long FetchedInput.compressedSize 0
32 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
36 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type
FetchedInput.type (object)
40 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
44 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.State
FetchedInput.state(object)
48 4
org.apache.hadoop.io.BoundedByteArrayOutputStream MemoryFetchedInput.byteStream
(object)
52 4
(loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
23264 [C
22448 java.lang.String
13232
org.apache.hadoop.io.BoundedByteArrayOutputStream
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
15656
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
9 264 (total)
{code}
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.003.patch
Fixed a NPE when getSize() is called after a free().
Removed actualSize and compressedSize from FetchedInput. Now child classes are
responsible for maintaining size (compressed or actual).
*JOL Dump:*
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7d 12 01 f8 (0101 00010010
0001 1000) (-134147459)
12 4
int FetchedInput.id 0
16 1
byte FetchedInput.state0
17 3
(alignment/padding gap)
20 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
24 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
28 4
int MemoryFetchedInput.size 0
32 4
byte[] MemoryFetchedInput.byteArray []
36 4
(loss due to the next object alignment)
Instance size: 40 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
14040
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
2 56 (total)
{code}
> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch,
> TEZ-3813.003.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is
> used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3809: - Attachment: TEZ-3809.002.patch Removed the unused imports and fixed the names for new methods. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch, TEZ-3809.002.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3283) Add a max cap to the heap computation done by Tez
[ https://issues.apache.org/jira/browse/TEZ-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3283: Assignee: Muhammad Samir Khan > Add a max cap to the heap computation done by Tez > - > > Key: TEZ-3283 > URL: https://issues.apache.org/jira/browse/TEZ-3283 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Muhammad Samir Khan > Labels: newbie > > Change the calculation to not use the fraction blindly. For large containers > - .e.g 10GB - this ends up leaving 2G of the container unused / outside of > the heap. > Instead we can add another parameter which indicates the max amount of memory > to leave outside the heap - defaulting to 1G -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3283) Add a max cap to the heap computation done by Tez
[ https://issues.apache.org/jira/browse/TEZ-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3283: - Attachment: tez-3283.001.patch Added a new parameter to specify the max java non heap memory, with a default of 1GB. > Add a max cap to the heap computation done by Tez > - > > Key: TEZ-3283 > URL: https://issues.apache.org/jira/browse/TEZ-3283 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Muhammad Samir Khan > Labels: newbie > Attachments: tez-3283.001.patch > > > Change the calculation to not use the fraction blindly. For large containers > - .e.g 10GB - this ends up leaving 2G of the container unused / outside of > the heap. > Instead we can add another parameter which indicates the max amount of memory > to leave outside the heap - defaulting to 1G -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3693) ControlledClock is not used
[ https://issues.apache.org/jira/browse/TEZ-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3693: Assignee: Muhammad Samir Khan > ControlledClock is not used > --- > > Key: TEZ-3693 > URL: https://issues.apache.org/jira/browse/TEZ-3693 > Project: Apache Tez > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Muhammad Samir Khan >Priority: Trivial > > The org.apache.tez.dag.app.ControlledClock class is not referenced in the > source. Oddly this is not a test class, like MockClock, as I would have > expected. If this is not part of the Tez API then it can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3693) ControlledClock is not used
[ https://issues.apache.org/jira/browse/TEZ-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3693: - Attachment: TEZ-3693.001.patch Removed ControlledClock > ControlledClock is not used > --- > > Key: TEZ-3693 > URL: https://issues.apache.org/jira/browse/TEZ-3693 > Project: Apache Tez > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Muhammad Samir Khan >Priority: Trivial > Attachments: TEZ-3693.001.patch > > > The org.apache.tez.dag.app.ControlledClock class is not referenced in the > source. Oddly this is not a test class, like MockClock, as I would have > expected. If this is not part of the Tez API then it can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3772) Allow slowstart for small jobs to be treated differently
Muhammad Samir Khan created TEZ-3772: Summary: Allow slowstart for small jobs to be treated differently Key: TEZ-3772 URL: https://issues.apache.org/jira/browse/TEZ-3772 Project: Apache Tez Issue Type: Improvement Reporter: Muhammad Samir Khan Assignee: Muhammad Samir Khan If there are a small number of reduces (configurable), then having a different threshold can benefit. Performance of jobs with a small number of reduce tasks can benefit significantly. Yes, the job could specify slowstart as 0.0 instead of the default, but that requires job owners to do something. It would be better if the defaults did something more optimal for both large and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3772: - Summary: Allow slowstart for small vertices to be treated differently (was: Allow slowstart for small jobs to be treated differently) > Allow slowstart for small vertices to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3772) Allow slowstart for small jobs to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3772: - Attachment: tez-3772.001.patch Added new configurations to distinguish between small/regular vertex and to set different min/max fractions for slowstart for small vertices. > Allow slowstart for small jobs to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063760#comment-16063760 ] Muhammad Samir Khan commented on TEZ-3772: -- For example, when there is a single reducer and some of the mappers fall on slow machines, the reducer can benefit by starting early. I ran an example with modified wordcount, where approx 10% of the tokenizer tasks will randomly sleep for 2 minutes and one summation task. I set the small job threshold to 1. With slowstart at 1.0, the total runtime was 360 seconds and with slowstart at 0.0, the total runtime was 270 seconds. For smaller jobs, we should be more aggressive with the slow start. I'll change the defaults to be 0.2 and 0.5 for min and max respectively. > Allow slowstart for small vertices to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3772: - Attachment: tez-3772.002.patch -Updated defaults for slowstart for small vertices. -Fixed javadoc links. > Allow slowstart for small vertices to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch, tez-3772.002.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3753) Improve error message in IFile for buffer length overflow
Muhammad Samir Khan created TEZ-3753: Summary: Improve error message in IFile for buffer length overflow Key: TEZ-3753 URL: https://issues.apache.org/jira/browse/TEZ-3753 Project: Apache Tez Issue Type: Bug Affects Versions: 0.7.1 Reporter: Muhammad Samir Khan When a record size is too big and the byte array doubling expansion crosses 2G the array size overflows and becomes negative. It would be good to fail the code paths saying record is too big so that error is easy to understand for users. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
