[jira] [Comment Edited] (TEZ-3814) Inserts into a bucketed table fail randomly with Hive on Tez
[ https://issues.apache.org/jira/browse/TEZ-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117544#comment-16117544 ] Anant Mittal edited comment on TEZ-3814 at 8/7/17 11:24 PM: [~gopalv] Thanks for the response. This is a functionality which is not working as expected. Please tell me if anything further can be done so that it satisfies being a bug. The JIRA pointed out was something new. I gave the suggestion a try but even that failed to fix the issue and we still see failures with the same error. I used the value 4000 for net.core.somaxconn. Please note that this is not a large load. The table used for insert has only 62 rows with a total size of 26948 bytes. was (Author: infinitymittal): [~gopalv] Thanks for the response. This is a functionality which is not working as expected. Please tell me if I anything further can be done so that it satisfies being a bug. The JIRA pointed out was something I had not yet tried then. I gave it a try but even that failed to fix the issue and we still see failures with the same error. I used the value 4000 for net.core.somaxconn. Please note that this is not a large load. The table used for insert has only 62 rows with a total size of 26948 bytes. > Inserts into a bucketed table fail randomly with Hive on Tez > > > Key: TEZ-3814 > URL: https://issues.apache.org/jira/browse/TEZ-3814 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Anant Mittal > Labels: Bucketing, Hive, Tez > > The MAP phase for Inserts into a bucketed table randomly fails with the error > "Vertex [Map 1] failed as task failed after vertex > succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:0". > The task fails because it fails for all attempts with " being > failed for too many output errors. failureFraction=0.2, > MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, > MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, > readErrorTimespan=0" > This happens more often if the table is ACID enabled and a delete operation > is performed before the inserts. > I have tried the following: > Changed tez.am.launch.cmd-opts, tez.task.launch.cmd-opts and > hive.tez.java.opts to use parallel GC. > tez.runtime.shuffle.max.allowed.failed.fetch.fraction = 0.95 > tez.runtime.shuffle.failed.check.since-last.completion=false > tez.runtime.shuffle.fetch.buffer.percent = 0.1 > tez.runtime.shuffle.memory.limit.percent = 0.25 > tez.runtime.shuffle.ssl.enable=false > Deleted ".../usercache//filecache" and ".../usercache//appcache" > I am using HDP 2.6 dsitribution. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (TEZ-3814) Inserts into a bucketed table fail randomly with Hive on Tez
[ https://issues.apache.org/jira/browse/TEZ-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Mittal reopened TEZ-3814: --- > Inserts into a bucketed table fail randomly with Hive on Tez > > > Key: TEZ-3814 > URL: https://issues.apache.org/jira/browse/TEZ-3814 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Anant Mittal > Labels: Bucketing, Hive, Tez > > The MAP phase for Inserts into a bucketed table randomly fails with the error > "Vertex [Map 1] failed as task failed after vertex > succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:0". > The task fails because it fails for all attempts with " being > failed for too many output errors. failureFraction=0.2, > MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, > MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, > readErrorTimespan=0" > This happens more often if the table is ACID enabled and a delete operation > is performed before the inserts. > I have tried the following: > Changed tez.am.launch.cmd-opts, tez.task.launch.cmd-opts and > hive.tez.java.opts to use parallel GC. > tez.runtime.shuffle.max.allowed.failed.fetch.fraction = 0.95 > tez.runtime.shuffle.failed.check.since-last.completion=false > tez.runtime.shuffle.fetch.buffer.percent = 0.1 > tez.runtime.shuffle.memory.limit.percent = 0.25 > tez.runtime.shuffle.ssl.enable=false > Deleted ".../usercache//filecache" and ".../usercache//appcache" > I am using HDP 2.6 dsitribution. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3814) Inserts into a bucketed table fail randomly with Hive on Tez
[ https://issues.apache.org/jira/browse/TEZ-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117544#comment-16117544 ] Anant Mittal commented on TEZ-3814: --- [~gopalv] Thanks for the response. This is a functionality which is not working as expected. Please tell me if I anything further can be done so that it satisfies being a bug. The JIRA pointed out was something I had not yet tried then. I gave it a try but even that failed to fix the issue and we still see failures with the same error. I used the value 4000 for net.core.somaxconn. Please note that this is not a large load. The table used for insert has only 62 rows with a total size of 26948 bytes. > Inserts into a bucketed table fail randomly with Hive on Tez > > > Key: TEZ-3814 > URL: https://issues.apache.org/jira/browse/TEZ-3814 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Anant Mittal > Labels: Bucketing, Hive, Tez > > The MAP phase for Inserts into a bucketed table randomly fails with the error > "Vertex [Map 1] failed as task failed after vertex > succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:0". > The task fails because it fails for all attempts with " being > failed for too many output errors. failureFraction=0.2, > MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, > MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, > readErrorTimespan=0" > This happens more often if the table is ACID enabled and a delete operation > is performed before the inserts. > I have tried the following: > Changed tez.am.launch.cmd-opts, tez.task.launch.cmd-opts and > hive.tez.java.opts to use parallel GC. > tez.runtime.shuffle.max.allowed.failed.fetch.fraction = 0.95 > tez.runtime.shuffle.failed.check.since-last.completion=false > tez.runtime.shuffle.fetch.buffer.percent = 0.1 > tez.runtime.shuffle.memory.limit.percent = 0.25 > tez.runtime.shuffle.ssl.enable=false > Deleted ".../usercache//filecache" and ".../usercache//appcache" > I am using HDP 2.6 dsitribution. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117457#comment-16117457
]
TezQA commented on TEZ-3813:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880711/TEZ-3813.003.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2603//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2603//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2603//console
This message is automatically generated.
> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch,
> TEZ-3813.003.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is
> used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3813 PreCommit Build #2603
Jira: https://issues.apache.org/jira/browse/TEZ-3813
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2603/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.48 KB...]
[INFO]
[INFO] Total time: 55:38 min
[INFO] Finished at: 2017-08-07T22:34:14Z
[INFO] Final Memory: 89M/1389M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880711/TEZ-3813.003.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2603//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2603//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2603//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
8fe01799cae2e517338c6d547aec916db4bded6d logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] Compressed 3.51 MB of artifacts by 32.9% relative to #2601
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117354#comment-16117354
]
TezQA commented on TEZ-3813:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880696/TEZ-3813.002.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2602//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2602//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2602//console
This message is automatically generated.
> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch,
> TEZ-3813.003.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is
> used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3813 PreCommit Build #2602
Jira: https://issues.apache.org/jira/browse/TEZ-3813
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2602/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.27 KB...]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:00 h
[INFO] Finished at: 2017-08-07T21:36:34Z
[INFO] Final Memory: 81M/1411M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880696/TEZ-3813.002.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2602//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2602//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2602//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
13491b4683aab50fd83a9a58b8123590a7e8a6b5 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.003.patch
Fixed a NPE when getSize() is called after a free().
Removed actualSize and compressedSize from FetchedInput. Now child classes are
responsible for maintaining size (compressed or actual).
*JOL Dump:*
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7d 12 01 f8 (0101 00010010
0001 1000) (-134147459)
12 4
int FetchedInput.id 0
16 1
byte FetchedInput.state0
17 3
(alignment/padding gap)
20 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
24 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
28 4
int MemoryFetchedInput.size 0
32 4
byte[] MemoryFetchedInput.byteArray []
36 4
(loss due to the next object alignment)
Instance size: 40 bytes
Space losses: 3 bytes internal + 4 bytes external = 7 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
14040
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
2 56 (total)
{code}
> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch, TEZ-3813.002.patch,
> TEZ-3813.003.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is
> used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs
[
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.002.patch
Some more optimizations related to enums, similar to TEZ-3732.
*JOL Dump:*
+Before:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00 (0001
) (1)
4 4
(object header) 00 00 00 00 (
) (0)
8 4
(object header) 7a 12 01 f8 (0010 00010010
0001 1000) (-134147462)
12 4
int FetchedInput.id 0
16 8
long FetchedInput.actualSize 0
24 8
long FetchedInput.compressedSize 0
32 4
org.apache.tez.runtime.library.common.InputAttemptIdentifier
FetchedInput.inputAttemptIdentifier null
36 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type
FetchedInput.type (object)
40 4
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback
FetchedInput.callback null
44 4
org.apache.tez.runtime.library.common.shuffle.FetchedInput.State
FetchedInput.state(object)
48 4
org.apache.hadoop.io.BoundedByteArrayOutputStream MemoryFetchedInput.byteStream
(object)
52 4
(loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}
Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd
footprint:
COUNT AVG SUM DESCRIPTION
11616 [B
23264 [C
22448 java.lang.String
13232
org.apache.hadoop.io.BoundedByteArrayOutputStream
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
12424
org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
15656
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
9 264 (total)
{code}
+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
Instantiated the sample instance via public
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object
internals:
OFFSET SIZE
TYPE DESCRIPTION VALUE
0 4
(object header) 01 00 00 00
[jira] [Commented] (TEZ-3431) Add unit tests for container release
[ https://issues.apache.org/jira/browse/TEZ-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117082#comment-16117082 ] Taklon Stephen Wu commented on TEZ-3431: I have written a patch this item, can I be the assignee? > Add unit tests for container release > > > Key: TEZ-3431 > URL: https://issues.apache.org/jira/browse/TEZ-3431 > Project: Apache Tez > Issue Type: Bug >Reporter: Sushmitha Sreenivasan > Labels: newbie > > * Add unit tests to verify that scheduler release container after expiry > time(HeldContainer.containerExpiryTime). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117078#comment-16117078
]
TezQA commented on TEZ-3809:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880667/TEZ-3809.002.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2601//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2601//console
This message is automatically generated.
> The buffer size allocated for InMemoryMapOutput can be optimized
>
>
> Key: TEZ-3809
> URL: https://issues.apache.org/jira/browse/TEZ-3809
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3809.001.patch, TEZ-3809.002.patch
>
>
> Related jiras: TEZ-3752 and TEZ-3732.
> -When shuffling input to memory, the decompressed length is used to create
> the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4
> bytes less (the IFile header). These 4 bytes can optimized and, in an extreme
> case of 10,000,000 fetches, can save ~38 MB (TEZ-3732).
> -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers
> to allocate the new InMemoryMapOutput. However, each input has two
> EOF_MARKERs while only two are needed at the end.
> -InMemoryWriter wraps the output BoundedByteArrayOutputStream in
> IFileOutputStream which will write checksum at close. This creates an
> inconsistency between the primary input buffers which don't have checksum and
> the merged buffers which do. IFileOutputStream wrap can be removed to save 4
> bytes per merged buffers.
> -InMemoryWriter does not account for two EOF_MARKERs written at close() in
> its accounting so that the getRawLength() method is off by two bytes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Success: TEZ-3809 PreCommit Build #2601
Jira: https://issues.apache.org/jira/browse/TEZ-3809
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2601/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.41 KB...]
[INFO] Tez SUCCESS [ 0.018 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 54:40 min
[INFO] Finished at: 2017-08-07T19:07:06Z
[INFO] Final Memory: 93M/1338M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12880667/TEZ-3809.002.patch
against master revision 8dcf8a1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2601//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2601//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
2c4486fb33f7b93b0277293295b55d07c28ffe68 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3809
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized
[ https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3809: - Attachment: TEZ-3809.002.patch Removed the unused imports and fixed the names for new methods. > The buffer size allocated for InMemoryMapOutput can be optimized > > > Key: TEZ-3809 > URL: https://issues.apache.org/jira/browse/TEZ-3809 > Project: Apache Tez > Issue Type: Bug >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: TEZ-3809.001.patch, TEZ-3809.002.patch > > > Related jiras: TEZ-3752 and TEZ-3732. > -When shuffling input to memory, the decompressed length is used to create > the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 > bytes less (the IFile header). These 4 bytes can optimized and, in an extreme > case of 10,000,000 fetches, can save ~38 MB (TEZ-3732). > -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers > to allocate the new InMemoryMapOutput. However, each input has two > EOF_MARKERs while only two are needed at the end. > -InMemoryWriter wraps the output BoundedByteArrayOutputStream in > IFileOutputStream which will write checksum at close. This creates an > inconsistency between the primary input buffers which don't have checksum and > the merged buffers which do. IFileOutputStream wrap can be removed to save 4 > bytes per merged buffers. > -InMemoryWriter does not account for two EOF_MARKERs written at close() in > its accounting so that the getRawLength() method is off by two bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
