[
https://issues.apache.org/jira/browse/TEZ-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167739#comment-17167739
]
TezQA commented on TEZ-4211:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
54s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} tez-runtime-library in master has 1 extant findbugs
warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
17s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/504/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4211 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008734/TEZ-4211.wip.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux c3ba2501631f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 2d7c60849 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/artifact/out/branch-findbugs-tez-runtime-library-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/testReport/ |
| Max. process+thread count | 126 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Optimise MergeManager final merge
> ---------------------------------
>
> Key: TEZ-4211
> URL: https://issues.apache.org/jira/browse/TEZ-4211
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Priority: Major
> Attachments: TEZ-4211.wip.patch
>
>
> There are cases, when entire data is held in memory and no disk segments are
> present in MergeManager. Currently, mergemanager spills mem segments to disk
> before proceeding.
>
> [https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java#L1184]
>
> {code:java}
> if (numMemDiskSegments > 0 && ioSortFactor > onDiskMapOutputs.size()) {
> ...
> ..
> TezMerger.writeFile(rIter, writer, progressable,
> TezRuntimeConfiguration.TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS_DEFAULT);
> ...
> ..
> {code}
> This can be optimised not to spill to disk when only mem segments are present.
> Snippet from logs in one of the apps (Q78)
> {noformat}
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839646500 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=859378362 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=856145179 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849878734 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=842666749 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839533127 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=860448335 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=844468505 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=850099810 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849206236 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=840238680 and #onDiskOutputs=0,
> size=0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
