[jira] [Commented] (TEZ-3605) Detect and prune empty partitions for the Ordered case
[
https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064225#comment-16064225
]
TezQA commented on TEZ-3605:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874554/TEZ-3605.012.patch
against master revision 5b0f5a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2546//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2546//console
This message is automatically generated.
> Detect and prune empty partitions for the Ordered case
> --
>
> Key: TEZ-3605
> URL: https://issues.apache.org/jira/browse/TEZ-3605
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch,
> TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch,
> TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch,
> TEZ-3605.009.patch, TEZ-3605.010.patch, TEZ-3605.011.patch, TEZ-3605.012.patch
>
>
> Analogous to the Unordered case we should not have empty partition
> entries/segments in the Ordered/DefaultSorter case. This will save writing
> unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
> job, this change would allow not fetching empty partitions and then throwing
> them away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Success: TEZ-3605 PreCommit Build #2546
Jira: https://issues.apache.org/jira/browse/TEZ-3605
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2546/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 339.81 KB...]
[INFO] Tez SUCCESS [ 0.028 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:01 h
[INFO] Finished at: 2017-06-27T04:08:30Z
[INFO] Final Memory: 81M/1315M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874554/TEZ-3605.012.patch
against master revision 5b0f5a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2546//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2546//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
24c5daa1aea76e681e0768377c42360aa255eb12 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3605
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3773) Tez build failed
[
https://issues.apache.org/jira/browse/TEZ-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064161#comment-16064161
]
KaiXu commented on TEZ-3773:
Thanks [~aplusplus], but after applying the patch also has the similar issue.
> Tez build failed
>
>
> Key: TEZ-3773
> URL: https://issues.apache.org/jira/browse/TEZ-3773
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.8.4
>Reporter: KaiXu
>
> tez build failed on both Ubuntu14.04 and CentOS, with root or other users:
> > tez-ui@0.2.0 build:mvn
> > /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp
> > TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build -prod
> /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/lib/mv.js:3
> const Funnel = require('broccoli-funnel');
> ^
> Use of const in strict mode.
> /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/lib/mv.js:3
> const Funnel = require('broccoli-funnel');
> ^
> SyntaxError: Use of const in strict mode.
> at exports.runInThisContext (vm.js:73:16)
> at Module._compile (module.js:443:25)
> at Object.Module._extensions..js (module.js:478:10)
> at Module.load (module.js:355:32)
> at Function.Module._load (module.js:310:12)
> at Module.require (module.js:365:17)
> at require (module.js:384:17)
> at Object.
> (/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/index.js:2:7)
> at Module._compile (module.js:460:26)
> at Object.Module._extensions..js (module.js:478:10)
> npm ERR! Linux 4.4.0-75-generic
> npm ERR! argv
> "/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node/node"
>
> "/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node/npm/bin/npm-cli"
> "run" "build:mvn"
> npm ERR! node v0.12.2
> npm ERR! npm v2.15.3
> npm ERR! code ELIFECYCLE
> npm ERR! tez-ui@0.2.0 build:mvn: `TMPDIR=tmp node/node
> ./node_modules/ember-cli/bin/ember build -prod`
> npm ERR! Exit status 1
> npm ERR!
> npm ERR! Failed at the tez-ui@0.2.0 build:mvn script 'TMPDIR=tmp node/node
> ./node_modules/ember-cli/bin/ember build -prod'.
> npm ERR! This is most likely a problem with the tez-ui package,
> npm ERR! not with npm itself.
> npm ERR! Tell the author that this fails on your system:
> npm ERR! TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build
> -prod
> npm ERR! You can get information on how to open an issue for this project
> with:
> npm ERR! npm bugs tez-ui
> npm ERR! Or if that isn't available, you can get their info via:
> npm ERR!
> npm ERR! npm owner ls tez-ui
> npm ERR! There is likely additional logging output above.
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec
> (ember build) on project tez-ui2: Command execution failed. Process exited
> with an error: 1 (Exit value: 1) -> [Help 1]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3769 PreCommit Build #2545
Jira: https://issues.apache.org/jira/browse/TEZ-3769
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2545/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 340.08 KB...]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 57:37 min
[INFO] Finished at: 2017-06-27T00:36:14Z
[INFO] Final Memory: 84M/1430M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874585/TEZ-3769.2.patch
against master revision 5b0f5a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 23 javac
compiler warnings (more than the master's current 21 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2545//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2545//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2545//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
759506046aa01413c748e14e226bc5eb0483eac9 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3769) Unordered: Fix wrong stats being sent out in the last event, when final merge is disabled
[
https://issues.apache.org/jira/browse/TEZ-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064047#comment-16064047
]
TezQA commented on TEZ-3769:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874585/TEZ-3769.2.patch
against master revision 5b0f5a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 23 javac
compiler warnings (more than the master's current 21 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2545//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2545//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2545//console
This message is automatically generated.
> Unordered: Fix wrong stats being sent out in the last event, when final merge
> is disabled
> -
>
> Key: TEZ-3769
> URL: https://issues.apache.org/jira/browse/TEZ-3769
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: TEZ-3769.1.patch, TEZ-3769.2.patch
>
>
> When final merge is disabled (without pipelining), wrong stats was sent out
> in the last event.
> It was based on {{numRecordsPerPartition}} which contains the overall
> partition data. It should be ideally be based on the spill result and its
> buffers.
> Also, {{finalSpill}} was unncessarily sending events when no data was present
> (i.e, when currentBuffer didn't have any data). This can be optimized to
> reduce the number of events being sent across.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3769) Unordered: Fix wrong stats being sent out in the last event, when final merge is disabled
[
https://issues.apache.org/jira/browse/TEZ-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3769:
--
Attachment: TEZ-3769.2.patch
Rebasing the patch (TEZ-3762 got committed) and added few more tests.
> Unordered: Fix wrong stats being sent out in the last event, when final merge
> is disabled
> -
>
> Key: TEZ-3769
> URL: https://issues.apache.org/jira/browse/TEZ-3769
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: TEZ-3769.1.patch, TEZ-3769.2.patch
>
>
> When final merge is disabled (without pipelining), wrong stats was sent out
> in the last event.
> It was based on {{numRecordsPerPartition}} which contains the overall
> partition data. It should be ideally be based on the spill result and its
> buffers.
> Also, {{finalSpill}} was unncessarily sending events when no data was present
> (i.e, when currentBuffer didn't have any data). This can be optimized to
> reduce the number of events being sent across.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3762) When final merge is disabled in unordered case, it should create index file instead of relying on cache
[
https://issues.apache.org/jira/browse/TEZ-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063907#comment-16063907
]
Zhiyuan Yang commented on TEZ-3762:
---
Thanks [~rajesh.balamohan]! Patch was committed to master branch.
{noformat}
commit 5b0f5a0c415869c60da45715e2a67adf0405b778
Author: Zhiyuan Yang
Date: Mon Jun 26 15:21:46 2017 -0700
TEZ-3762. When final merge is disabled in unordered case, it should create
index file instead of relying on cache (zhiyuany)
{noformat}
> When final merge is disabled in unordered case, it should create index file
> instead of relying on cache
> ---
>
> Key: TEZ-3762
> URL: https://issues.apache.org/jira/browse/TEZ-3762
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Rajesh Balamohan
>Assignee: Zhiyuan Yang
> Fix For: 0.9.0
>
> Attachments: TEZ-3762.1.patch, TEZ-3762.2.patch, TEZ-3762.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3762) When final merge is disabled in unordered case, it should create index file instead of relying on cache
[ https://issues.apache.org/jira/browse/TEZ-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3762: -- Fix Version/s: 0.9.0 > When final merge is disabled in unordered case, it should create index file > instead of relying on cache > --- > > Key: TEZ-3762 > URL: https://issues.apache.org/jira/browse/TEZ-3762 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Rajesh Balamohan >Assignee: Zhiyuan Yang > Fix For: 0.9.0 > > Attachments: TEZ-3762.1.patch, TEZ-3762.2.patch, TEZ-3762.3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[
https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063890#comment-16063890
]
TezQA commented on TEZ-3772:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874553/tez-3772.002.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2544//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2544//console
This message is automatically generated.
> Allow slowstart for small vertices to be treated differently
>
>
> Key: TEZ-3772
> URL: https://issues.apache.org/jira/browse/TEZ-3772
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: tez-3772.001.patch, tez-3772.002.patch
>
>
> If there are a small number of reduces (configurable), then having a
> different threshold can benefit. Performance of jobs with a small number of
> reduce tasks can benefit significantly. Yes, the job could specify slowstart
> as 0.0 instead of the default, but that requires job owners to do something.
> It would be better if the defaults did something more optimal for both large
> and small jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Success: TEZ-3772 PreCommit Build #2544
Jira: https://issues.apache.org/jira/browse/TEZ-3772
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2544/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 339.89 KB...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 54:58 min
[INFO] Finished at: 2017-06-26T22:14:26Z
[INFO] Final Memory: 94M/1475M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874553/tez-3772.002.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2544//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2544//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4078986c55967a9213078a8a3e4982882f371fa0 logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.50 MB of artifacts by 11.6% relative to #2543
[description-setter] Description set: TEZ-3772
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3762) When final merge is disabled in unordered case, it should create index file instead of relying on cache
[ https://issues.apache.org/jira/browse/TEZ-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063885#comment-16063885 ] Rajesh Balamohan commented on TEZ-3762: --- lgtm. +1. Thanks [~aplusplus] > When final merge is disabled in unordered case, it should create index file > instead of relying on cache > --- > > Key: TEZ-3762 > URL: https://issues.apache.org/jira/browse/TEZ-3762 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Rajesh Balamohan >Assignee: Zhiyuan Yang > Attachments: TEZ-3762.1.patch, TEZ-3762.2.patch, TEZ-3762.3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3776) ShuffleHandler can theoretically NPE if all partitions in the ranged fetch are empty
Kuhu Shukla created TEZ-3776:
Summary: ShuffleHandler can theoretically NPE if all partitions in
the ranged fetch are empty
Key: TEZ-3776
URL: https://issues.apache.org/jira/browse/TEZ-3776
Project: Apache Tez
Issue Type: Bug
Reporter: Kuhu Shukla
{code}
if (index.getPartLength() != 0) {
if (firstIndex == null) {
firstIndex = index;
}
lastIndex = index;
}
ShuffleHeader header = new ShuffleHeader(mapId, index.getPartLength(),
index.getRawLength(), reduce);
DataOutputBuffer dob = new DataOutputBuffer();
header.write(dob);
ch.write(wrappedBuffer(dob.getData(), 0, dob.getLength()));
}
final long rangeOffset = firstIndex.getStartOffset(); <
{code}
In cases where tez.runtime.empty.partitions.info-via-events.enabled is false
and is respected (say post TEZ-3287, if not handled correctly at the output
side, the shuffle handler assumes that if it has got to this point there is at
least one non empty partition. The ShuffleHandler should protect itself from
this NPE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3605) Detect and prune empty partitions for the Ordered case
[ https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3605: - Attachment: TEZ-3605.012.patch Fixing the numspill=0 case for DefaultSorter to honor sendPartitionDetails flag. > Detect and prune empty partitions for the Ordered case > -- > > Key: TEZ-3605 > URL: https://issues.apache.org/jira/browse/TEZ-3605 > Project: Apache Tez > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch, > TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch, > TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch, > TEZ-3605.009.patch, TEZ-3605.010.patch, TEZ-3605.011.patch, TEZ-3605.012.patch > > > Analogous to the Unordered case we should not have empty partition > entries/segments in the Ordered/DefaultSorter case. This will save writing > unnecessary data. > Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced > job, this change would allow not fetching empty partitions and then throwing > them away. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3772: - Attachment: tez-3772.002.patch -Updated defaults for slowstart for small vertices. -Fixed javadoc links. > Allow slowstart for small vertices to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch, tez-3772.002.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3772) Allow slowstart for small vertices to be treated differently
[ https://issues.apache.org/jira/browse/TEZ-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063760#comment-16063760 ] Muhammad Samir Khan commented on TEZ-3772: -- For example, when there is a single reducer and some of the mappers fall on slow machines, the reducer can benefit by starting early. I ran an example with modified wordcount, where approx 10% of the tokenizer tasks will randomly sleep for 2 minutes and one summation task. I set the small job threshold to 1. With slowstart at 1.0, the total runtime was 360 seconds and with slowstart at 0.0, the total runtime was 270 seconds. For smaller jobs, we should be more aggressive with the slow start. I'll change the defaults to be 0.2 and 0.5 for min and max respectively. > Allow slowstart for small vertices to be treated differently > > > Key: TEZ-3772 > URL: https://issues.apache.org/jira/browse/TEZ-3772 > Project: Apache Tez > Issue Type: Improvement >Reporter: Muhammad Samir Khan >Assignee: Muhammad Samir Khan > Attachments: tez-3772.001.patch > > > If there are a small number of reduces (configurable), then having a > different threshold can benefit. Performance of jobs with a small number of > reduce tasks can benefit significantly. Yes, the job could specify slowstart > as 0.0 instead of the default, but that requires job owners to do something. > It would be better if the defaults did something more optimal for both large > and small jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Success: TEZ-3762 PreCommit Build #2543
Jira: https://issues.apache.org/jira/browse/TEZ-3762
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2543/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 339.58 KB...]
[INFO] Tez SUCCESS [ 0.018 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 58:53 min
[INFO] Finished at: 2017-06-26T19:40:49Z
[INFO] Final Memory: 80M/1422M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874532/TEZ-3762.3.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2543//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2543//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
9f07a3b12b7453503a592bbf5b9691568da70e0b logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3762
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3762) When final merge is disabled in unordered case, it should create index file instead of relying on cache
[
https://issues.apache.org/jira/browse/TEZ-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063651#comment-16063651
]
TezQA commented on TEZ-3762:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874532/TEZ-3762.3.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2543//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2543//console
This message is automatically generated.
> When final merge is disabled in unordered case, it should create index file
> instead of relying on cache
> ---
>
> Key: TEZ-3762
> URL: https://issues.apache.org/jira/browse/TEZ-3762
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Rajesh Balamohan
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3762.1.patch, TEZ-3762.2.patch, TEZ-3762.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (TEZ-3605) Detect and prune empty partitions for the Ordered case
[
https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063606#comment-16063606
]
Kuhu Shukla edited comment on TEZ-3605 at 6/26/17 7:19 PM:
---
bq. and invokes a merger on an empty list (not sure how this is handled)
Empty List is handled fine
{quote}
if (segments.size() == 0) {
LOG.info("Nothing to merge. Returning an empty iterator");
return new EmptyIterator();
}
{quote}
It is when the segment size is zero when it gets into trouble due to a stream
with no bytes to read.
The latest patch fixes this issue by segments for only non-empty partitions.
was (Author: kshukla):
bq. and invokes a merger on an empty list (not sure how this is handled)
Empty List is handled fine
{quote}
if (segments.size() == 0) {
LOG.info("Nothing to merge. Returning an empty iterator");
return new EmptyIterator();
}
{quote}
It is when the segment size is zero when it gets into trouble due to a stream
with no bytes to read.
> Detect and prune empty partitions for the Ordered case
> --
>
> Key: TEZ-3605
> URL: https://issues.apache.org/jira/browse/TEZ-3605
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch,
> TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch,
> TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch,
> TEZ-3605.009.patch, TEZ-3605.010.patch, TEZ-3605.011.patch
>
>
> Analogous to the Unordered case we should not have empty partition
> entries/segments in the Ordered/DefaultSorter case. This will save writing
> unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
> job, this change would allow not fetching empty partitions and then throwing
> them away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3605) Detect and prune empty partitions for the Ordered case
[
https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063606#comment-16063606
]
Kuhu Shukla commented on TEZ-3605:
--
bq. and invokes a merger on an empty list (not sure how this is handled)
Empty List is handled fine
{quote}
if (segments.size() == 0) {
LOG.info("Nothing to merge. Returning an empty iterator");
return new EmptyIterator();
}
{quote}
It is when the segment size is zero when it gets into trouble due to a stream
with no bytes to read.
> Detect and prune empty partitions for the Ordered case
> --
>
> Key: TEZ-3605
> URL: https://issues.apache.org/jira/browse/TEZ-3605
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch,
> TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch,
> TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch,
> TEZ-3605.009.patch, TEZ-3605.010.patch, TEZ-3605.011.patch
>
>
> Analogous to the Unordered case we should not have empty partition
> entries/segments in the Ordered/DefaultSorter case. This will save writing
> unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
> job, this change would allow not fetching empty partitions and then throwing
> them away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3775 PreCommit Build #2542
Jira: https://issues.apache.org/jira/browse/TEZ-3775
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2542/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 345.67 KB...]
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-ui
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874520/TEZ-3775.1.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2542//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2542//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
5d8ac989e4db3dd6cef3dc42cb48dde10afa0a21 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3605) Detect and prune empty partitions for the Ordered case
[ https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3605: - Attachment: TEZ-3605.011.patch Uploading a new patch that has changes to the merge/flush logic. SpillRecord contains entries for all partitions with length=0 in the empty case. The file.out write is done only for non-empty partitions. The defaultSorter's special case for numSpills=0 is also changed as part of this fix. Added some tests to cover some aspects of the change. > Detect and prune empty partitions for the Ordered case > -- > > Key: TEZ-3605 > URL: https://issues.apache.org/jira/browse/TEZ-3605 > Project: Apache Tez > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch, > TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch, > TEZ-3605.006.patch, TEZ-3605.007.patch, TEZ-3605.008.patch, > TEZ-3605.009.patch, TEZ-3605.010.patch, TEZ-3605.011.patch > > > Analogous to the Unordered case we should not have empty partition > entries/segments in the Ordered/DefaultSorter case. This will save writing > unnecessary data. > Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced > job, this change would allow not fetching empty partitions and then throwing > them away. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3775) Tez UI: Show DAG context in document title
[
https://issues.apache.org/jira/browse/TEZ-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063582#comment-16063582
]
TezQA commented on TEZ-3775:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12874520/TEZ-3775.1.patch
against master revision a7f93ae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2542//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2542//console
This message is automatically generated.
> Tez UI: Show DAG context in document title
> ---
>
> Key: TEZ-3775
> URL: https://issues.apache.org/jira/browse/TEZ-3775
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3775.1.patch
>
>
> In Tez UI 0.7, DAG (vertex, app, task, attempt) context was shown in the
> document title. This was lost in the 0.9 UI migration. This jira attempts to
> bring that feature back. This feature is essential when supporting large
> clusters where a dev or support person may have dozens of dags open at the
> same time. Having context in the document title (the tab title), will allow
> us to quickly navigate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3773) Tez build failed
[
https://issues.apache.org/jira/browse/TEZ-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063572#comment-16063572
]
Zhiyuan Yang commented on TEZ-3773:
---
This was fixed in 0.8.6. Please refer to TEZ-3658 patch.
> Tez build failed
>
>
> Key: TEZ-3773
> URL: https://issues.apache.org/jira/browse/TEZ-3773
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.8.4
>Reporter: KaiXu
>
> tez build failed on both Ubuntu14.04 and CentOS, with root or other users:
> > tez-ui@0.2.0 build:mvn
> > /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp
> > TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build -prod
> /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/lib/mv.js:3
> const Funnel = require('broccoli-funnel');
> ^
> Use of const in strict mode.
> /home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/lib/mv.js:3
> const Funnel = require('broccoli-funnel');
> ^
> SyntaxError: Use of const in strict mode.
> at exports.runInThisContext (vm.js:73:16)
> at Module._compile (module.js:443:25)
> at Object.Module._extensions..js (module.js:478:10)
> at Module.load (module.js:355:32)
> at Function.Module._load (module.js:310:12)
> at Module.require (module.js:365:17)
> at require (module.js:384:17)
> at Object.
> (/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node_modules/ember-cli-moment-shim/node_modules/broccoli-stew/index.js:2:7)
> at Module._compile (module.js:460:26)
> at Object.Module._extensions..js (module.js:478:10)
> npm ERR! Linux 4.4.0-75-generic
> npm ERR! argv
> "/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node/node"
>
> "/home/xuk1/sourceCode/apache-tez-0.8.4-src/tez-ui2/src/main/webapp/node/npm/bin/npm-cli"
> "run" "build:mvn"
> npm ERR! node v0.12.2
> npm ERR! npm v2.15.3
> npm ERR! code ELIFECYCLE
> npm ERR! tez-ui@0.2.0 build:mvn: `TMPDIR=tmp node/node
> ./node_modules/ember-cli/bin/ember build -prod`
> npm ERR! Exit status 1
> npm ERR!
> npm ERR! Failed at the tez-ui@0.2.0 build:mvn script 'TMPDIR=tmp node/node
> ./node_modules/ember-cli/bin/ember build -prod'.
> npm ERR! This is most likely a problem with the tez-ui package,
> npm ERR! not with npm itself.
> npm ERR! Tell the author that this fails on your system:
> npm ERR! TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build
> -prod
> npm ERR! You can get information on how to open an issue for this project
> with:
> npm ERR! npm bugs tez-ui
> npm ERR! Or if that isn't available, you can get their info via:
> npm ERR!
> npm ERR! npm owner ls tez-ui
> npm ERR! There is likely additional logging output above.
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec
> (ember build) on project tez-ui2: Command execution failed. Process exited
> with an error: 1 (Exit value: 1) -> [Help 1]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3762) When final merge is disabled in unordered case, it should create index file instead of relying on cache
[ https://issues.apache.org/jira/browse/TEZ-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3762: -- Attachment: TEZ-3762.3.patch > When final merge is disabled in unordered case, it should create index file > instead of relying on cache > --- > > Key: TEZ-3762 > URL: https://issues.apache.org/jira/browse/TEZ-3762 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Rajesh Balamohan >Assignee: Zhiyuan Yang > Attachments: TEZ-3762.1.patch, TEZ-3762.2.patch, TEZ-3762.3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3775) Tez UI: Show DAG context in document title
[ https://issues.apache.org/jira/browse/TEZ-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3775: - Attachment: TEZ-3775.1.patch [~rohini], I think this is the feature you were requesting. Can you have a look from a user's perspective? [~Sreenath], can you have a review and suggest improvements? > Tez UI: Show DAG context in document title > --- > > Key: TEZ-3775 > URL: https://issues.apache.org/jira/browse/TEZ-3775 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3775.1.patch > > > In Tez UI 0.7, DAG (vertex, app, task, attempt) context was shown in the > document title. This was lost in the 0.9 UI migration. This jira attempts to > bring that feature back. This feature is essential when supporting large > clusters where a dev or support person may have dozens of dags open at the > same time. Having context in the document title (the tab title), will allow > us to quickly navigate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3775) Tez UI: Show DAG context in document title
Jonathan Eagles created TEZ-3775: Summary: Tez UI: Show DAG context in document title Key: TEZ-3775 URL: https://issues.apache.org/jira/browse/TEZ-3775 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Jonathan Eagles Assignee: Jonathan Eagles In Tez UI 0.7, DAG (vertex, app, task, attempt) context was shown in the document title. This was lost in the 0.9 UI migration. This jira attempts to bring that feature back. This feature is essential when supporting large clusters where a dev or support person may have dozens of dags open at the same time. Having context in the document title (the tab title), will allow us to quickly navigate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3274) Vertex with MRInput and broadcast input does not respect slow start
[ https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063155#comment-16063155 ] Jonathan Eagles commented on TEZ-3274: -- In addition, I see the min-src-fraction for shuffle-vertex-manager is added to the DeprecatedKeys map that translates MR configs to tez configs. This will be a useful addition in this case as well. > Vertex with MRInput and broadcast input does not respect slow start > --- > > Key: TEZ-3274 > URL: https://issues.apache.org/jira/browse/TEZ-3274 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Eric Badger > Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch, > TEZ-3274.003.patch, TEZ-3274.004.patch > > > Vertices with shuffle input and MRInput choose RootInputVertexManager (and > not ShuffleVertexManager) and start containers and tasks immediately. In this > scenario, resources can be wasted since they do not respect > tez.shuffle-vertex-manager.min-src-fraction > tez.shuffle-vertex-manager.max-src-fraction. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3774) hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0)
[
https://issues.apache.org/jira/browse/TEZ-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063023#comment-16063023
]
xujie commented on TEZ-3774:
how move to hive jira?
> hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> ---
>
> Key: TEZ-3774
> URL: https://issues.apache.org/jira/browse/TEZ-3774
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.5
>Reporter: xujie
>
> Status: Running (Executing on YARN cluster with App id
> application_1494499520849_1063145)
>
> VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED
> KILLED
>
> Map 1 . KILLED 15 1401 0
> 1
> Map 10 . SUCCEEDED 15 1500 0
> 0
> Map 12 . SUCCEEDED 15 1500 0
> 0
> Map 13 . SUCCEEDED 1 100 0
> 0
> Map 14 . SUCCEEDED 1 100 0
> 0
> Map 15 . SUCCEEDED 1 100 0
> 0
> Map 16 . SUCCEEDED 1 100 0
> 0
> Map 17 . SUCCEEDED 1 100 0
> 0
> Map 4 .. SUCCEEDED 15 1500 0
> 0
> Map 5 .. SUCCEEDED 1 100 0
> 0
> Map 6 .. SUCCEEDED 1 100 0
> 0
> Map 7 .. SUCCEEDED 1 100 0
> 0
> Map 8 .. SUCCEEDED 1 100 0
> 0
> Map 9 .. SUCCEEDED 1 100 0
> 0
> Reducer 11FAILED315 00 315 774
> 314
> Reducer 2 KILLED315 00 315 0
> 315
>
> VERTICES: 14/16 [==>>] 9%ELAPSED TIME: 130.89 s
>
>
> Status: Failed
> Vertex re-running, vertexName=Map 12,
> vertexId=vertex_1494499520849_1063145_1_07
> Vertex re-running, vertexName=Map 1,
> vertexId=vertex_1494499520849_1063145_1_11
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 12,
> vertexId=vertex_1494499520849_1063145_1_07
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 5,
> vertexId=vertex_1494499520849_1063145_1_02
> Vertex failed, vertexName=Reducer 11,
> vertexId=vertex_1494499520849_1063145_1_14, diagnostics=[Task failed,
> taskId=task_1494499520849_1063145_1_14_13, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1494499520849_1063145_1_14_13_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"0002991000585141671"},"value":{"_col0":"","_col1":""}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
>
[jira] [Commented] (TEZ-3774) hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0)
[
https://issues.apache.org/jira/browse/TEZ-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062997#comment-16062997
]
Rajesh Balamohan commented on TEZ-3774:
---
This is more of a hive problem than Tez issue. Can you plz move this to Hive
jira?. You may want to mention the versions of hive/tez in the jira.
> hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> ---
>
> Key: TEZ-3774
> URL: https://issues.apache.org/jira/browse/TEZ-3774
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.5
>Reporter: xujie
>
> Status: Running (Executing on YARN cluster with App id
> application_1494499520849_1063145)
>
> VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED
> KILLED
>
> Map 1 . KILLED 15 1401 0
> 1
> Map 10 . SUCCEEDED 15 1500 0
> 0
> Map 12 . SUCCEEDED 15 1500 0
> 0
> Map 13 . SUCCEEDED 1 100 0
> 0
> Map 14 . SUCCEEDED 1 100 0
> 0
> Map 15 . SUCCEEDED 1 100 0
> 0
> Map 16 . SUCCEEDED 1 100 0
> 0
> Map 17 . SUCCEEDED 1 100 0
> 0
> Map 4 .. SUCCEEDED 15 1500 0
> 0
> Map 5 .. SUCCEEDED 1 100 0
> 0
> Map 6 .. SUCCEEDED 1 100 0
> 0
> Map 7 .. SUCCEEDED 1 100 0
> 0
> Map 8 .. SUCCEEDED 1 100 0
> 0
> Map 9 .. SUCCEEDED 1 100 0
> 0
> Reducer 11FAILED315 00 315 774
> 314
> Reducer 2 KILLED315 00 315 0
> 315
>
> VERTICES: 14/16 [==>>] 9%ELAPSED TIME: 130.89 s
>
>
> Status: Failed
> Vertex re-running, vertexName=Map 12,
> vertexId=vertex_1494499520849_1063145_1_07
> Vertex re-running, vertexName=Map 1,
> vertexId=vertex_1494499520849_1063145_1_11
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 12,
> vertexId=vertex_1494499520849_1063145_1_07
> Vertex re-running, vertexName=Map 4,
> vertexId=vertex_1494499520849_1063145_1_03
> Vertex re-running, vertexName=Map 10,
> vertexId=vertex_1494499520849_1063145_1_06
> Vertex re-running, vertexName=Map 5,
> vertexId=vertex_1494499520849_1063145_1_02
> Vertex failed, vertexName=Reducer 11,
> vertexId=vertex_1494499520849_1063145_1_14, diagnostics=[Task failed,
> taskId=task_1494499520849_1063145_1_14_13, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1494499520849_1063145_1_14_13_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"0002991000585141671"},"value":{"_col0":"","_col1":""}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
[jira] [Commented] (TEZ-3774) hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0)
[
https://issues.apache.org/jira/browse/TEZ-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062874#comment-16062874
]
xujie commented on TEZ-3774:
set hive.execution.engine=tez;
WITH TEMP_S02_CARD_DATE_H AS
(SELECT
CARD_NBRAS CARD_NUM
,MIN(CARD_ACTIVE_DATE) AS CARD_ACTIVE_DATE
,MAX(DATE_ONE_ACTIVE)AS DATE_ONE_ACTIVE
,MAX(DATE_EXPIRE)AS DATE_EXPIRE
--FROM MID.FS_VP_AMED_CUR
FROM BASE.VP_AMED
WHERE DT = '20170604'
GROUP BY CARD_NBR
)
,
-- 卡片信息
TMP_CREDIT_CARD_01 AS
(SELECT
POST_TO_ACCT AS POST_TO_ACCT -- 对应帐户的帐号
,CARD_NBR AS CARD_NBR -- 卡号
,ORG AS ORG-- 币种
,USER_CODE_2 AS USER_CODE_2-- 是否该帐户第一张卡标志
,DATE_EXPIRE AS DATE_EXPIRE-- 卡片到期日
,BLOCK_CODEAS BLOCK_CODE -- BLOCK_CODE
,CARD_ACTIVE_DATE AS CARD_ACTIVE_DATE -- 卡片激活日期
,DATE_ONE_ACTIVE AS DATE_ONE_ACTIVE-- 一帐通激活日期
,LOGO AS LOGO -- TYPE卡种卡类
,CARD_STATUS_CDAS CARD_STATUS_CD -- 卡状态代码
,FIRST_CARD_VERIFY_DATEAS FIRST_CARD_VERIFY_DATE -- 可激活日期 --20130914
--,ROW_NUMBER() OVER(PARTITION BY POST_TO_ACCT ORDER BY APPLICATION_NO) AS
CARD_SEQ
FROM(SELECT
E1.POST_TO_ACCT AS POST_TO_ACCT
,E1.CARD_NBR AS CARD_NBR
,E1.ORGAS ORG
,E1.USER_CODE_2AS USER_CODE_2
,E1.USER_3 AS APPLICATION_NO
,E3.DATE_EXPIREAS DATE_EXPIRE
,E1.BLOCK_CODE AS BLOCK_CODE
,E3.CARD_ACTIVE_DATE AS CARD_ACTIVE_DATE
,E3.DATE_ONE_ACTIVEAS DATE_ONE_ACTIVE
,E1.LOGO AS LOGO
,E1.CURR_FIRST_USAGE_FLAG AS CARD_STATUS_CD
,E1.DATE_FIRST_CARD_VERIFY AS FIRST_CARD_VERIFY_DATE
--FROM MID.FS_VP_AMED_CUR E1
FROM BASE.VP_AMED E1
LEFT JOIN TEMP_S02_CARD_DATE_H E3
ON E1.CARD_NBR = E3.CARD_NUM
WHERE E1.DT = '20170604'
) T
)
,
-- 注销账户判断
TEMP_ACCT_WRITE_OFF AS
(SELECT
ACCT
,CASE WHEN SUM(IND) > 0
THEN 1
ELSE 0
END AS PASS_WRITE_OFF_IND
--账户双边每边至少有一个BLOCK CODE为B,Q,A,M,S之一
FROM
(SELECT
ACCT
,CASE WHEN SUM(IND) >= 1 THEN 1
ELSE 0
END AS IND
FROM(SELECT
ACCT
,ORG
,CASE WHEN (BLOCK_CODE_1 IN ('B','Q','A','M','S') OR BLOCK_CODE_2 IN
('B','Q','A','M','S')) THEN 1
ELSE 0
END AS IND
FROM BASE.VP_AMBS_TMP_CREDIT_ACCT_H_01
WHERE DT = '20170604'
) T
GROUP BY ACCT
UNION ALL
--卡片存在X
SELECT
ACCT
,CASE WHEN SUM(CARD_NBR_SUM) >= 1 THEN 1
ELSE 0
END AS IND
FROM(SELECT
POST_TO_ACCTAS ACCT
,SUM(CASE WHEN COALESCE(BLOCK_CODE,'') ='X' THEN 1
ELSE 0
END) AS CARD_NBR_SUM
FROM TMP_CREDIT_CARD_01
GROUP BY POST_TO_ACCT) T
GROUP BY ACCT
) T
GROUP BY ACCT)
,
--取账单日和下一账单日
TEMP_STMT_DATE AS
(SELECT
F1.ACCT AS ACCOUNT_NUM
,F1.ORG AS ACCOUNT_MODIFIER_NUM
,MAX(CASE WHEN F1.DATE_LAST_CYCLE IS NOT NULL THEN F1.DATE_LAST_CYCLE
ELSE CONCAT('201705',F1.BILLING_CYCLE)
END) AS STMT_DATE
--账单日
,MAX(CONCAT('201706',F1.BILLING_CYCLE)) AS NEXT_STMT_DATE--下一账单日
FROM BASE.VP_AMBS F1 --取上个月的
WHERE F1.DT = '20170531'
GROUP BY F1.ACCT,F1.ORG
)
,
TMP_CREDIT_ACCT_H_G1 AS
(SELECT
ACCT
,COALESCE(BLOCK_CODE_1,' ') AS BLOCK_CODE_1
,COALESCE(BLOCK_CODE_2,' ') AS BLOCK_CODE_2
,BLOCK_CODE_1_SET_DATE AS BLOCK_CODE_1_SET_DATE
,BLOCK_CODE_2_SET_DATE AS BLOCK_CODE_2_SET_DATE
,CUST_TYPE
FROM BASE.VP_AMBS_TMP_CREDIT_ACCT_H_01
WHERE DT = '20170604'
AND ORG = '242'
)
,
TMP_CREDIT_ACCT_H_G2 AS
(SELECT
ACCT
,COALESCE(BLOCK_CODE_1,' ') AS BLOCK_CODE_1
,CASE WHEN BLOCK_CODE_2 = 'Z' AND COALESCE(BLOCK_CODE_2_MEMO,' ') IN
('UCF','CUCF')
THEN ' ' -- 美元边需要将z+ucf/cucf的block_code变为' '
ELSE COALESCE(BLOCK_CODE_2,' ')
END AS BLOCK_CODE_2
,BLOCK_CODE_1_SET_DATE AS BLOCK_CODE_1_SET_DATE
,BLOCK_CODE_2_SET_DATE AS BLOCK_CODE_2_SET_DATE
,CUST_TYPE
FROM BASE.VP_AMBS_TMP_CREDIT_ACCT_H_01
WHERE DT = '20170604'
AND ORG = '241'
)
,
-- 汇总双边账户的block_code
TMP_CREDIT_ACCT_H_04 AS
(SELECT
G1.ACCT
,G1.BLOCK_CODE_1AS BLOCK_CODE_1 -- 人民币边block_code_1
,coalesce(G3.PRI,0) AS PRI_1--
人民币边block_code_1的优先级
,coalesce(G1.BLOCK_CODE_1_SET_DATE,'29991231') AS BLOCK_CODE_1_SET_DATE
-- 人民币边Block_Code_1_Set_Date
,G1.BLOCK_CODE_2AS BLO
[jira] [Created] (TEZ-3774) hive use tez engine occur org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0)
xujie created TEZ-3774:
--
Summary: hive use tez engine occur
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
Key: TEZ-3774
URL: https://issues.apache.org/jira/browse/TEZ-3774
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.8.5
Reporter: xujie
Status: Running (Executing on YARN cluster with App id
application_1494499520849_1063145)
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
Map 1 . KILLED 15 1401 0 1
Map 10 . SUCCEEDED 15 1500 0 0
Map 12 . SUCCEEDED 15 1500 0 0
Map 13 . SUCCEEDED 1 100 0 0
Map 14 . SUCCEEDED 1 100 0 0
Map 15 . SUCCEEDED 1 100 0 0
Map 16 . SUCCEEDED 1 100 0 0
Map 17 . SUCCEEDED 1 100 0 0
Map 4 .. SUCCEEDED 15 1500 0 0
Map 5 .. SUCCEEDED 1 100 0 0
Map 6 .. SUCCEEDED 1 100 0 0
Map 7 .. SUCCEEDED 1 100 0 0
Map 8 .. SUCCEEDED 1 100 0 0
Map 9 .. SUCCEEDED 1 100 0 0
Reducer 11FAILED315 00 315 774 314
Reducer 2 KILLED315 00 315 0 315
VERTICES: 14/16 [==>>] 9%ELAPSED TIME: 130.89 s
Status: Failed
Vertex re-running, vertexName=Map 12, vertexId=vertex_1494499520849_1063145_1_07
Vertex re-running, vertexName=Map 1, vertexId=vertex_1494499520849_1063145_1_11
Vertex re-running, vertexName=Map 10, vertexId=vertex_1494499520849_1063145_1_06
Vertex re-running, vertexName=Map 4, vertexId=vertex_1494499520849_1063145_1_03
Vertex re-running, vertexName=Map 10, vertexId=vertex_1494499520849_1063145_1_06
Vertex re-running, vertexName=Map 4, vertexId=vertex_1494499520849_1063145_1_03
Vertex re-running, vertexName=Map 12, vertexId=vertex_1494499520849_1063145_1_07
Vertex re-running, vertexName=Map 4, vertexId=vertex_1494499520849_1063145_1_03
Vertex re-running, vertexName=Map 10, vertexId=vertex_1494499520849_1063145_1_06
Vertex re-running, vertexName=Map 5, vertexId=vertex_1494499520849_1063145_1_02
Vertex failed, vertexName=Reducer 11,
vertexId=vertex_1494499520849_1063145_1_14, diagnostics=[Task failed,
taskId=task_1494499520849_1063145_1_14_13, diagnostics=[TaskAttempt 0
failed, info=[Error: Error while running task ( failure ) :
attempt_1494499520849_1063145_1_14_13_0:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=0)
{"key":{"reducesinkkey0":"0002991000585141671"},"value":{"_col0":"","_col1":""}}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
org.apache.ha
