[jira] [Commented] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
[ https://issues.apache.org/jira/browse/TEZ-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884071#comment-15884071 ] Rajesh Balamohan commented on TEZ-3638: --- In that case we do not need to log additionally about the number of tasks to be removed. Moving existing log to debug should be sufficient. > VertexImpl logs too much at info when removing tasks after auto-reduce > parallelism > -- > > Key: TEZ-3638 > URL: https://issues.apache.org/jira/browse/TEZ-3638 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3638.1.patch > > > Removing task is logged at info level and excessive if parallelism > is reduced by 1k-10k tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
[
https://issues.apache.org/jira/browse/TEZ-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884068#comment-15884068
]
Jonathan Eagles commented on TEZ-3638:
--
[~rajesh.balamohan], Found the below line is already printed as part of
auto-reduce parallelism.
{noformat}
INFO | Vertex vertex_1487998327863_0008_1_01 [Summation] parallelism set to 18
from 4000
{noformat}
Do you want the additional line or does the line above already suffice?
> VertexImpl logs too much at info when removing tasks after auto-reduce
> parallelism
> --
>
> Key: TEZ-3638
> URL: https://issues.apache.org/jira/browse/TEZ-3638
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3638.1.patch
>
>
> Removing task is logged at info level and excessive if parallelism
> is reduced by 1k-10k tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
[
https://issues.apache.org/jira/browse/TEZ-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884062#comment-15884062
]
Rajesh Balamohan commented on TEZ-3638:
---
Current log line can be at debug level with {}. Adding "Removed tasks"
after the while loop with info logging would be useful.
> VertexImpl logs too much at info when removing tasks after auto-reduce
> parallelism
> --
>
> Key: TEZ-3638
> URL: https://issues.apache.org/jira/browse/TEZ-3638
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3638.1.patch
>
>
> Removing task is logged at info level and excessive if parallelism
> is reduced by 1k-10k tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
[ https://issues.apache.org/jira/browse/TEZ-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3638: - Attachment: TEZ-3638.1.patch [~rajesh.balamohan], was wonder if we should log at debug, remove, or just replace with a "Removed tasks". > VertexImpl logs too much at info when removing tasks after auto-reduce > parallelism > -- > > Key: TEZ-3638 > URL: https://issues.apache.org/jira/browse/TEZ-3638 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3638.1.patch > > > Removing task is logged at info level and excessive if parallelism > is reduced by 1k-10k tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
[ https://issues.apache.org/jira/browse/TEZ-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884029#comment-15884029 ] Jonathan Eagles commented on TEZ-3638: -- Not sure what info if any should be logged here. Posting patch to drop logging from info level to debug level. During some extreme auto-reduction test cases the log lines were overwhelming the dag log file and didn't provide much value. > VertexImpl logs too much at info when removing tasks after auto-reduce > parallelism > -- > > Key: TEZ-3638 > URL: https://issues.apache.org/jira/browse/TEZ-3638 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > > Removing task is logged at info level and excessive if parallelism > is reduced by 1k-10k tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TEZ-3638) VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
Jonathan Eagles created TEZ-3638: Summary: VertexImpl logs too much at info when removing tasks after auto-reduce parallelism Key: TEZ-3638 URL: https://issues.apache.org/jira/browse/TEZ-3638 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Removing task is logged at info level and excessive if parallelism is reduced by 1k-10k tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Failed: TEZ-1526 PreCommit Build #2277
Jira: https://issues.apache.org/jira/browse/TEZ-1526
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2277/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 318.81 KB...]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-runtime-library
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854613/TEZ-1526.6.patch
against master revision 2158b95.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerUnpartitioned
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2277//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2277//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a3f90b0166429ba8a3f945913f79858895544fff logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.44 MB of artifacts by 27.2% relative to #2273
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerUnpartitioned.testAutoGrouping
Error Message:
null
Stack Trace:
java.lang.NullPointerException: null
at
org.apache.tez.dag.records.TaskAttemptIdentifierImpl.(TaskAttemptIdentifierImpl.java:31)
at
org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerUnpartitioned.testAutoGroupingHelper(TestCartesianProductVertexManagerUnpartitioned.java:295)
at
org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerUnpartitioned.testAutoGrouping(TestCartesianProductVertexManagerUnpartitioned.java:240)
[jira] [Commented] (TEZ-1526) LoadingCache for TezTaskID slow for large jobs
[
https://issues.apache.org/jira/browse/TEZ-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883859#comment-15883859
]
TezQA commented on TEZ-1526:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854613/TEZ-1526.6.patch
against master revision 2158b95.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerUnpartitioned
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2277//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2277//console
This message is automatically generated.
> LoadingCache for TezTaskID slow for large jobs
> --
>
> Key: TEZ-1526
> URL: https://issues.apache.org/jira/browse/TEZ-1526
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: performance
> Attachments: 10-TezTaskIDs.patch, TEZ-1526.3.patch,
> TEZ-1526.4.patch, TEZ-1526.5.patch, TEZ-1526.6.patch, TEZ-1526-v1.patch,
> TEZ-1526-v2.patch
>
>
> Using the LoadingCache with default builder settings. 100,000 TezTaskIDs are
> created in 10 seconds on my setup. With a LoadingCache initialCapacity of
> 10,000 they are created in 300 ms. With no LoadingCache, they are created in
> 10 ms. A test case in attached to illustrate the condition I would like to be
> sped up.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (TEZ-1526) LoadingCache for TezTaskID slow for large jobs
[ https://issues.apache.org/jira/browse/TEZ-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-1526: - Attachment: TEZ-1526.6.patch > LoadingCache for TezTaskID slow for large jobs > -- > > Key: TEZ-1526 > URL: https://issues.apache.org/jira/browse/TEZ-1526 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Labels: performance > Attachments: 10-TezTaskIDs.patch, TEZ-1526.3.patch, > TEZ-1526.4.patch, TEZ-1526.5.patch, TEZ-1526.6.patch, TEZ-1526-v1.patch, > TEZ-1526-v2.patch > > > Using the LoadingCache with default builder settings. 100,000 TezTaskIDs are > created in 10 seconds on my setup. With a LoadingCache initialCapacity of > 10,000 they are created in 300 ms. With no LoadingCache, they are created in > 10 ms. A test case in attached to illustrate the condition I would like to be > sped up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TEZ-3625) Dag.getVertex should obtain a readlock
[ https://issues.apache.org/jira/browse/TEZ-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883486#comment-15883486 ] Siddharth Seth edited comment on TEZ-3625 at 2/24/17 8:54 PM: -- This change ended up resulting in a race and occasional deadlock between dagCompletion and some call from the VertexManagerPlugin for dags with 0 fragments. Reverting. The extra appThreads has caused quite a few problems. Will look to see how this particular deadlock can be avoided. Ideally, there should be a readLock on getVertex. was (Author: sseth): This change ended up resulting in a race and occasional deadlock between dagCompletion and some call from the VertexManagerPlugin for dags with 0 fragments. Reverting. > Dag.getVertex should obtain a readlock > -- > > Key: TEZ-3625 > URL: https://issues.apache.org/jira/browse/TEZ-3625 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Fix For: 0.9.0, 0.8.5 > > Attachments: am_jstack, TEZ-3625.01.patch > > > getVertex(String) does not obtain a readLock. It should, similar to > getVertex(TezVertexId) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3625) Dag.getVertex should obtain a readlock
[ https://issues.apache.org/jira/browse/TEZ-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-3625: Attachment: am_jstack > Dag.getVertex should obtain a readlock > -- > > Key: TEZ-3625 > URL: https://issues.apache.org/jira/browse/TEZ-3625 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Fix For: 0.9.0, 0.8.5 > > Attachments: am_jstack, TEZ-3625.01.patch > > > getVertex(String) does not obtain a readLock. It should, similar to > getVertex(TezVertexId) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (TEZ-3625) Dag.getVertex should obtain a readlock
[ https://issues.apache.org/jira/browse/TEZ-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reopened TEZ-3625: - > Dag.getVertex should obtain a readlock > -- > > Key: TEZ-3625 > URL: https://issues.apache.org/jira/browse/TEZ-3625 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Fix For: 0.9.0, 0.8.5 > > Attachments: TEZ-3625.01.patch > > > getVertex(String) does not obtain a readLock. It should, similar to > getVertex(TezVertexId) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3625) Dag.getVertex should obtain a readlock
[ https://issues.apache.org/jira/browse/TEZ-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883486#comment-15883486 ] Siddharth Seth commented on TEZ-3625: - This change ended up resulting in a race and occasional deadlock between dagCompletion and some call from the VertexManagerPlugin for dags with 0 fragments. Reverting. > Dag.getVertex should obtain a readlock > -- > > Key: TEZ-3625 > URL: https://issues.apache.org/jira/browse/TEZ-3625 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Fix For: 0.9.0, 0.8.5 > > Attachments: TEZ-3625.01.patch > > > getVertex(String) does not obtain a readLock. It should, similar to > getVertex(TezVertexId) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3274) Vertex with MRInput and shuffle input does not respect slow start
[ https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883086#comment-15883086 ] Eric Badger commented on TEZ-3274: -- [~sseth], do you have any comments on the patch in its current state (aka without doing anything to auto parallelism). > Vertex with MRInput and shuffle input does not respect slow start > - > > Key: TEZ-3274 > URL: https://issues.apache.org/jira/browse/TEZ-3274 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Eric Badger > Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch > > > Vertices with shuffle input and MRInput choose RootInputVertexManager (and > not ShuffleVertexManager) and start containers and tasks immediately. In this > scenario, resources can be wasted since they do not respect > tez.shuffle-vertex-manager.min-src-fraction > tez.shuffle-vertex-manager.max-src-fraction. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-996) Support for fetching credentials for mapreduce.job.hdfs-servers
[ https://issues.apache.org/jira/browse/TEZ-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882452#comment-15882452 ] Denis Jannot commented on TEZ-996: -- [~rohini] Is there a way to set this parameter from the Hive shell or on any Tez configuration file ? > Support for fetching credentials for mapreduce.job.hdfs-servers > --- > > Key: TEZ-996 > URL: https://issues.apache.org/jira/browse/TEZ-996 > Project: Apache Tez > Issue Type: Bug >Reporter: Rohini Palaniswamy > > mapreduce.job.hdfs-servers is a property that JobClient looks at and fetches > delegation tokens before submitting the job in MR. This is used lot of cross > cluster communication. Would be good to have support added in Tez by default > for that instead of multiple projects like Pig and Hive repeating it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3637) TezMerger logs too much on INFO level
[
https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882355#comment-15882355
]
TezQA commented on TEZ-3637:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854406/TEZ-3637.02.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2276//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2276//console
This message is automatically generated.
> TezMerger logs too much on INFO level
> -
>
> Key: TEZ-3637
> URL: https://issues.apache.org/jira/browse/TEZ-3637
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
> Attachments: TEZ-3637.01.patch, TEZ-3637.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Failed: TEZ-3637 PreCommit Build #2276
Jira: https://issues.apache.org/jira/browse/TEZ-3637
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2276/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 317.50 KB...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854406/TEZ-3637.02.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2276//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2276//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
865a22ed5ce633c983092718b6d7734bed412e48 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
Failed: TEZ-3637 PreCommit Build #2275
Jira: https://issues.apache.org/jira/browse/TEZ-3637
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2275/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 319.56 KB...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854406/TEZ-3637.02.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestSecureShuffle
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2275//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2275//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
961ad8512fb4bf5537fc6dda170a411d03436ea6 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.test.TestSecureShuffle.testSecureShuffle[test[sslInCluster:true,
resultWithTezSSL:0, resultWithoutTezSSL:1, asyncHttp:false]]
Error Message:
test timed out after 50 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 50 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1454)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy96.getDAGStatus(Unknown Source)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatusViaAM(DAGClientRPCImpl.java:198)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:97)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:371)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:221)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:269)
at
org.apache.tez.mapreduce.examples.TestOrderedWordCount.run(TestOrderedWordCount.java:585)
at
org.apache.tez.test.TestSecureShuffle.baseTest(TestSecureShuffle.java:158)
at
org.apache.tez.test.TestSecureShuffle.testSecureShuffle(TestSecureShuffle.java:176)
[jira] [Commented] (TEZ-3637) TezMerger logs too much on INFO level
[
https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882344#comment-15882344
]
TezQA commented on TEZ-3637:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854406/TEZ-3637.02.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestSecureShuffle
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2275//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2275//console
This message is automatically generated.
> TezMerger logs too much on INFO level
> -
>
> Key: TEZ-3637
> URL: https://issues.apache.org/jira/browse/TEZ-3637
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
> Attachments: TEZ-3637.01.patch, TEZ-3637.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3637) TezMerger logs too much on INFO level
[ https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882277#comment-15882277 ] Rajesh Balamohan commented on TEZ-3637: --- Can the stats size be 0 and lead to error during division (in corner cases when empty partitions are not sent via events)? . statsInMemTotal/statsInMemLastLog can be made "private final" before commit. > TezMerger logs too much on INFO level > - > > Key: TEZ-3637 > URL: https://issues.apache.org/jira/browse/TEZ-3637 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Attachments: TEZ-3637.01.patch, TEZ-3637.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3637) TezMerger logs too much on INFO level
[
https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882273#comment-15882273
]
TezQA commented on TEZ-3637:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854398/TEZ-3637.01.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.analyzer.TestAnalyzer
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2274//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2274//console
This message is automatically generated.
> TezMerger logs too much on INFO level
> -
>
> Key: TEZ-3637
> URL: https://issues.apache.org/jira/browse/TEZ-3637
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
> Attachments: TEZ-3637.01.patch, TEZ-3637.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Failed: TEZ-3637 PreCommit Build #2274
Jira: https://issues.apache.org/jira/browse/TEZ-3637
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2274/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 319.29 KB...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-job-analyzer
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12854398/TEZ-3637.01.patch
against master revision 359cd75.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.analyzer.TestAnalyzer
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2274//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2274//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
04a0ec3130b98f71e3b9bd4a064cc8b8666a4f12 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED: org.apache.tez.analyzer.TestAnalyzer.testWithATS
Error Message:
test timed out after 30 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 30 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1454)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy93.getDAGStatus(Unknown Source)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatusViaAM(DAGClientRPCImpl.java:198)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:97)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:371)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:221)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:269)
at org.apache.tez.analyzer.TestAnalyzer.runDAG(TestAnalyzer.java:243)
at
org.apache.tez.analyzer.TestAnalyzer.testBasicInputFailureWithoutExit(TestAnalyzer.java:494)
at org.apache.tez.analyzer.TestAnalyzer.runTests(TestAnalyzer.java:373)
at
org.apache.tez.analyzer.TestAnalyzer.testWithATS(TestAnalyzer.java:354)
[jira] [Updated] (TEZ-3637) TezMerger logs too much on INFO level
[ https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-3637: Attachment: TEZ-3637.02.patch Updated with some aggregates for in-memory fetches. > TezMerger logs too much on INFO level > - > > Key: TEZ-3637 > URL: https://issues.apache.org/jira/browse/TEZ-3637 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Attachments: TEZ-3637.01.patch, TEZ-3637.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3637) TezMerger logs too much on INFO level
[ https://issues.apache.org/jira/browse/TEZ-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882175#comment-15882175 ] Rajesh Balamohan commented on TEZ-3637: --- lgtm. +1. Minor comment (lastInMemSegmentLogTime can be moved to starting of the file). Log mining tools have to rely on debug logs as in-memory closing would be printed every 30 seconds. > TezMerger logs too much on INFO level > - > > Key: TEZ-3637 > URL: https://issues.apache.org/jira/browse/TEZ-3637 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Attachments: TEZ-3637.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
