[jira] [Commented] (TEZ-3716) Allow attempt retries to be treated the same as the first attempt
[
https://issues.apache.org/jira/browse/TEZ-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009122#comment-16009122
]
TezQA commented on TEZ-3716:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867920/TEZ-3716.02.patch
against master revision dd9c517.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestTaskSchedulerManager
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2440//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2440//console
This message is automatically generated.
> Allow attempt retries to be treated the same as the first attempt
> -
>
> Key: TEZ-3716
> URL: https://issues.apache.org/jira/browse/TEZ-3716
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3716.01.patch, TEZ-3716.02.patch
>
>
> Essentially - don't change attempt priority due to a failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3716) Allow attempt retries to be treated the same as the first attempt
[
https://issues.apache.org/jira/browse/TEZ-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009028#comment-16009028
]
TezQA commented on TEZ-3716:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867682/TEZ-3716.01.patch
against master revision dd9c517.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
org.apache.tez.dag.app.dag.impl.TestTaskImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2439//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2439//console
This message is automatically generated.
> Allow attempt retries to be treated the same as the first attempt
> -
>
> Key: TEZ-3716
> URL: https://issues.apache.org/jira/browse/TEZ-3716
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3716.01.patch
>
>
> Essentially - don't change attempt priority due to a failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3724) Tez UI on HTTP "corrects" HTTPS REST calls to HTTP
[
https://issues.apache.org/jira/browse/TEZ-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008769#comment-16008769
]
TezQA commented on TEZ-3724:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867848/TEZ-3724.1.patch
against master revision de21f99.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2438//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2438//console
This message is automatically generated.
> Tez UI on HTTP "corrects" HTTPS REST calls to HTTP
> --
>
> Key: TEZ-3724
> URL: https://issues.apache.org/jira/browse/TEZ-3724
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3724.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3715) Differentiate between TaskAttempt submission and TaskAttempt started
[
https://issues.apache.org/jira/browse/TEZ-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008763#comment-16008763
]
TezQA commented on TEZ-3715:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867671/TEZ-3715.01.patch
against master revision de21f99.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 32 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2437//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2437//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2437//console
This message is automatically generated.
> Differentiate between TaskAttempt submission and TaskAttempt started
>
>
> Key: TEZ-3715
> URL: https://issues.apache.org/jira/browse/TEZ-3715
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3715.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3723) TezIndexRecord#hasData() returns true for empty index record in the Unordered case
[
https://issues.apache.org/jira/browse/TEZ-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008385#comment-16008385
]
TezQA commented on TEZ-3723:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867790/TEZ-3723.002.patch
against master revision de21f99.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.TestShuffleUtils
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2436//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2436//console
This message is automatically generated.
> TezIndexRecord#hasData() returns true for empty index record in the Unordered
> case
> --
>
> Key: TEZ-3723
> URL: https://issues.apache.org/jira/browse/TEZ-3723
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: TEZ-3723.001.patch, TEZ-3723.002.patch
>
>
> In the case of Unordered writer, use of {{hasData()}} returns true for empty
> partitions since the rawlength is zero (as writer skips writing them out
> altogether) and the check in hasData will fail as {{rawlength ==
> header.length +2}} fails.
> This currently works for the ordered case(and the method is used here unlike
> in unordered writer) but after TEZ-3605, it will fail for both scenarios.
> Even if we don't use the method, the logic should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3722) Have MultiMROutput support different configurations for different writers
[
https://issues.apache.org/jira/browse/TEZ-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007587#comment-16007587
]
TezQA commented on TEZ-3722:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867690/TEZ-3722.patch
against master revision a55fe80.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.analyzer.TestAnalyzer
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2434//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2434//console
This message is automatically generated.
> Have MultiMROutput support different configurations for different writers
> -
>
> Key: TEZ-3722
> URL: https://issues.apache.org/jira/browse/TEZ-3722
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
> Attachments: TEZ-3722.patch
>
>
> TEZ-3215 added the support for MultiMROutput so that a task can write its
> output to different HDFS files depending on the data. Internally
> MultiMROutput creates different writers keyed by the path. All writers use
> the same configuration.
> We have use cases that different writers need to have different
> configurations, it can be used to specify different output formats, or
> different configuration values for the same output format. So it will be
> useful to allow applications to specify different configuration for different
> writers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3715) Differentiate between TaskAttempt submission and TaskAttempt started
[
https://issues.apache.org/jira/browse/TEZ-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007435#comment-16007435
]
TezQA commented on TEZ-3715:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867671/TEZ-3715.01.patch
against master revision a55fe80.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 32 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2432//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2432//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2432//console
This message is automatically generated.
> Differentiate between TaskAttempt submission and TaskAttempt started
>
>
> Key: TEZ-3715
> URL: https://issues.apache.org/jira/browse/TEZ-3715
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3715.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3716) Allow attempt retries to be treated the same as the first attempt
[
https://issues.apache.org/jira/browse/TEZ-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007422#comment-16007422
]
TezQA commented on TEZ-3716:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867682/TEZ-3716.01.patch
against master revision a55fe80.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2433//console
This message is automatically generated.
> Allow attempt retries to be treated the same as the first attempt
> -
>
> Key: TEZ-3716
> URL: https://issues.apache.org/jira/browse/TEZ-3716
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3716.01.patch
>
>
> Essentially - don't change attempt priority due to a failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3666) Integer overflow in ShuffleVertexManagerBase
[
https://issues.apache.org/jira/browse/TEZ-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007123#comment-16007123
]
TezQA commented on TEZ-3666:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867630/TEZ-3666-2.patch
against master revision dec7c1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2431//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2431//console
This message is automatically generated.
> Integer overflow in ShuffleVertexManagerBase
>
>
> Key: TEZ-3666
> URL: https://issues.apache.org/jira/browse/TEZ-3666
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: TEZ-3666-2.patch, TEZ-3666.patch
>
>
> In function getExpectedStatsInAtIndex, {{statsInMB[index] * numTasks /
> numVMEventsReceived}} could cause Integer overflow, for example when
> statsInMB[index] == 3 and numTasks == 20.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads
[
https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006014#comment-16006014
]
TezQA commented on TEZ-3719:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867505/TEZ-3719.2.patch
against master revision dec7c1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version ) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2430//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2430//console
This message is automatically generated.
> DAGImpl.computeProgress slows down dispatcher and ipc threads
> -
>
> Key: TEZ-3719
> URL: https://issues.apache.org/jira/browse/TEZ-3719
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Gopal V
> Attachments: dispatcher_holding_lock.png, TEZ-3719.1.patch,
> TEZ-3719.2.patch, TEZ-3719.wip.patch, tez_computeProgress_waiting_for_lock.png
>
>
> The Vertex readLock is held while computing progress, which prevents
> scheduling of tasks from happening on that vertex (which needs a writeLock).
> Making the locked section do as little work as possible would speedup the AM.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product
[
https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006008#comment-16006008
]
TezQA commented on TEZ-3708:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867500/TEZ-3708.4.patch
against master revision dec7c1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2429//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2429//console
This message is automatically generated.
> Improve parallelism and auto grouping of unpartitioned cartesian product
>
>
> Key: TEZ-3708
> URL: https://issues.apache.org/jira/browse/TEZ-3708
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3708.1.patch, TEZ-3708.2.patch, TEZ-3708.3.patch,
> TEZ-3708.4.patch
>
>
> Current unpartitioned cartesian product has a few limitations
> 1. parallelism can be not enough in case of large split and small # src task
> 2. parallelism can be too much in in case of large # src task
> 3. workload is not ideally distributed across the worker. Even with auto
> grouping, grouping by size may not be accurate because same size can means
> different #record and different cartesian product ops.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads
[
https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005930#comment-16005930
]
TezQA commented on TEZ-3719:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867452/TEZ-3719.1.patch
against master revision dec7c1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2428//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2428//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2428//console
This message is automatically generated.
> DAGImpl.computeProgress slows down dispatcher and ipc threads
> -
>
> Key: TEZ-3719
> URL: https://issues.apache.org/jira/browse/TEZ-3719
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Gopal V
> Attachments: dispatcher_holding_lock.png, TEZ-3719.1.patch,
> TEZ-3719.wip.patch, tez_computeProgress_waiting_for_lock.png
>
>
> The Vertex readLock is held while computing progress, which prevents
> scheduling of tasks from happening on that vertex (which needs a writeLock).
> Making the locked section do as little work as possible would speedup the AM.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product
[
https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003877#comment-16003877
]
TezQA commented on TEZ-3708:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867214/TEZ-3708.3.patch
against master revision 93bd26e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestTezJobs
org.apache.tez.test.TestFaultTolerance
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
org.apache.tez.runtime.library.cartesianproduct.TestFairCartesianProductVertexManager
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2427//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2427//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2427//console
This message is automatically generated.
> Improve parallelism and auto grouping of unpartitioned cartesian product
>
>
> Key: TEZ-3708
> URL: https://issues.apache.org/jira/browse/TEZ-3708
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3708.1.patch, TEZ-3708.2.patch, TEZ-3708.3.patch
>
>
> Current unpartitioned cartesian product has a few limitations
> 1. parallelism can be not enough in case of large split and small # src task
> 2. parallelism can be too much in in case of large # src task
> 3. workload is not ideally distributed across the worker. Even with auto
> grouping, grouping by size may not be accurate because same size can means
> different #record and different cartesian product ops.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3717) Master does not compile with -Phadoop28.
[
https://issues.apache.org/jira/browse/TEZ-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002469#comment-16002469
]
TezQA commented on TEZ-3717:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12867069/TEZ-3717.01.patch
against master revision 93bd26e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2426//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2426//console
This message is automatically generated.
> Master does not compile with -Phadoop28.
>
>
> Key: TEZ-3717
> URL: https://issues.apache.org/jira/browse/TEZ-3717
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3717.01.patch
>
>
> Tez build with -Phadoop28 fails with DFSConfigKeys class not found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product
[
https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001255#comment-16001255
]
TezQA commented on TEZ-3708:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866515/TEZ-3708.2.patch
against master revision d9f542f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2424//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2424//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2424//console
This message is automatically generated.
> Improve parallelism and auto grouping of unpartitioned cartesian product
>
>
> Key: TEZ-3708
> URL: https://issues.apache.org/jira/browse/TEZ-3708
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3708.1.patch, TEZ-3708.2.patch
>
>
> Current unpartitioned cartesian product has a few limitations
> 1. parallelism can be not enough in case of large split and small # src task
> 2. parallelism can be too much in in case of large # src task
> 3. workload is not ideally distributed across the worker. Even with auto
> grouping, grouping by size may not be accurate because same size can means
> different #record and different cartesian product ops.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3714) Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form Hive side.
[
https://issues.apache.org/jira/browse/TEZ-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001079#comment-16001079
]
TezQA commented on TEZ-3714:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866906/TEZ-3714.1.patch
against master revision d9f542f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2423//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2423//console
This message is automatically generated.
> Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form
> Hive side.
> --
>
> Key: TEZ-3714
> URL: https://issues.apache.org/jira/browse/TEZ-3714
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3714.1.patch
>
>
> As of now Hive Queries pages does a reverse lookup to display DAG and App
> IDs. i.e. For each query, the UI looks for DAGs where callerId = hive query
> id.
> This change prevents this ATS fetch when the data is available.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000555#comment-16000555
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866856/TEZ-3700.addendum.2.patch
against master revision d9f542f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2422//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2422//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 0.9.0
>
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch,
> TEZ-3700.4.patch, TEZ-3700.5.patch, TEZ-3700.addendum.1.patch,
> TEZ-3700.addendum.2.patch, TEZ-3700.addendum.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000271#comment-16000271
]
TezQA commented on TEZ-3700:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866822/TEZ-3700.addendum.1.patch
against master revision 4ed4a56.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleInputEventHandlerImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2421//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2421//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 0.9.0
>
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch,
> TEZ-3700.4.patch, TEZ-3700.5.patch, TEZ-3700.addendum.1.patch,
> TEZ-3700.addendum.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999141#comment-15999141
]
TezQA commented on TEZ-3700:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866105/TEZ-3700.addendum.patch
against master revision 4ed4a56.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleInputEventHandlerImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2420//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2420//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 0.9.0
>
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch,
> TEZ-3700.4.patch, TEZ-3700.5.patch, TEZ-3700.addendum.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments
[
https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997652#comment-15997652
]
TezQA commented on TEZ-3709:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866512/TEZ-3709.3.patch
against master revision 68fe023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2419//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2419//console
This message is automatically generated.
> TezMerger is slow for high number of segments
> -
>
> Key: TEZ-3709
> URL: https://issues.apache.org/jira/browse/TEZ-3709
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: performance
> Attachments: TEZ-3709.1.patch, TEZ-3709.2.patch, TEZ-3709.3.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole
> list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
> ArrayList#remove(0)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3697) Adding #output_record in vertex manager event payload
[
https://issues.apache.org/jira/browse/TEZ-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997487#comment-15997487
]
TezQA commented on TEZ-3697:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866480/TEZ-3697.3.patch
against master revision 68fe023.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2418//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2418//console
This message is automatically generated.
> Adding #output_record in vertex manager event payload
> -
>
> Key: TEZ-3697
> URL: https://issues.apache.org/jira/browse/TEZ-3697
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3697.1.patch, TEZ-3697.2.patch, TEZ-3697.3.patch
>
>
> This enables estimation of total number of records generated by source
> vertex. The overhead is one more integer per payload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments
[
https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997142#comment-15997142
]
TezQA commented on TEZ-3709:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866432/TEZ-3709.2.patch
against master revision fb7b146.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2417//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2417//console
This message is automatically generated.
> TezMerger is slow for high number of segments
> -
>
> Key: TEZ-3709
> URL: https://issues.apache.org/jira/browse/TEZ-3709
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: performance
> Attachments: TEZ-3709.1.patch, TEZ-3709.2.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole
> list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
> ArrayList#remove(0)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3691) Setup fetchers to use shared executor.
[
https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996336#comment-15996336
]
TezQA commented on TEZ-3691:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866350/TEZ-3691.05.patch
against master revision fb7b146.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2416//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2416//console
This message is automatically generated.
> Setup fetchers to use shared executor.
> --
>
> Key: TEZ-3691
> URL: https://issues.apache.org/jira/browse/TEZ-3691
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch,
> TEZ-3691.04.patch, TEZ-3691.05.patch
>
>
> Change fetchers in ShuffleScheduler & ShuffleManger to use the shared
> executor service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996321#comment-15996321
]
TezQA commented on TEZ-3673:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866347/TEZ-3673.05.patch
against master revision fb7b146.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2415//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2415//console
This message is automatically generated.
> Allocate smaller buffers in UnorderedPartitionedKVWriter
>
>
> Key: TEZ-3673
> URL: https://issues.apache.org/jira/browse/TEZ-3673
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch,
> TEZ-3673.04.patch, TEZ-3673.05.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not
> get filled up. In PipelinedSorter, we start off with 32MB chunks. But
> UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need
> to revisit this allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3691) Setup fetchers to use shared executor.
[
https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996172#comment-15996172
]
TezQA commented on TEZ-3691:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866321/TEZ-3691.04.patch
against master revision 7b30785.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleInputEventHandlerImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2414//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2414//console
This message is automatically generated.
> Setup fetchers to use shared executor.
> --
>
> Key: TEZ-3691
> URL: https://issues.apache.org/jira/browse/TEZ-3691
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch,
> TEZ-3691.04.patch
>
>
> Change fetchers in ShuffleScheduler & ShuffleManger to use the shared
> executor service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3697) Adding #output_record in vertex manager event payload
[
https://issues.apache.org/jira/browse/TEZ-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996061#comment-15996061
]
TezQA commented on TEZ-3697:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866305/TEZ-3697.2.patch
against master revision 7b30785.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2413//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2413//console
This message is automatically generated.
> Adding #output_record in vertex manager event payload
> -
>
> Key: TEZ-3697
> URL: https://issues.apache.org/jira/browse/TEZ-3697
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3697.1.patch, TEZ-3697.2.patch
>
>
> This enables estimation of total number of records generated by source
> vertex. The overhead is one more integer per payload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments
[
https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995942#comment-15995942
]
TezQA commented on TEZ-3709:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866288/TEZ-3709.1.patch
against master revision 7b30785.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.common.TestTezSharedExecutor
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2412//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2412//console
This message is automatically generated.
> TezMerger is slow for high number of segments
> -
>
> Key: TEZ-3709
> URL: https://issues.apache.org/jira/browse/TEZ-3709
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Labels: performance
> Attachments: TEZ-3709.1.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole
> list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
> ArrayList#remove(0)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled
[
https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995911#comment-15995911
]
TezQA commented on TEZ-3696:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866276/TEZ-3696.004.patch
against master revision 7b30785.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.common.TestTezSharedExecutor
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2411//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2411//console
This message is automatically generated.
> Jobs can hang when both concurrency and speculation are enabled
> ---
>
> Key: TEZ-3696
> URL: https://issues.apache.org/jira/browse/TEZ-3696
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: TEZ-3696.001.patch, TEZ-3696.002.patch,
> TEZ-3696.003.patch, TEZ-3696.004.patch
>
>
> We can reproduce the hung job by doing the following:
> 1. Run a sleep job with a concurrency of 1, speculation enabled, and 3 tasks
> {noformat}
> HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar
> $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.vertex.max-task-concurrency=1
> -Dtez.am.speculation.enabled=true -Dtez.task.timeout-ms=6 -m 3 -mt 6
> -ir 0 -irt 0 -r 0 -rt 0
> {noformat}
> 2. Let the 1st task run to completion and then stop the 2nd task so that a
> speculative attempt is scheduled. Once the speculative attempt is scheduled
> for the 2nd task, continue the original attempt and let it complete.
> {noformat}
> kill -STOP
> // wait a few seconds for a speculative attempt to kick off
> kill -CONT
> {noformat}
> 3. Kill the 3rd task, which will create a 2nd attempt
> {noformat}
> kill -9
> {noformat}
> 4. The next thing to be drawn off of the queue will be the speculative
> attempt of the 2nd task. However, it is already completed, so it will just
> sit in the final state and the job will hang.
> Basically, for the failure to happen, the number of speculative tasks that
> are scheduled, but not yet ran has to be >= the concurrency of the job and
> there has to be at least 1 task failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product
[
https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995137#comment-15995137
]
TezQA commented on TEZ-3708:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12866207/TEZ-3708.1.patch
against master revision 7b30785.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2410//console
This message is automatically generated.
> Improve parallelism and auto grouping of unpartitioned cartesian product
>
>
> Key: TEZ-3708
> URL: https://issues.apache.org/jira/browse/TEZ-3708
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3708.1.patch
>
>
> Current unpartitioned cartesian product has a few limitations
> 1. parallelism can be not enough in case of large split and small # src task
> 2. parallelism can be too much in in case of large # src task
> 3. workload is not ideally distributed across the worker. Even with auto
> grouping, grouping by size may not be accurate because same size can means
> different #record and different cartesian product ops.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[
https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992677#comment-15992677
]
TezQA commented on TEZ-3691:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865920/TEZ-3691.02.patch
against master revision ebf3fb1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2409//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2409//console
This message is automatically generated.
> Move ShuffleScheduler to use the framework ExecutorService.
> ---
>
> Key: TEZ-3691
> URL: https://issues.apache.org/jira/browse/TEZ-3691
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch
>
>
> Change ShuffleScheduler to use the shard executor service, so that threads
> can be shared b/w tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[
https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992516#comment-15992516
]
TezQA commented on TEZ-3691:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865888/TEZ-3691.01.patch
against master revision ebf3fb1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffleInputEventHandlerOrderedGrouped
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2408//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2408//console
This message is automatically generated.
> Move ShuffleScheduler to use the framework ExecutorService.
> ---
>
> Key: TEZ-3691
> URL: https://issues.apache.org/jira/browse/TEZ-3691
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3691.01.patch
>
>
> Change ShuffleScheduler to use the shard executor service, so that threads
> can be shared b/w tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3707) TezSharedExecutor race condition in awaitTermination vs isTerminated.
[
https://issues.apache.org/jira/browse/TEZ-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992391#comment-15992391
]
TezQA commented on TEZ-3707:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865872/TEZ-3707.01.patch
against master revision ebf3fb1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2407//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2407//console
This message is automatically generated.
> TezSharedExecutor race condition in awaitTermination vs isTerminated.
> -
>
> Key: TEZ-3707
> URL: https://issues.apache.org/jira/browse/TEZ-3707
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3707.01.patch
>
>
> awaitTermination finishes if all the futures are resolved. isTerminated
> returns true if futures is empty. future done is invoked after the future is
> resolve, which can lead to awaitTermination finishing but isTerminated is
> false until done is invoked.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991785#comment-15991785
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865828/TEZ-3700.5.patch
against master revision ebf3fb1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2406//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2406//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch,
> TEZ-3700.4.patch, TEZ-3700.5.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989755#comment-15989755
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865655/TEZ-3700.4.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2405//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2405//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch,
> TEZ-3700.4.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled
[
https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989447#comment-15989447
]
TezQA commented on TEZ-3696:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865607/TEZ-3696.003.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2404//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2404//console
This message is automatically generated.
> Jobs can hang when both concurrency and speculation are enabled
> ---
>
> Key: TEZ-3696
> URL: https://issues.apache.org/jira/browse/TEZ-3696
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: TEZ-3696.001.patch, TEZ-3696.002.patch,
> TEZ-3696.003.patch
>
>
> We can reproduce the hung job by doing the following:
> 1. Run a sleep job with a concurrency of 1, speculation enabled, and 3 tasks
> {noformat}
> HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar
> $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.vertex.max-task-concurrency=1
> -Dtez.am.speculation.enabled=true -Dtez.task.timeout-ms=6 -m 3 -mt 6
> -ir 0 -irt 0 -r 0 -rt 0
> {noformat}
> 2. Let the 1st task run to completion and then stop the 2nd task so that a
> speculative attempt is scheduled. Once the speculative attempt is scheduled
> for the 2nd task, continue the original attempt and let it complete.
> {noformat}
> kill -STOP
> // wait a few seconds for a speculative attempt to kick off
> kill -CONT
> {noformat}
> 3. Kill the 3rd task, which will create a 2nd attempt
> {noformat}
> kill -9
> {noformat}
> 4. The next thing to be drawn off of the queue will be the speculative
> attempt of the 2nd task. However, it is already completed, so it will just
> sit in the final state and the job will hang.
> Basically, for the failure to happen, the number of speculative tasks that
> are scheduled, but not yet ran has to be >= the concurrency of the job and
> there has to be at least 1 task failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3703) Use a sha comparison to compare vertex and dag resources, if there is a mismatch
[
https://issues.apache.org/jira/browse/TEZ-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989349#comment-15989349
]
TezQA commented on TEZ-3703:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865597/TEZ-3703.02.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2403//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2403//console
This message is automatically generated.
> Use a sha comparison to compare vertex and dag resources, if there is a
> mismatch
>
>
> Key: TEZ-3703
> URL: https://issues.apache.org/jira/browse/TEZ-3703
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3703.01.patch, TEZ-3703.02.patch
>
>
> In DAG validate, when resources are compared between Dag and Vertices -
> instead of failing immediately on conflicting LR.equals, fall back to a sha
> comparison.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988306#comment-15988306
]
TezQA commented on TEZ-3673:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865463/TEZ-3673.04.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2402//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2402//console
This message is automatically generated.
> Allocate smaller buffers in UnorderedPartitionedKVWriter
>
>
> Key: TEZ-3673
> URL: https://issues.apache.org/jira/browse/TEZ-3673
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch,
> TEZ-3673.04.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not
> get filled up. In PipelinedSorter, we start off with 32MB chunks. But
> UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need
> to revisit this allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988231#comment-15988231
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865456/TEZ-3700.3.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2401//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2401//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988218#comment-15988218
]
TezQA commented on TEZ-3673:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865324/TEZ-3673.03.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.api.TestTezRuntimeConfiguration
The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2400//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2400//console
This message is automatically generated.
> Allocate smaller buffers in UnorderedPartitionedKVWriter
>
>
> Key: TEZ-3673
> URL: https://issues.apache.org/jira/browse/TEZ-3673
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not
> get filled up. In PipelinedSorter, we start off with 32MB chunks. But
> UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need
> to revisit this allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3703) Use a sha comparison to compare vertex and dag resources, if there is a mismatch
[
https://issues.apache.org/jira/browse/TEZ-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988067#comment-15988067
]
TezQA commented on TEZ-3703:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865444/TEZ-3703.01.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.api.TestDAGVerify
org.apache.tez.common.TestTezSharedExecutor
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2399//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2399//console
This message is automatically generated.
> Use a sha comparison to compare vertex and dag resources, if there is a
> mismatch
>
>
> Key: TEZ-3703
> URL: https://issues.apache.org/jira/browse/TEZ-3703
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3703.01.patch
>
>
> In DAG validate, when resources are compared between Dag and Vertices -
> instead of failing immediately on conflicting LR.equals, fall back to a sha
> comparison.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled
[
https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987145#comment-15987145
]
TezQA commented on TEZ-3696:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865397/TEZ-3696.002.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2398//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2398//console
This message is automatically generated.
> Jobs can hang when both concurrency and speculation are enabled
> ---
>
> Key: TEZ-3696
> URL: https://issues.apache.org/jira/browse/TEZ-3696
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: TEZ-3696.001.patch, TEZ-3696.002.patch
>
>
> We can reproduce the hung job by doing the following:
> 1. Run a sleep job with a concurrency of 1, speculation enabled, and 3 tasks
> {noformat}
> HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar
> $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.vertex.max-task-concurrency=1
> -Dtez.am.speculation.enabled=true -Dtez.task.timeout-ms=6 -m 3 -mt 6
> -ir 0 -irt 0 -r 0 -rt 0
> {noformat}
> 2. Let the 1st task run to completion and then stop the 2nd task so that a
> speculative attempt is scheduled. Once the speculative attempt is scheduled
> for the 2nd task, continue the original attempt and let it complete.
> {noformat}
> kill -STOP
> // wait a few seconds for a speculative attempt to kick off
> kill -CONT
> {noformat}
> 3. Kill the 3rd task, which will create a 2nd attempt
> {noformat}
> kill -9
> {noformat}
> 4. The next thing to be drawn off of the queue will be the speculative
> attempt of the 2nd task. However, it is already completed, so it will just
> sit in the final state and the job will hang.
> Basically, for the failure to happen, the number of speculative tasks that
> are scheduled, but not yet ran has to be >= the concurrency of the job and
> there has to be at least 1 task failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled
[
https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986868#comment-15986868
]
TezQA commented on TEZ-3696:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865162/TEZ-3696.001.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2397//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2397//console
This message is automatically generated.
> Jobs can hang when both concurrency and speculation are enabled
> ---
>
> Key: TEZ-3696
> URL: https://issues.apache.org/jira/browse/TEZ-3696
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: TEZ-3696.001.patch
>
>
> We can reproduce the hung job by doing the following:
> 1. Run a sleep job with a concurrency of 1, speculation enabled, and 3 tasks
> {noformat}
> HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar
> $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.vertex.max-task-concurrency=1
> -Dtez.am.speculation.enabled=true -Dtez.task.timeout-ms=6 -m 3 -mt 6
> -ir 0 -irt 0 -r 0 -rt 0
> {noformat}
> 2. Let the 1st task run to completion and then stop the 2nd task so that a
> speculative attempt is scheduled. Once the speculative attempt is scheduled
> for the 2nd task, continue the original attempt and let it complete.
> {noformat}
> kill -STOP
> // wait a few seconds for a speculative attempt to kick off
> kill -CONT
> {noformat}
> 3. Kill the 3rd task, which will create a 2nd attempt
> {noformat}
> kill -9
> {noformat}
> 4. The next thing to be drawn off of the queue will be the speculative
> attempt of the 2nd task. However, it is already completed, so it will just
> sit in the final state and the job will hang.
> Basically, for the failure to happen, the number of speculative tasks that
> are scheduled, but not yet ran has to be >= the concurrency of the job and
> there has to be at least 1 task failure.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986600#comment-15986600
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865327/TEZ-3700.2.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2396//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2396//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[
https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986278#comment-15986278
]
TezQA commented on TEZ-3700:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865298/TEZ-3700.1.patch
against master revision 247719d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2395//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2395//console
This message is automatically generated.
> Consumer attempt should kill itself instead of failing during validation
> checks with final merge avoidance
> --
>
> Key: TEZ-3700
> URL: https://issues.apache.org/jira/browse/TEZ-3700
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3700.1.patch
>
>
> Currently when if data is received from different attempts with final merge
> disabled (with/without pipleining), consumer attempt ends up with failure.
> Instead it should issue kill request so that the job deos not end up with
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3699) For large dataset, pipelined shuffle throws exceptions in consumer side for UnorderedPartitioned edge
[
https://issues.apache.org/jira/browse/TEZ-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984702#comment-15984702
]
TezQA commented on TEZ-3699:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865128/TEZ-3699.2.patch
against master revision db6f05f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2394//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2394//console
This message is automatically generated.
> For large dataset, pipelined shuffle throws exceptions in consumer side for
> UnorderedPartitioned edge
> -
>
> Key: TEZ-3699
> URL: https://issues.apache.org/jira/browse/TEZ-3699
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3699.1.patch, TEZ-3699.2.patch
>
>
> {noformat}
> 2017-04-25 21:16:18,179 [INFO] [Fetcher_B {Map_1} #1] |ShuffleManager.fetch|:
> Completed fetch for attempt: {110, 0,
> attempt_1490656001509_1098_1_04_000110_0_10013_4, 2, 4} to MEMORY,
> csize=10505, dsize=32638, EndTime=1493180178179, TimeTaken=1, Rate=10.01 MB/s
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |HttpConnection.url|:
> for
> url=http://node120:13562/mapOutput?job=job_1490656001509_1098=1=43=attempt_1490656001509_1098_1_04_75_0_10008_15,attempt_1490656001509_1098_1_04_35_0_10008_16,attempt_1490656001509_1098_1_04_
> 55_0_10007_16,attempt_1490656001509_1098_1_04_55_0_10007_16,attempt_1490656001509_1098_1_04_75_0_10008_16,attempt_1490656001509_1098_1_04_000195_0_10013_1,attempt_1490656001509_1098_1_04_75_0_10008_17,attempt_1490656001509_1098_1_04_000180_0_10013_1,attempt_1490656001509_1098_1_04_000195_0_10013_2,att
> empt_1490656001509_1098_1_04_75_0_10008_18,attempt_1490656001509_1098_1_04_000180_0_10013_2,attempt_1490656001509_1098_1_04_000195_0_10013_3=true
> sent hash and receievd reply 0 ms
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |ShuffleManager.fetch|:
> Completed fetch for attempt: {75, 0,
> attempt_1490656001509_1098_1_04_75_0_10008_15, 1, 15} to MEMORY,
> csize=23437, dsize=70987, EndTime=1493180178183, TimeTaken=0, Rate=0.00 MB/s
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |shuffle.Fetcher|:
> Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=35,
> attemptNumber=0,
> pathComponent=attempt_1490656001509_1098_1_04_35_0_10008_16, spillType=1,
> spillId=16]. len=0, decomp=0. ExceptionMessage=Not a valid ifile header
> 2017-04-25 21:16:18,185 [WARN] [Fetcher_B {Map_1} #2] |shuffle.Fetcher|:
> Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=35,
> attemptNumber=0,
> pathComponent=attempt_1490656001509_1098_1_04_35_0_10008_16, spillType=1,
> spillId=16] from cn120-10.l42scl.hortonworks.com
> java.io.IOException: Not a valid ifile header
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:833)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:840)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:608)
> at
> org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:134)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:814)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:539)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:428)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:226)
> at
>
[jira] [Commented] (TEZ-3699) For large dataset, pipelined shuffle throws exceptions in consumer side for UnorderedPartitioned edge
[
https://issues.apache.org/jira/browse/TEZ-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984567#comment-15984567
]
TezQA commented on TEZ-3699:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12865081/TEZ-3699.1.patch
against master revision db6f05f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2393//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2393//console
This message is automatically generated.
> For large dataset, pipelined shuffle throws exceptions in consumer side for
> UnorderedPartitioned edge
> -
>
> Key: TEZ-3699
> URL: https://issues.apache.org/jira/browse/TEZ-3699
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3699.1.patch
>
>
> {noformat}
> 2017-04-25 21:16:18,179 [INFO] [Fetcher_B {Map_1} #1] |ShuffleManager.fetch|:
> Completed fetch for attempt: {110, 0,
> attempt_1490656001509_1098_1_04_000110_0_10013_4, 2, 4} to MEMORY,
> csize=10505, dsize=32638, EndTime=1493180178179, TimeTaken=1, Rate=10.01 MB/s
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |HttpConnection.url|:
> for
> url=http://node120:13562/mapOutput?job=job_1490656001509_1098=1=43=attempt_1490656001509_1098_1_04_75_0_10008_15,attempt_1490656001509_1098_1_04_35_0_10008_16,attempt_1490656001509_1098_1_04_
> 55_0_10007_16,attempt_1490656001509_1098_1_04_55_0_10007_16,attempt_1490656001509_1098_1_04_75_0_10008_16,attempt_1490656001509_1098_1_04_000195_0_10013_1,attempt_1490656001509_1098_1_04_75_0_10008_17,attempt_1490656001509_1098_1_04_000180_0_10013_1,attempt_1490656001509_1098_1_04_000195_0_10013_2,att
> empt_1490656001509_1098_1_04_75_0_10008_18,attempt_1490656001509_1098_1_04_000180_0_10013_2,attempt_1490656001509_1098_1_04_000195_0_10013_3=true
> sent hash and receievd reply 0 ms
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |ShuffleManager.fetch|:
> Completed fetch for attempt: {75, 0,
> attempt_1490656001509_1098_1_04_75_0_10008_15, 1, 15} to MEMORY,
> csize=23437, dsize=70987, EndTime=1493180178183, TimeTaken=0, Rate=0.00 MB/s
> 2017-04-25 21:16:18,183 [INFO] [Fetcher_B {Map_1} #2] |shuffle.Fetcher|:
> Failed to read data to memory for InputAttemptIdentifier [inputIdentifier=35,
> attemptNumber=0,
> pathComponent=attempt_1490656001509_1098_1_04_35_0_10008_16, spillType=1,
> spillId=16]. len=0, decomp=0. ExceptionMessage=Not a valid ifile header
> 2017-04-25 21:16:18,185 [WARN] [Fetcher_B {Map_1} #2] |shuffle.Fetcher|:
> Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=35,
> attemptNumber=0,
> pathComponent=attempt_1490656001509_1098_1_04_35_0_10008_16, spillType=1,
> spillId=16] from cn120-10.l42scl.hortonworks.com
> java.io.IOException: Not a valid ifile header
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.verifyHeaderMagic(IFile.java:833)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.isCompressedFlagEnabled(IFile.java:840)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:608)
> at
> org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:134)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:814)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:539)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:428)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:226)
>
[jira] [Commented] (TEZ-3697) Adding #output_record in vertex manager event payload
[
https://issues.apache.org/jira/browse/TEZ-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983988#comment-15983988
]
TezQA commented on TEZ-3697:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864827/TEZ-3697.1.patch
against master revision db6f05f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2392//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2392//console
This message is automatically generated.
> Adding #output_record in vertex manager event payload
> -
>
> Key: TEZ-3697
> URL: https://issues.apache.org/jira/browse/TEZ-3697
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3697.1.patch
>
>
> This enables estimation of total number of records generated by source
> vertex. The overhead is one more integer per payload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3698) UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
[
https://issues.apache.org/jira/browse/TEZ-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982731#comment-15982731
]
TezQA commented on TEZ-3698:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864925/TEZ-3698.1.patch
against master revision ad8a80d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.output.TestOnFileUnorderedKVOutput
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2391//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2391//console
This message is automatically generated.
> UnorderedKV writer should be able to honor
> tez.runtime.enable.final-merge.in.output without pipelinedshuffle
>
>
> Key: TEZ-3698
> URL: https://issues.apache.org/jira/browse/TEZ-3698
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3698.1.patch
>
>
> Final merge can be disabled with "tez.runtime.enable.final-merge.in.output"
> setting. Currently this works with UnorderedKV writer only with pipelined
> shuffle. It should be able to honor this parameter, without pipelined shuffle
> as well to avoid final merge.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3695) TestTezSharedExecutor fails sporadically
[
https://issues.apache.org/jira/browse/TEZ-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981842#comment-15981842
]
TezQA commented on TEZ-3695:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864771/TEZ-3695.001.patch
against master revision bb4fb64.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2390//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2390//console
This message is automatically generated.
> TestTezSharedExecutor fails sporadically
>
>
> Key: TEZ-3695
> URL: https://issues.apache.org/jira/browse/TEZ-3695
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3695.001.patch
>
>
> TestTezSharedExecutor#testSerialExecution is timing out more often than not
> for me when running the full TezTezSharedExecutor test suite.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3694) Adopt YARN-5007 in MiniTezCluster
[
https://issues.apache.org/jira/browse/TEZ-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979884#comment-15979884
]
TezQA commented on TEZ-3694:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864595/TEZ-3694.1.patch
against master revision bb4fb64.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.analyzer.TestAnalyzer
org.apache.tez.history.TestHistoryParser
org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2389//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2389//console
This message is automatically generated.
> Adopt YARN-5007 in MiniTezCluster
> -
>
> Key: TEZ-3694
> URL: https://issues.apache.org/jira/browse/TEZ-3694
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: 0.9.0
>
> Attachments: TEZ-3694.1.patch
>
>
> Master branch won't build on hadoop trunk because YARN-5007 removed enableAHS
> param from MiniYarnCluster ctor, which breaks MiniTezCluster. We should adopt
> the change and use config to enable timeline service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-2049) Remove YARN references from Tez AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979874#comment-15979874
]
TezQA commented on TEZ-2049:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864596/TEZ-2049.1.patch
against master revision bb4fb64.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.dag.impl.TestDAGImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2388//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2388//console
This message is automatically generated.
> Remove YARN references from Tez AsyncDispatcher
> ---
>
> Key: TEZ-2049
> URL: https://issues.apache.org/jira/browse/TEZ-2049
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Zhiyuan Yang
> Attachments: TEZ-2049.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3690) Tez on hadoop 3 build failed due to hdfs client/server jar separation
[
https://issues.apache.org/jira/browse/TEZ-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979290#comment-15979290
]
TezQA commented on TEZ-3690:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864545/TEZ-3690.2.patch
against master revision ca607d6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2387//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2387//console
This message is automatically generated.
> Tez on hadoop 3 build failed due to hdfs client/server jar separation
> -
>
> Key: TEZ-3690
> URL: https://issues.apache.org/jira/browse/TEZ-3690
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: TEZ-3690.2.patch, TEZ-3690.patch
>
>
> Build TEZ master branch against hadoop 3 alpha 3 (trunk branch), and found
> issues:
> {noformat}
> ...
> api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> {noformat}
> This is due to HDFS-6200 that we are separating client and server jars. We
> should fix downstream projects for this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3675) Handle changes to ResourceCalculatorProcessTree in YARN-3427 for Hadoop 3.x
[
https://issues.apache.org/jira/browse/TEZ-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977895#comment-15977895
]
TezQA commented on TEZ-3675:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864328/TEZ-3675.1.patch
against master revision ca607d6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2386//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2386//console
This message is automatically generated.
> Handle changes to ResourceCalculatorProcessTree in YARN-3427 for Hadoop 3.x
> ---
>
> Key: TEZ-3675
> URL: https://issues.apache.org/jira/browse/TEZ-3675
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3675.1.patch
>
>
> Removing some APIs. We'll likely need to shim this class.
> Ideally, create the shim for 2.6 to use the old methods, and everything else
> uses the new methods.
> Alternately, change the min hadoop version to 2.7
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3689) Change minimum hadoop version to 2.7.0
[
https://issues.apache.org/jira/browse/TEZ-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977502#comment-15977502
]
TezQA commented on TEZ-3689:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864348/TEZ-3689.02.patch
against master revision e9d0b1b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 38 javac
compiler warnings (more than the master's current 32 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2385//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2385//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2385//console
This message is automatically generated.
> Change minimum hadoop version to 2.7.0
> --
>
> Key: TEZ-3689
> URL: https://issues.apache.org/jira/browse/TEZ-3689
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3689.01.patch, TEZ-3689.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3687) Code smell in equals implementation
[
https://issues.apache.org/jira/browse/TEZ-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976433#comment-15976433
]
TezQA commented on TEZ-3687:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864216/TEZ-3687.01.patch
against master revision fb0e45b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2384//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2384//console
This message is automatically generated.
> Code smell in equals implementation
> ---
>
> Key: TEZ-3687
> URL: https://issues.apache.org/jira/browse/TEZ-3687
> Project: Apache Tez
> Issue Type: Bug
>Reporter: JC
>Assignee: Akira Ajisaka
>Priority: Trivial
> Attachments: TEZ-3687.01.patch
>
>
> Hi
> In a recent snapshot mirrored from GitHub (tez), I've found code smells that
> may case a bug.
> Path: tez-api/src/main/java/org/apache/tez/dag/api/client/DAGStatus.java
> {code}
> 161 public boolean equals(Object obj) {
> 162 // Source explicitly exclude from equals
> 163 if (obj instanceof DAGStatus){
> 164 DAGStatus other = (DAGStatus)obj;
> 165 return getState() == other.getState()
> 166 && getDiagnostics().equals(other.getDiagnostics())
> 167 && getDAGProgress().equals(getDAGProgress())
> 168 && getVertexProgress().equals(getVertexProgress())
> 169 &&
> 170 ((getDAGCounters() == null && other.getDAGCounters() == null)
> 171 || getDAGCounters().equals(other.getDAGCounters()));
> 172 }
> 173 return false;
> 174 }
> {code}
> In Line 167 and 168, the conditions should be
> getDAGProgress().equals(other.getDAGProgress())
> getVertexProgress().equals(other.getVertexProgress())
> respectively?
> Path: tez-api/src/main/java/org/apache/tez/dag/api/client/VertexStatus.java
> {code}
> 119 public boolean equals(Object obj) {
> 120 if (obj instanceof VertexStatus){
> 121 VertexStatus other = (VertexStatus)obj;
> 122 return getState().equals(other.getState())
> 123 && getDiagnostics().equals(getDiagnostics())
> 124 && getProgress().equals(getProgress())
> 125 &&
> 126 ((getVertexCounters() == null && other.getVertexCounters() ==
> null)
> 127 || getVertexCounters().equals(other.getVertexCounters()));
> 128 }
> 129 return false;
> 130 }
> {code}
> in In Line 123 and 124, the conditions should be
> getDiagnostics().equals(other.getDiagnostics())
> getProgress().equals(other.getProgress())
> respectively?
> These might be trivial things but wanted to report just in case. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976239#comment-15976239
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864206/TEZ-3680.7.patch
against master revision fb0e45b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestTaskSchedulerManager
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2383//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2383//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch, TEZ-3680.6.patch,
> TEZ-3680.7.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3690) Tez on hadoop 3 build failed due to hdfs client/server jar separation
[
https://issues.apache.org/jira/browse/TEZ-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975897#comment-15975897
]
TezQA commented on TEZ-3690:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864175/TEZ-3690.patch
against master revision fb0e45b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2382//console
This message is automatically generated.
> Tez on hadoop 3 build failed due to hdfs client/server jar separation
> -
>
> Key: TEZ-3690
> URL: https://issues.apache.org/jira/browse/TEZ-3690
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: TEZ-3690.patch
>
>
> Build TEZ master branch against hadoop 3 alpha 3 (trunk branch), and found
> issues:
> {noformat}
> ...
> api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> {noformat}
> This is due to HDFS-6200 that we are separating client and server jars. We
> should fix downstream projects for this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3689) Change minimum hadoop version to 2.7.0
[
https://issues.apache.org/jira/browse/TEZ-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975719#comment-15975719
]
TezQA commented on TEZ-3689:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864087/TEZ-3689.01.patch
against master revision fb0e45b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 38 javac
compiler warnings (more than the master's current 32 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2381//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2381//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2381//console
This message is automatically generated.
> Change minimum hadoop version to 2.7.0
> --
>
> Key: TEZ-3689
> URL: https://issues.apache.org/jira/browse/TEZ-3689
> Project: Apache Tez
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3689.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3631) Tez UI: TEZ_DAG_EXTRA_INFO compatibility changes - Makes All DAGs page faster.
[
https://issues.apache.org/jira/browse/TEZ-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975017#comment-15975017
]
TezQA commented on TEZ-3631:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864026/TEZ-3631.1.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2380//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2380//console
This message is automatically generated.
> Tez UI: TEZ_DAG_EXTRA_INFO compatibility changes - Makes All DAGs page faster.
> --
>
> Key: TEZ-3631
> URL: https://issues.apache.org/jira/browse/TEZ-3631
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3631.1.patch
>
>
> TEZ-3611 will be moving dagPlan and counters into a new entity of type
> TEZ_DAG_EXTRA_INFO. Thus TEZ_DAG_ID entity would be lighter and would load
> faster.
> This ticket is to ensure that the UI works as expected with the new changes
> and is backward compatible.
> Need to consider the following cases:
> 1. TEZ_DAG_EXTRA_INFO must be loaded only from the inner pages and not in All
> DAGs page
> 2. Add TEZ_DAG_EXTRA_INFO must be part of Dag data zip
> 3. Caller Contex should be taken from the DAG JSON and not from DAG Plan
> 4. Display a button in All DAGs table for the user to load counter data if
> required. Display a tooltip saying that this operation would be costly.
> 5. Maintain backward compatibility
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3274) Vertex with MRInput and shuffle input does not respect slow start
[
https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974874#comment-15974874
]
TezQA commented on TEZ-3274:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864015/TEZ-3274.003.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.dag.impl.TestVertexImpl
org.apache.tez.dag.library.vertexmanager.TestFairShuffleVertexManager
org.apache.tez.dag.library.vertexmanager.TestShuffleVertexManagerBase
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2379//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2379//console
This message is automatically generated.
> Vertex with MRInput and shuffle input does not respect slow start
> -
>
> Key: TEZ-3274
> URL: https://issues.apache.org/jira/browse/TEZ-3274
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Eric Badger
> Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch,
> TEZ-3274.003.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and
> not ShuffleVertexManager) and start containers and tasks immediately. In this
> scenario, resources can be wasted since they do not respect
> tez.shuffle-vertex-manager.min-src-fraction
> tez.shuffle-vertex-manager.max-src-fraction.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks
[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974718#comment-15974718
]
TezQA commented on TEZ-1187:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12864004/TEZ-1187.09.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2378//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2378//console
This message is automatically generated.
> Share Thread pools between different tasks
> --
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.02.patch, TEZ-1187.03.patch, TEZ-1187.04.patch,
> TEZ-1187.05.patch, TEZ-1187.06.patch, TEZ-1187.07.patch, TEZ-1187.08.patch,
> TEZ-1187.09.patch, TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks
[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974459#comment-15974459
]
TezQA commented on TEZ-1187:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863986/TEZ-1187.08.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 32 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2377//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2377//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2377//console
This message is automatically generated.
> Share Thread pools between different tasks
> --
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.02.patch, TEZ-1187.03.patch, TEZ-1187.04.patch,
> TEZ-1187.05.patch, TEZ-1187.06.patch, TEZ-1187.07.patch, TEZ-1187.08.patch,
> TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3688) Add analyzer to provide the set of machines that participated in the vertex and the number of tasks that ran on that node
[
https://issues.apache.org/jira/browse/TEZ-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974441#comment-15974441
]
TezQA commented on TEZ-3688:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863984/TEZ-3688.2.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
See
https://builds.apache.org/job/PreCommit-TEZ-Build/2376//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2376//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2376//artifact/patchprocess/newPatchFindbugsWarningsjob-analyzer.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2376//console
This message is automatically generated.
> Add analyzer to provide the set of machines that participated in the vertex
> and the number of tasks that ran on that node
> -
>
> Key: TEZ-3688
> URL: https://issues.apache.org/jira/browse/TEZ-3688
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Nita Dembla
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3688.1.patch, TEZ-3688.2.patch
>
>
> Need an analyzer to parse DAG info and outputs the how many tasks of a vertex
> ran on a node. This would help in isolating task scheduling issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3688) Add analyzer to provide the set of machines that participated in the vertex and the number of tasks that ran on that node
[
https://issues.apache.org/jira/browse/TEZ-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974335#comment-15974335
]
TezQA commented on TEZ-3688:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863969/TEZ-3688.1.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2375//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2375//artifact/patchprocess/newPatchFindbugsWarningsjob-analyzer.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2375//console
This message is automatically generated.
> Add analyzer to provide the set of machines that participated in the vertex
> and the number of tasks that ran on that node
> -
>
> Key: TEZ-3688
> URL: https://issues.apache.org/jira/browse/TEZ-3688
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Nita Dembla
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3688.1.patch
>
>
> Need an analyzer to parse DAG info and outputs the how many tasks of a vertex
> ran on a node. This would help in isolating task scheduling issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974181#comment-15974181
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863957/TEZ-3680.6.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2374//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2374//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch, TEZ-3680.6.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967168#comment-15967168
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863204/TEZ-3680.5.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2373//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2373//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks
[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965891#comment-15965891
]
TezQA commented on TEZ-1187:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12863019/TEZ-1187.07.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2372//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2372//console
This message is automatically generated.
> Share Thread pools between different tasks
> --
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.02.patch, TEZ-1187.03.patch, TEZ-1187.04.patch,
> TEZ-1187.05.patch, TEZ-1187.06.patch, TEZ-1187.07.patch, TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks
[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965332#comment-15965332
]
TezQA commented on TEZ-1187:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862963/TEZ-1187.06.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2371//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2371//console
This message is automatically generated.
> Share Thread pools between different tasks
> --
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.02.patch, TEZ-1187.03.patch, TEZ-1187.04.patch,
> TEZ-1187.05.patch, TEZ-1187.06.patch, TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3611) Create lightweight summary events for ATS.
[
https://issues.apache.org/jira/browse/TEZ-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964226#comment-15964226
]
TezQA commented on TEZ-3611:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862804/TEZ-3611.06.patch
against master revision f355a05.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2370//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2370//console
This message is automatically generated.
> Create lightweight summary events for ATS.
> --
>
> Key: TEZ-3611
> URL: https://issues.apache.org/jira/browse/TEZ-3611
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3611.01.patch, TEZ-3611.02.patch, TEZ-3611.03.patch,
> TEZ-3611.04.patch, TEZ-3611.05.patch, TEZ-3611.06.patch
>
>
> All DAG*Event are summary logs. DAGSubmittedEvent has dagPlan in it and
> DAGFinishedEvent has counters in it. And hence these events very large, which
> makes the dag landing page on the tez-ui very expensive.
> Split these events into two parts:
> * One which contains only data used by the tez-ui "All Dags" page which will
> go into summary log.
> * Another which will contain the current object, which will be part of the
> appropriate TimelineEntityGroupId.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960428#comment-15960428
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862439/TEZ-3680.4.patch
against master revision 04f0c76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2369//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2369//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch, TEZ-3680.4.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960331#comment-15960331
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862431/TEZ-3680.3.patch
against master revision 04f0c76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2368//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2368//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958516#comment-15958516
]
TezQA commented on TEZ-3680:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862245/TEZ-3680.2.patch
against master revision 04f0c76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2367//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2367//console
This message is automatically generated.
> Optimizations to UnorderedPartitionedKVWriter
> -
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3285) Tez UI: Lock down dependency versions
[
https://issues.apache.org/jira/browse/TEZ-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957109#comment-15957109
]
TezQA commented on TEZ-3285:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12862087/TEZ-3285.6.patch
against master revision d0b189a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2366//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2366//console
This message is automatically generated.
> Tez UI: Lock down dependency versions
> -
>
> Key: TEZ-3285
> URL: https://issues.apache.org/jira/browse/TEZ-3285
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3285.1.patch, TEZ-3285.2.patch, TEZ-3285.3.patch,
> TEZ-3285.4.patch, TEZ-3285.5.patch, TEZ-3285.6.patch,
> TEZ-3285_batch-0.8_1.patch
>
>
> All dependencies of tez-ui is having fixed versions. But the dependencies of
> our dependencies are not. Hence a level down in the dependency tree, the
> build might be looking for the latest packages in every build. This affects
> the reliability of the UI build. This must be fixed in both the package
> managers used by Tez UI - NPM & Bower.
> -NPM:-
> -npm shrinkwrap create a separate JSON from the currently installed packages,
> and ensure that the complete dependency tree is intact across all the build.-
> Yarn:
> Replace NPM with Yarn. Yarn is a package manager developed to solve this
> issue and many more.
> Bower:
> Bower shrinkwrap resolver plugin can be used to lock the dependency versions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3681) Improve UI error message while trying to sort running DAGs with Auto Refresh enabled
[
https://issues.apache.org/jira/browse/TEZ-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956100#comment-15956100
]
TezQA commented on TEZ-3681:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861971/TEZ-3681.001.patch
against master revision d0b189a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2365//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2365//console
This message is automatically generated.
> Improve UI error message while trying to sort running DAGs with Auto Refresh
> enabled
>
>
> Key: TEZ-3681
> URL: https://issues.apache.org/jira/browse/TEZ-3681
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: TEZ-3681.001.patch
>
>
> {noformat}
> Cannot sort:
> Sorting on Status is disabled for running DAGs!
> {noformat}
> The current message is incorrect. You can sort by status on running DAGs,
> just not while Auto Refresh is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3654) CartesianProduct edge won't work with GroupInputEdge
[
https://issues.apache.org/jira/browse/TEZ-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956002#comment-15956002
]
TezQA commented on TEZ-3654:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861958/TEZ-3654.4.patch
against master revision d0b189a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2364//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2364//console
This message is automatically generated.
> CartesianProduct edge won't work with GroupInputEdge
>
>
> Key: TEZ-3654
> URL: https://issues.apache.org/jira/browse/TEZ-3654
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3654.1.patch, TEZ-3654.2.patch, TEZ-3654.3.patch,
> TEZ-3654.4.patch
>
>
> If a vertex group is used as source of cartesian product, it expands into
> multiple vertices that share the same edge properties and each CP edge is
> taken as individual cartesian product source by CP vertex manager. CP vertex
> manager will find there are more CP edge than expected and abort the AM.
> Ideally group edge semantic should be fixed: both task and vertex manager
> should see same number of sources; also not every edge can be simply
> duplicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3679) remove the redundant words in UnorderedKVInput.java
[
https://issues.apache.org/jira/browse/TEZ-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952129#comment-15952129
]
TezQA commented on TEZ-3679:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861584/TEZ-3679.2.patch
against master revision e09ea27.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2363//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2363//console
This message is automatically generated.
> remove the redundant words in UnorderedKVInput.java
> ---
>
> Key: TEZ-3679
> URL: https://issues.apache.org/jira/browse/TEZ-3679
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0, 0.8.6
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: TEZ-3679.1.patch, TEZ-3679.2.patch
>
>
> In UnorderedKVInput.java,i found the redundant words "git diff" in the head.I
> guess someone left it.Then I remove it to make the license head correct
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3616) TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952090#comment-15952090
]
TezQA commented on TEZ-3616:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861578/TEZ-3616.003.patch
against master revision e09ea27.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2362//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2362//console
This message is automatically generated.
> TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
> --
>
> Key: TEZ-3616
> URL: https://issues.apache.org/jira/browse/TEZ-3616
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
> Environment: Ubuntu 14.04
>Reporter: Sonia Garudi
>Assignee: Fei Hui
> Labels: ppc64le, x86
> Attachments: TEZ-3616.001.patch, TEZ-3616.002.patch,
> TEZ-3616.003.patch
>
>
> In tez-runtime-library project, the
> TestMergeManager#testLocalDiskMergeMultipleTasks test fails intermittently
> with the following error:
> testLocalDiskMergeMultipleTasks(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager)
> Time elapsed: 1.395 sec <<< FAILURE!
> java.lang.AssertionError: Values should be different. Actual: 1
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at org.junit.Assert.assertNotEquals(Assert.java:198)
> at org.junit.Assert.assertNotEquals(Assert.java:209)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager.testLocalDiskMergeMultipleTasks(TestMergeManager.java:878)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager.testLocalDiskMergeMultipleTasks(TestMergeManager.java:628)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3285) Tez UI: Lock down dependency versions
[
https://issues.apache.org/jira/browse/TEZ-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952006#comment-15952006
]
TezQA commented on TEZ-3285:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861566/TEZ-3285.5.patch
against master revision e09ea27.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2361//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2361//console
This message is automatically generated.
> Tez UI: Lock down dependency versions
> -
>
> Key: TEZ-3285
> URL: https://issues.apache.org/jira/browse/TEZ-3285
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3285.1.patch, TEZ-3285.2.patch, TEZ-3285.3.patch,
> TEZ-3285.4.patch, TEZ-3285.5.patch, TEZ-3285_batch-0.8_1.patch
>
>
> All dependencies of tez-ui is having fixed versions. But the dependencies of
> our dependencies are not. Hence a level down in the dependency tree, the
> build might be looking for the latest packages in every build. This affects
> the reliability of the UI build. This must be fixed in both the package
> managers used by Tez UI - NPM & Bower.
> -NPM:-
> -npm shrinkwrap create a separate JSON from the currently installed packages,
> and ensure that the complete dependency tree is intact across all the build.-
> Yarn:
> Replace NPM with Yarn. Yarn is a package manager developed to solve this
> issue and many more.
> Bower:
> Bower shrinkwrap resolver plugin can be used to lock the dependency versions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3285) Tez UI: Lock down dependency versions
[
https://issues.apache.org/jira/browse/TEZ-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951681#comment-15951681
]
TezQA commented on TEZ-3285:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861497/TEZ-3285.4.patch
against master revision e09ea27.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:red}-1 release audit{color}. The applied patch generated 2
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2360//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2360//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2360//console
This message is automatically generated.
> Tez UI: Lock down dependency versions
> -
>
> Key: TEZ-3285
> URL: https://issues.apache.org/jira/browse/TEZ-3285
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3285.1.patch, TEZ-3285.2.patch, TEZ-3285.3.patch,
> TEZ-3285.4.patch, TEZ-3285_batch-0.8_1.patch
>
>
> All dependencies of tez-ui is having fixed versions. But the dependencies of
> our dependencies are not. Hence a level down in the dependency tree, the
> build might be looking for the latest packages in every build. This affects
> the reliability of the UI build. This must be fixed in both the package
> managers used by Tez UI - NPM & Bower.
> -NPM:-
> -npm shrinkwrap create a separate JSON from the currently installed packages,
> and ensure that the complete dependency tree is intact across all the build.-
> Yarn:
> Replace NPM with Yarn. Yarn is a package manager developed to solve this
> issue and many more.
> Bower:
> Bower shrinkwrap resolver plugin can be used to lock the dependency versions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3679) remove the redundant words in UnorderedKVInput.java
[
https://issues.apache.org/jira/browse/TEZ-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950757#comment-15950757
]
TezQA commented on TEZ-3679:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861423/TEZ-3679.1.patch
against master revision 750f8f7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2359//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2359//console
This message is automatically generated.
> remove the redundant words in UnorderedKVInput.java
> ---
>
> Key: TEZ-3679
> URL: https://issues.apache.org/jira/browse/TEZ-3679
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0, 0.8.6
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: TEZ-3679.1.patch
>
>
> In UnorderedKVInput.java,i found the redundant words "git diff" in the head.I
> guess someone left it.Then I remove it to make the license head correct
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3678) The command "hadoop dfs" should be replaced by "hadoop fs" in install markdown
[
https://issues.apache.org/jira/browse/TEZ-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950310#comment-15950310
]
TezQA commented on TEZ-3678:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861372/TEZ-3678.1.patch
against master revision 750f8f7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2358//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2358//console
This message is automatically generated.
> The command "hadoop dfs" should be replaced by "hadoop fs" in install
> markdown
> ---
>
> Key: TEZ-3678
> URL: https://issues.apache.org/jira/browse/TEZ-3678
> Project: Apache Tez
> Issue Type: Improvement
>Affects Versions: 0.9.0, 0.8.6
>Reporter: Saijin Huang
>Assignee: Saijin Huang
> Attachments: TEZ-3678.1.patch
>
>
> When i read the install guide for tez,i found step 3 still use the command
> "hadoop dfs" which is deprecated in hdfs.Also,In the same page,the command
> "hadoop fs" always was used.In order to keep consistent with the other
> sections in the same page, i suggest that the command "hadoop dfs" should be
> replaced by "hadoop fs" in install markdown.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3677) add the symbol "<" to annotate the license head
[
https://issues.apache.org/jira/browse/TEZ-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948431#comment-15948431
]
TezQA commented on TEZ-3677:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12861156/TEZ-3677.1.patch
against master revision 906c2a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2357//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2357//console
This message is automatically generated.
> add the symbol "<" to annotate the license head
> ---
>
> Key: TEZ-3677
> URL: https://issues.apache.org/jira/browse/TEZ-3677
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0, 0.8.6
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: TEZ-3677.1.patch
>
>
> In by-laws.md,i found the license head leak a symbol "<" which leads to fail
> to annotate the head.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3521) LogicalIOProcessorRuntimeTask should use edge name to index IOs
[
https://issues.apache.org/jira/browse/TEZ-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945875#comment-15945875
]
TezQA commented on TEZ-3521:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860922/TEZ-3521.2.patch
against master revision 906c2a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2356//console
This message is automatically generated.
> LogicalIOProcessorRuntimeTask should use edge name to index IOs
> ---
>
> Key: TEZ-3521
> URL: https://issues.apache.org/jira/browse/TEZ-3521
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Fix For: TEZ-1190
>
> Attachments: TEZ-3521.1.patch, TEZ-3521.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3654) CartesianProduct edge won't work with GroupInputEdge
[
https://issues.apache.org/jira/browse/TEZ-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945773#comment-15945773
]
TezQA commented on TEZ-3654:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860908/TEZ-3654.3.patch
against master revision 906c2a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2355//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2355//console
This message is automatically generated.
> CartesianProduct edge won't work with GroupInputEdge
>
>
> Key: TEZ-3654
> URL: https://issues.apache.org/jira/browse/TEZ-3654
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3654.1.patch, TEZ-3654.2.patch, TEZ-3654.3.patch
>
>
> If a vertex group is used as source of cartesian product, it expands into
> multiple vertices that share the same edge properties and each CP edge is
> taken as individual cartesian product source by CP vertex manager. CP vertex
> manager will find there are more CP edge than expected and abort the AM.
> Ideally group edge semantic should be fixed: both task and vertex manager
> should see same number of sources; also not every edge can be simply
> duplicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3672) remove the duplicate apache license head
[
https://issues.apache.org/jira/browse/TEZ-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943215#comment-15943215
]
TezQA commented on TEZ-3672:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860640/TEZ-3672.1.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2354//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2354//console
This message is automatically generated.
> remove the duplicate apache license head
> --
>
> Key: TEZ-3672
> URL: https://issues.apache.org/jira/browse/TEZ-3672
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0, 0.8.4, 0.8.5
>Reporter: Saijin Huang
>Priority: Trivial
> Attachments: TEZ-3672.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3671) move the apache license head to appropriate position for class TestCompositeDataMovementEvent
[
https://issues.apache.org/jira/browse/TEZ-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942932#comment-15942932
]
TezQA commented on TEZ-3671:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860615/TEZ-3671.1.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2353//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2353//console
This message is automatically generated.
> move the apache license head to appropriate position for class
> TestCompositeDataMovementEvent
> -
>
> Key: TEZ-3671
> URL: https://issues.apache.org/jira/browse/TEZ-3671
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Saijin Huang
>Priority: Minor
> Attachments: TEZ-3671.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3670) hive on tez throw FileNotFoundException,when enable hdfs federation
[
https://issues.apache.org/jira/browse/TEZ-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942784#comment-15942784
]
TezQA commented on TEZ-3670:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860599/TEZ-3670.1.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2352//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2352//console
This message is automatically generated.
> hive on tez throw FileNotFoundException,when enable hdfs federation
>
>
> Key: TEZ-3670
> URL: https://issues.apache.org/jira/browse/TEZ-3670
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Saijin Huang
>
> hive version is apache hive-2.1.0 tez version is apache tez-0.8.4 i set
> tez.lib.uris=viewfs://nsX/nameservice/hive/lib/tez-0.8.4.tar.gz
> when i run a tez job on hive ,it always appears the problem
> {code}
> 2017-03-23T15:26:24,255 INFO client.TezClientUtils (:()) - Using
> tez.lib.uris value from configuration:
> viewfs://nsX/nameservice/hive/lib/tez-0.8.4.tar.gz
> 2017-03-23T15:26:24,255 INFO client.TezClientUtils (:()) - Using
> tez.lib.uris.classpath value from configuration:
> $HADOOP_CONF_DIR,/opt/ZDH/parcels/lib/hadoop/*,/opt/ZDH/parcels/lib/hadoop/lib/*,/opt/ZDH/parcels/lib/hadoop-hdfs/*,/opt/ZDH/parcels/lib/hadoop-hdfs/lib/*,/opt/ZDH/parcels/lib/hadoop-yarn/*,/opt/ZDH/parcels/lib/hadoop-yarn/lib/*,/opt/ZDH/parcels/lib/hadoop-mapreduce/*,/opt/ZDH/parcels/lib/hadoop-mapreduce/lib/*,/opt/ZDH/parcels/lib/tez/*,/opt/ZDH/parcels/lib/tez/lib/*
> 2017-03-23T15:26:24,260 ERROR exec.Task (:()) - Failed to execute tez graph.
> java.io.FileNotFoundException: /hive
> at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:402)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:377)
> at
> org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:1031)
> at
> org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:275)
> at
> org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)
> at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)
> at org.apache.tez.client.TezClient.start(TezClient.java:447)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:390)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:320)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:611)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:203)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:277)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1917)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1586)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1331)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1092)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1080)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
>
[jira] [Commented] (TEZ-3666) Integer overflow in ShuffleVertexManagerBase
[
https://issues.apache.org/jira/browse/TEZ-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942762#comment-15942762
]
TezQA commented on TEZ-3666:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860597/TEZ-3666.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestTezJobs
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2351//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2351//console
This message is automatically generated.
> Integer overflow in ShuffleVertexManagerBase
>
>
> Key: TEZ-3666
> URL: https://issues.apache.org/jira/browse/TEZ-3666
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: TEZ-3666.patch
>
>
> In function getExpectedStatsInAtIndex, {{statsInMB[index] * numTasks /
> numVMEventsReceived}} could cause Integer overflow, for example when
> statsInMB[index] == 3 and numTasks == 20.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3616) TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942242#comment-15942242
]
TezQA commented on TEZ-3616:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860560/TEZ-3616.002.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2350//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2350//console
This message is automatically generated.
> TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
> --
>
> Key: TEZ-3616
> URL: https://issues.apache.org/jira/browse/TEZ-3616
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
> Environment: Ubuntu 14.04
>Reporter: Sonia Garudi
>Assignee: Fei Hui
> Labels: ppc64le, x86
> Attachments: TEZ-3616.001.patch, TEZ-3616.002.patch
>
>
> In tez-runtime-library project, the
> TestMergeManager#testLocalDiskMergeMultipleTasks test fails intermittently
> with the following error:
> testLocalDiskMergeMultipleTasks(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager)
> Time elapsed: 1.395 sec <<< FAILURE!
> java.lang.AssertionError: Values should be different. Actual: 1
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at org.junit.Assert.assertNotEquals(Assert.java:198)
> at org.junit.Assert.assertNotEquals(Assert.java:209)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager.testLocalDiskMergeMultipleTasks(TestMergeManager.java:878)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager.testLocalDiskMergeMultipleTasks(TestMergeManager.java:628)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-1187) Share Thread pools between different tasks
[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941754#comment-15941754
]
TezQA commented on TEZ-1187:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860529/TEZ-1187.05.patch
against master revision e375b9d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2349//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2349//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2349//console
This message is automatically generated.
> Share Thread pools between different tasks
> --
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.02.patch, TEZ-1187.03.patch, TEZ-1187.04.patch,
> TEZ-1187.05.patch, TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3668) Explicitly include hadoop-mapreduce-client-shuffle for test in root pom
[
https://issues.apache.org/jira/browse/TEZ-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937761#comment-15937761
]
TezQA commented on TEZ-3668:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860074/TEZ-3668.2.patch
against master revision cbcb3a7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2348//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2348//console
This message is automatically generated.
> Explicitly include hadoop-mapreduce-client-shuffle for test in root pom
> ---
>
> Key: TEZ-3668
> URL: https://issues.apache.org/jira/browse/TEZ-3668
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3668.1.patch, TEZ-3668.2.patch
>
>
> Previously hadoop-mapreduce-client-shuffle was dependency of
> hadoop-mapreduce-client-jobclient, but this is no longer true after
> MAPREDUCE-6791. Now we need to explicitly include it for test code.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3652) Remove ShuffleClientMetrics
[
https://issues.apache.org/jira/browse/TEZ-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937581#comment-15937581
]
TezQA commented on TEZ-3652:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860048/TEZ-3652.4.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2345//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2345//console
This message is automatically generated.
> Remove ShuffleClientMetrics
> ---
>
> Key: TEZ-3652
> URL: https://issues.apache.org/jira/browse/TEZ-3652
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Labels: incompatibleChange
> Attachments: TEZ-3652.1.patch, TEZ-3652.2.patch, TEZ-3652.3.patch,
> TEZ-3652.4.patch
>
>
> This part of code was directly from MapReduce. It used deprecated metrics1
> which was removed in hadoop 3. This metrics is no longer useful in Tez and
> nearly no one is using it, so we can just remove it from our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3667) Stop using org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
[
https://issues.apache.org/jira/browse/TEZ-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937578#comment-15937578
]
TezQA commented on TEZ-3667:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860047/TEZ-3667.2.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestDAGRecovery2
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2346//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2346//console
This message is automatically generated.
> Stop using
> org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
>
>
> Key: TEZ-3667
> URL: https://issues.apache.org/jira/browse/TEZ-3667
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3667.1.patch, TEZ-3667.2.patch
>
>
> This was renamed in hadoop 3 somehow. We can just set the value by ourselves
> because all we need is that default 'false'.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3652) Remove ShuffleClientMetrics
[
https://issues.apache.org/jira/browse/TEZ-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937576#comment-15937576
]
TezQA commented on TEZ-3652:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860048/TEZ-3652.4.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2347//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2347//console
This message is automatically generated.
> Remove ShuffleClientMetrics
> ---
>
> Key: TEZ-3652
> URL: https://issues.apache.org/jira/browse/TEZ-3652
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Labels: incompatibleChange
> Attachments: TEZ-3652.1.patch, TEZ-3652.2.patch, TEZ-3652.3.patch,
> TEZ-3652.4.patch
>
>
> This part of code was directly from MapReduce. It used deprecated metrics1
> which was removed in hadoop 3. This metrics is no longer useful in Tez and
> nearly no one is using it, so we can just remove it from our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3652) Remove ShuffleClientMetrics
[
https://issues.apache.org/jira/browse/TEZ-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937361#comment-15937361
]
TezQA commented on TEZ-3652:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860015/TEZ-3652.3.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.sort.impl.TestPipelinedSorter
org.apache.tez.mapreduce.processor.map.TestMapProcessor
org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2342//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2342//console
This message is automatically generated.
> Remove ShuffleClientMetrics
> ---
>
> Key: TEZ-3652
> URL: https://issues.apache.org/jira/browse/TEZ-3652
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Labels: incompatibleChange
> Attachments: TEZ-3652.1.patch, TEZ-3652.2.patch, TEZ-3652.3.patch
>
>
> This part of code was directly from MapReduce. It used deprecated metrics1
> which was removed in hadoop 3. This metrics is no longer useful in Tez and
> nearly no one is using it, so we can just remove it from our codebase.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3667) Stop using org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
[
https://issues.apache.org/jira/browse/TEZ-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937358#comment-15937358
]
TezQA commented on TEZ-3667:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12860018/TEZ-3667.1.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.sort.impl.TestPipelinedSorter
The following test timeouts occurred in :
org.apache.tez.dag.library.veTests
org.apache.tez.runtime.common.resources.TestWeightedScalingMemoryDistributor
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVTests
org.apache.tez.maprTests
org.apache.tez.dag.history.logging.impl.TestHistoryEventJsonConversion
org.apache.tez.dag.history.TestHistoryEventType
org.apache.tez.dag.app.dag.impl.TesTests
org.apache.tez.dag.app.dag.impl.TestTests
org.apache.tez.dag.app.dag.impl.TestVertexImpl2
org.apache.tez.dag.app.TestMockDAGATests
org.apache.tez.dag.app.TestPTests
org.apache.tez.dag.app.rm.TestLocalTaskSchedulerSTests
org.apache.tez.dag.app.rm.TestContainerReusTeTests
org.apache.tez.history.TestHistoryParser
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2341//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2341//console
This message is automatically generated.
> Stop using
> org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
>
>
> Key: TEZ-3667
> URL: https://issues.apache.org/jira/browse/TEZ-3667
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3667.1.patch
>
>
> This was renamed in hadoop 3 somehow. We can just set the value by ourselves
> because all we need is that default 'false'.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3665) TestATSV15HistoryLoggingService should use mocked TimelineClient
[
https://issues.apache.org/jira/browse/TEZ-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936910#comment-15936910
]
TezQA commented on TEZ-3665:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12859964/TEZ.3665.2.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2340//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2340//console
This message is automatically generated.
> TestATSV15HistoryLoggingService should use mocked TimelineClient
>
>
> Key: TEZ-3665
> URL: https://issues.apache.org/jira/browse/TEZ-3665
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3665.1.patch, TEZ.3665.2.patch
>
>
> Currently it's extending TimelineClient and won't build on hadoop3. WIth
> mocked TimelineClient, we will be able to build against both hadoop2 and
> hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3665) TestATSV15HistoryLoggingService should use mocked TimelineClient
[
https://issues.apache.org/jira/browse/TEZ-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935732#comment-15935732
]
TezQA commented on TEZ-3665:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12859832/TEZ-3665.1.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2339//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2339//console
This message is automatically generated.
> TestATSV15HistoryLoggingService should use mocked TimelineClient
>
>
> Key: TEZ-3665
> URL: https://issues.apache.org/jira/browse/TEZ-3665
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3665.1.patch
>
>
> Currently it's extending TimelineClient and won't build on hadoop3. WIth
> mocked TimelineClient, we will be able to build against both hadoop2 and
> hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3654) CartesianProduct edge won't work with GroupInputEdge
[
https://issues.apache.org/jira/browse/TEZ-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935403#comment-15935403
]
TezQA commented on TEZ-3654:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12859785/TEZ-3654.2.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2338//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2338//console
This message is automatically generated.
> CartesianProduct edge won't work with GroupInputEdge
>
>
> Key: TEZ-3654
> URL: https://issues.apache.org/jira/browse/TEZ-3654
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3654.1.patch, TEZ-3654.2.patch
>
>
> If a vertex group is used as source of cartesian product, it expands into
> multiple vertices that share the same edge properties and each CP edge is
> taken as individual cartesian product source by CP vertex manager. CP vertex
> manager will find there are more CP edge than expected and abort the AM.
> Ideally group edge semantic should be fixed: both task and vertex manager
> should see same number of sources; also not every edge can be simply
> duplicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TEZ-3654) CartesianProduct edge won't work with GroupInputEdge
[
https://issues.apache.org/jira/browse/TEZ-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934166#comment-15934166
]
TezQA commented on TEZ-3654:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12859699/TEZ-3654.1.patch
against master revision 293c593.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2337//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2337//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2337//console
This message is automatically generated.
> CartesianProduct edge won't work with GroupInputEdge
>
>
> Key: TEZ-3654
> URL: https://issues.apache.org/jira/browse/TEZ-3654
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3654.1.patch
>
>
> If a vertex group is used as source of cartesian product, it expands into
> multiple vertices that share the same edge properties and each CP edge is
> taken as individual cartesian product source by CP vertex manager. CP vertex
> manager will find there are more CP edge than expected and abort the AM.
> Ideally group edge semantic should be fixed: both task and vertex manager
> should see same number of sources; also not every edge can be simply
> duplicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
