[jira] [Commented] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
[ https://issues.apache.org/jira/browse/TEZ-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340827#comment-15340827 ] Zhiyuan Yang commented on TEZ-3307: --- When I integrate CartesianProduct stuff into Hive, there is case where I want to reuse VMPlugin userpayload but it's not easy to keep it around. > Make Vertex.getVertexManagerPlugin public > - > > Key: TEZ-3307 > URL: https://issues.apache.org/jira/browse/TEZ-3307 > Project: Apache Tez > Issue Type: Improvement >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Minor > Attachments: TEZ-3307.1.patch > > > This one is now package private. We should make this public in case some one > need to get VertexManagerPlugin after setting it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3307 PreCommit Build #1807
Jira: https://issues.apache.org/jira/browse/TEZ-3307
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1807/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4104 lines...]
[INFO] Total time: 58:22 min
[INFO] Finished at: 2016-06-21T01:06:46+00:00
[INFO] Final Memory: 71M/1061M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12812044/TEZ-3307.1.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1807//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1807//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
c607fe8a00b8fe9190ad9fea8c763e4120cf1f14 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.19 MB of artifacts by 30.3% relative to #1804
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
[
https://issues.apache.org/jira/browse/TEZ-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340822#comment-15340822
]
TezQA commented on TEZ-3307:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12812044/TEZ-3307.1.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1807//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1807//console
This message is automatically generated.
> Make Vertex.getVertexManagerPlugin public
> -
>
> Key: TEZ-3307
> URL: https://issues.apache.org/jira/browse/TEZ-3307
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: TEZ-3307.1.patch
>
>
> This one is now package private. We should make this public in case some one
> need to get VertexManagerPlugin after setting it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
[ https://issues.apache.org/jira/browse/TEZ-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340819#comment-15340819 ] Hitesh Shah commented on TEZ-3307: -- Any reason why this is needed? Shouldnt it just be possible to overwrite the value for VM instead of getting it and trying to modify it? > Make Vertex.getVertexManagerPlugin public > - > > Key: TEZ-3307 > URL: https://issues.apache.org/jira/browse/TEZ-3307 > Project: Apache Tez > Issue Type: Improvement >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Minor > Attachments: TEZ-3307.1.patch > > > This one is now package private. We should make this public in case some one > need to get VertexManagerPlugin after setting it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3308) Add counters to capture input split length.
[ https://issues.apache.org/jira/browse/TEZ-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned TEZ-3308: -- Assignee: Harish Jaiprakash > Add counters to capture input split length. > --- > > Key: TEZ-3308 > URL: https://issues.apache.org/jira/browse/TEZ-3308 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > > Add counters to capture input split length at task level. This is to > facilitate easy debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3308) Add counters to capture input split length.
Harish Jaiprakash created TEZ-3308: -- Summary: Add counters to capture input split length. Key: TEZ-3308 URL: https://issues.apache.org/jira/browse/TEZ-3308 Project: Apache Tez Issue Type: Improvement Reporter: Harish Jaiprakash Add counters to capture input split length at task level. This is to facilitate easy debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
[ https://issues.apache.org/jira/browse/TEZ-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3307: -- Priority: Minor (was: Major) > Make Vertex.getVertexManagerPlugin public > - > > Key: TEZ-3307 > URL: https://issues.apache.org/jira/browse/TEZ-3307 > Project: Apache Tez > Issue Type: Improvement >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Minor > Attachments: TEZ-3307.1.patch > > > This one is now package private. We should make this public in case some one > need to get VertexManagerPlugin after setting it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
[ https://issues.apache.org/jira/browse/TEZ-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3307: -- Attachment: TEZ-3307.1.patch > Make Vertex.getVertexManagerPlugin public > - > > Key: TEZ-3307 > URL: https://issues.apache.org/jira/browse/TEZ-3307 > Project: Apache Tez > Issue Type: Improvement >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3307.1.patch > > > This one is now package private. We should make this public in case some one > need to get VertexManagerPlugin after setting it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3307) Make Vertex.getVertexManagerPlugin public
Zhiyuan Yang created TEZ-3307: - Summary: Make Vertex.getVertexManagerPlugin public Key: TEZ-3307 URL: https://issues.apache.org/jira/browse/TEZ-3307 Project: Apache Tez Issue Type: Improvement Reporter: Zhiyuan Yang Assignee: Zhiyuan Yang This one is now package private. We should make this public in case some one need to get VertexManagerPlugin after setting it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3291:
--
Attachment: TEZ-3291.5.patch
Modified the patch to do the following.
Tez does not need to handle special cases like localhost. Instead, if all
splits do not have any location information and if the overall splits is less
than the min-split size, tez can skip computing new desired split (i.e by
saying Tez does not have enough information to compute new desired splits).
Higher level apps can handle the localhost scenario and remove the locations if
needed. For instance, Tez already exposes SplitLocationProvider which can be
used by higher level apps and they can pass the appropriate locationprovider
when computing getGroupedSplits.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.4.patch,
> TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340306#comment-15340306 ] Hitesh Shah commented on TEZ-3009: -- \cc [~sseth] as he was already reviewing this > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved TEZ-3014. -- Resolution: Cannot Reproduce > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reopened TEZ-3014: -- > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[
https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339109#comment-15339109
]
TezQA commented on TEZ-3009:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1806//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1806//console
This message is automatically generated.
> Errors that occur during container task acquisition are not logged
> --
>
> Key: TEZ-3009
> URL: https://issues.apache.org/jira/browse/TEZ-3009
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch
>
>
> If TezChild encounters an error while trying to obtain a task the error will
> be silently handled. This results in a mysterious shutdown of containers
> with no cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3009 PreCommit Build #1806
Jira: https://issues.apache.org/jira/browse/TEZ-3009
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1806/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4128 lines...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12778863/TEZ-3009.002.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1806//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1806//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
8ef8d6341d530f299085fe4e20fa4cc7c3d42502 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TA attempt_0_0001_0_01_04_1,
,
Container: [ContainerId: container_1_0001_01_01, NodeId: host1:0,
NodeHttpAddress: host1:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1272)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 222614220
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getConfig();
->
[jira] [Commented] (TEZ-3272) Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
[
https://issues.apache.org/jira/browse/TEZ-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339094#comment-15339094
]
Tsuyoshi Ozawa commented on TEZ-3272:
-
This patch only changes pom.xml, so we don't need to add any test.
> Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
> -
>
> Key: TEZ-3272
> URL: https://issues.apache.org/jira/browse/TEZ-3272
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3272.001.patch
>
>
> The pom.xml includes the following classes. It will be useful to include
> AMContainerImpl and AMNodeImpl.
> {noformat}
> org.apache.tez.dag.app.dag.impl.DAGImpl,
> org.apache.tez.dag.app.dag.impl.VertexImpl,
> org.apache.tez.dag.app.dag.impl.TaskImpl,
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3272) Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
[
https://issues.apache.org/jira/browse/TEZ-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339092#comment-15339092
]
TezQA commented on TEZ-3272:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12811755/TEZ-3272.001.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1805//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1805//console
This message is automatically generated.
> Add AMContainerImpl and AMNodeImpl to StateMachine visualization list
> -
>
> Key: TEZ-3272
> URL: https://issues.apache.org/jira/browse/TEZ-3272
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3272.001.patch
>
>
> The pom.xml includes the following classes. It will be useful to include
> AMContainerImpl and AMNodeImpl.
> {noformat}
> org.apache.tez.dag.app.dag.impl.DAGImpl,
> org.apache.tez.dag.app.dag.impl.VertexImpl,
> org.apache.tez.dag.app.dag.impl.TaskImpl,
> org.apache.tez.dag.app.dag.impl.TaskAttemptImpl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3272 PreCommit Build #1805
Jira: https://issues.apache.org/jira/browse/TEZ-3272
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1805/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4104 lines...]
[INFO] Total time: 52:20 min
[INFO] Finished at: 2016-06-20T07:10:16+00:00
[INFO] Final Memory: 91M/1182M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12811755/TEZ-3272.001.patch
against master revision 993a7b5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1805//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1805//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
d1f46c56788a7a317a4102ceb0e55cfa29ccdc62 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.20 MB of artifacts by 30.3% relative to #1804
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339049#comment-15339049 ] Tsuyoshi Ozawa commented on TEZ-3009: - [~hitesh] could you check this? Jason's patch LGTM, while my review is non-binding one. > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa resolved TEZ-3014. - Resolution: Fixed Closing this for now. Feel free to reopen this. > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3113) massive increase of run time using PipelinedSorter rather than DefaultSorter
[
https://issues.apache.org/jira/browse/TEZ-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339044#comment-15339044
]
Tsuyoshi Ozawa commented on TEZ-3113:
-
> Observed behaviour of PipelinedSorter is that several hundred thousand
> different files are put flat in the same per-TezChild local temporary
> directories, and thing become very slow (not alleging any causality)
I think this behaviour can cause file-system level lock contention if lots
threads access.
One possible solution is partitioning by adding upper limits per directory or
changing directory structure. [~rajesh.balamohan] what do you think?
> massive increase of run time using PipelinedSorter rather than DefaultSorter
>
>
> Key: TEZ-3113
> URL: https://issues.apache.org/jira/browse/TEZ-3113
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.2
> Environment: scalding 0.15-SNAPSHOT per
> https://github.com/twitter/scalding/pull/1446
> cascading 3.1.0-wip-54
> tez-0.8.2
> OpenJDK 8 on AMD64
> Hadoop 2.6.0 (YARN, HDFS); Apache distribution
> Debian Linux 8
> 8 * Intel Core i7-3770K
>Reporter: Cyrille Chépélov
>
> While running a (fairly complex) scalding DAG that was working fine using
> tez-0.6.2, now under tez-0.8.2, the run time became suddenly extremely large.
> Reverting "tez.runtime.sorter.class" -> "LEGACY" restored proper behaviour.
> Difficulties can be traced to this shape of code:
> {code:scala}
> val x: TypedPipe[(String, String)] = ??? // get *LARGE* dataset
> x
> .group
> .mapValues(x => 1L)
> .sum
> .write(TypedTsvHeader("foo.tsv", ('key, 'count)))
> {code}
> where the incoming data contains many, many different keys. Observed
> behaviour of PipelinedSorter is that several hundred thousand different files
> are put flat in the same per-TezChild local temporary directories, and thing
> become very slow (not alleging any causality).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
