[jira] [Updated] (TEZ-1714) Locking issue with StateChangeNotifier
[ https://issues.apache.org/jira/browse/TEZ-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1714: Affects Version/s: (was: 0.5.1) > Locking issue with StateChangeNotifier > -- > > Key: TEZ-1714 > URL: https://issues.apache.org/jira/browse/TEZ-1714 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Priority: Critical > > The StateChangeNotifier takes a read lock and notifies listeners using a > direct method call. This notification could lead to the listener being > complete. At this point, it may decide to unregister from further status > updates and this should be allowed. However unregister tries to take a write > lock on the StateChangeNotifier and the result is a deadlock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1897) Allow higher concurrency in AsyncDispatcher
Bikas Saha created TEZ-1897: --- Summary: Allow higher concurrency in AsyncDispatcher Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1898) findbugs in tez-api module
Hitesh Shah created TEZ-1898: Summary: findbugs in tez-api module Key: TEZ-1898 URL: https://issues.apache.org/jira/browse/TEZ-1898 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-api.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1899) Fix findbugs warnings in tez-common
Hitesh Shah created TEZ-1899: Summary: Fix findbugs warnings in tez-common Key: TEZ-1899 URL: https://issues.apache.org/jira/browse/TEZ-1899 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-common.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1900) Fix findbugs warnings in tez-dag
Hitesh Shah created TEZ-1900: Summary: Fix findbugs warnings in tez-dag Key: TEZ-1900 URL: https://issues.apache.org/jira/browse/TEZ-1900 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah Might need to be split out more. https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-dag.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1901) Fix findbugs warnings in tez-mapreduce
Hitesh Shah created TEZ-1901: Summary: Fix findbugs warnings in tez-mapreduce Key: TEZ-1901 URL: https://issues.apache.org/jira/browse/TEZ-1901 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-examples.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1901) Fix findbugs warnings in tez-examples
[ https://issues.apache.org/jira/browse/TEZ-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1901: - Summary: Fix findbugs warnings in tez-examples (was: Fix findbugs warnings in tez-mapreduce ) > Fix findbugs warnings in tez-examples > -- > > Key: TEZ-1901 > URL: https://issues.apache.org/jira/browse/TEZ-1901 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah > > https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-examples.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-316) Address findbugs warnings in tez codebase
[ https://issues.apache.org/jira/browse/TEZ-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-316: Assignee: (was: Rekha Joshi) > Address findbugs warnings in tez codebase > - > > Key: TEZ-316 > URL: https://issues.apache.org/jira/browse/TEZ-316 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Blocker > > findbugs output attached to TEZ-272. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1898) Fix findbugs warnings in tez-api module
[ https://issues.apache.org/jira/browse/TEZ-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1898: - Summary: Fix findbugs warnings in tez-api module (was: findbugs in tez-api module) > Fix findbugs warnings in tez-api module > --- > > Key: TEZ-1898 > URL: https://issues.apache.org/jira/browse/TEZ-1898 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Hitesh Shah > > https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-api.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-316) Address findbugs warnings in tez codebase
[ https://issues.apache.org/jira/browse/TEZ-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261462#comment-14261462 ] Hitesh Shah commented on TEZ-316: - Splitting out the tasks so that other folks can help. > Address findbugs warnings in tez codebase > - > > Key: TEZ-316 > URL: https://issues.apache.org/jira/browse/TEZ-316 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Blocker > > findbugs output attached to TEZ-272. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-316) [Umbrella] Address findbugs warnings in tez codebase
[ https://issues.apache.org/jira/browse/TEZ-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-316: Summary: [Umbrella] Address findbugs warnings in tez codebase (was: Address findbugs warnings in tez codebase) > [Umbrella] Address findbugs warnings in tez codebase > > > Key: TEZ-316 > URL: https://issues.apache.org/jira/browse/TEZ-316 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Blocker > > findbugs output attached to TEZ-272. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1903) Fix findbugs warnings in tez-runtime-internals
Hitesh Shah created TEZ-1903: Summary: Fix findbugs warnings in tez-runtime-internals Key: TEZ-1903 URL: https://issues.apache.org/jira/browse/TEZ-1903 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1902) Fix findbugs warnings in tez-mapreduce
Hitesh Shah created TEZ-1902: Summary: Fix findbugs warnings in tez-mapreduce Key: TEZ-1902 URL: https://issues.apache.org/jira/browse/TEZ-1902 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1905) Fix findbugs warnings in tez-tests
Hitesh Shah created TEZ-1905: Summary: Fix findbugs warnings in tez-tests Key: TEZ-1905 URL: https://issues.apache.org/jira/browse/TEZ-1905 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-tests.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1904) Fix findbugs warnings in tez-runtime-library
Hitesh Shah created TEZ-1904: Summary: Fix findbugs warnings in tez-runtime-library Key: TEZ-1904 URL: https://issues.apache.org/jira/browse/TEZ-1904 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1906) Fix findbugs warnings in tez-yarn-timeline-history-with-acls
Hitesh Shah created TEZ-1906: Summary: Fix findbugs warnings in tez-yarn-timeline-history-with-acls Key: TEZ-1906 URL: https://issues.apache.org/jira/browse/TEZ-1906 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah Assignee: Hitesh Shah https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-yarn-timeline-history-with-acls.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1907) Fix javadoc warnings in tez codebase
Hitesh Shah created TEZ-1907: Summary: Fix javadoc warnings in tez codebase Key: TEZ-1907 URL: https://issues.apache.org/jira/browse/TEZ-1907 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Priority: Blocker https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/patchJavadocWarnings.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1908) Analyse and fix javac warnings in tez codebase
Hitesh Shah created TEZ-1908: Summary: Analyse and fix javac warnings in tez codebase Key: TEZ-1908 URL: https://issues.apache.org/jira/browse/TEZ-1908 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Priority: Critical https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/patchJavacWarnings.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1898) Fix findbugs warnings in tez-api module
[ https://issues.apache.org/jira/browse/TEZ-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1898: - Attachment: TEZ-1898.1.patch > Fix findbugs warnings in tez-api module > --- > > Key: TEZ-1898 > URL: https://issues.apache.org/jira/browse/TEZ-1898 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: TEZ-1898.1.patch > > > https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-api.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1909) Remove need to copy over all events from attempt 1 to attempt 2 dir
Hitesh Shah created TEZ-1909: Summary: Remove need to copy over all events from attempt 1 to attempt 2 dir Key: TEZ-1909 URL: https://issues.apache.org/jira/browse/TEZ-1909 Project: Apache Tez Issue Type: Sub-task Reporter: Hitesh Shah Use of file versions should prevent the need for copying over data into a second attempt dir. Care needs to be taken to handle "last corrupt record" handling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1909) Remove need to copy over all events from attempt 1 to attempt 2 dir
[ https://issues.apache.org/jira/browse/TEZ-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1909: - Assignee: Jeff Zhang > Remove need to copy over all events from attempt 1 to attempt 2 dir > --- > > Key: TEZ-1909 > URL: https://issues.apache.org/jira/browse/TEZ-1909 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > > Use of file versions should prevent the need for copying over data into a > second attempt dir. Care needs to be taken to handle "last corrupt record" > handling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1898) Fix findbugs warnings in tez-api module
[
https://issues.apache.org/jira/browse/TEZ-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261604#comment-14261604
]
Hadoop QA commented on TEZ-1898:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689574/TEZ-1898.1.patch
against master revision dcd73b3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 465 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/9//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-yarn-timeline-history-with-acls.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/9//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/9//console
This message is automatically generated.
> Fix findbugs warnings in tez-api module
> ---
>
> Key: TEZ-1898
> URL: https://issues.apache.org/jira/browse/TEZ-1898
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
> Attachments: TEZ-1898.1.patch
>
>
> https://builds.apache.org/job/PreCommit-Tez-Build/8/artifact/patchprocess/newPatchFindbugsWarningstez-api.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-1910) Build fails against hadoop-2.2.0
Hitesh Shah created TEZ-1910: Summary: Build fails against hadoop-2.2.0 Key: TEZ-1910 URL: https://issues.apache.org/jira/browse/TEZ-1910 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Priority: Blocker https://builds.apache.org/job/Tez-Build-Hadoop-2.2/2/console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1897) Allow higher concurrency in AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Attachment: TEZ-1897.1.patch Changed to use ExecutorService and added test. [~hitesh] [~zjffdu] Please review. Follow up jira will be to use this to run multiple speculators/vertex manager events in parallel to offload all user code processing. > Allow higher concurrency in AsyncDispatcher > --- > > Key: TEZ-1897 > URL: https://issues.apache.org/jira/browse/TEZ-1897 > Project: Apache Tez > Issue Type: Task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-1897.1.patch > > > Currently, it processes events on a single thread. For events that can be > executed in parallel, e.g. vertex manager events, allowing higher concurrency > may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1896) Move the default heartbeat timeout and checkinterval to TezConfiguration
[ https://issues.apache.org/jira/browse/TEZ-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1896: Attachment: TEZ-1896-1.patch Attach the patch. > Move the default heartbeat timeout and checkinterval to TezConfiguration > > > Key: TEZ-1896 > URL: https://issues.apache.org/jira/browse/TEZ-1896 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-1896-1.patch > > > Currently the default heartbeat timeout and checkinterval is hard coded in > TaskHeartbeatHandler & ContainerHeartbeatHandler. Move them to > TezConfiguration to make it visible to users so that users can have a > standard to compare what value is suitable for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Allow higher concurrency in AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261847#comment-14261847
]
Jeff Zhang commented on TEZ-1897:
-
Comments:
1. Should remove the public modifier from {code}public AsyncDispatcher(String
name, BlockingQueue eventQueue, int numThreads) {code}, because as we
discussion at in TEZ-1867, we will lose processing order if we create multiple
threads, And this may cause some unexpected weird issues. So at least there
should be only one thread for the central dispatcher, the multiple thread can
only been used internally in registerAndCreateDispatcher
2. This patch cause TestExceptionPropagation#testExceptionPropagationNonSession
fail. This test case will sleep for 1 second to get the diagnostics from RM (
this is due to YARN issue YARN-2560). And I notice that in the
AsyncDispatcher#serviceStop, you will make it sleep 1 second, is it necessary ?
The test case can pass if I remove it.
3. Another minor issue is that AsyncDispatcher would sleep for 1 second when
draining event, is it too long ? Maybe 100 milliseconds would be better.
{code}
waitForDrained.wait(1000);
LOG.info("Waiting for AsyncDispatcher to drain.");
{code}
> Allow higher concurrency in AsyncDispatcher
> ---
>
> Key: TEZ-1897
> URL: https://issues.apache.org/jira/browse/TEZ-1897
> Project: Apache Tez
> Issue Type: Task
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: TEZ-1897.1.patch
>
>
> Currently, it processes events on a single thread. For events that can be
> executed in parallel, e.g. vertex manager events, allowing higher concurrency
> may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1909) Remove need to copy over all events from attempt 1 to attempt 2 dir
[ https://issues.apache.org/jira/browse/TEZ-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261858#comment-14261858 ] Jeff Zhang commented on TEZ-1909: - [~hitesh] Do you mean just use the attempt as the version number ? and just store the recovery event incrementally ? > Remove need to copy over all events from attempt 1 to attempt 2 dir > --- > > Key: TEZ-1909 > URL: https://issues.apache.org/jira/browse/TEZ-1909 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > > Use of file versions should prevent the need for copying over data into a > second attempt dir. Care needs to be taken to handle "last corrupt record" > handling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1895) Vertex reRunning should decrease successfulMembers of VertexGroupInfo
[ https://issues.apache.org/jira/browse/TEZ-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1895: Target Version/s: (was: 0.7.0) > Vertex reRunning should decrease successfulMembers of VertexGroupInfo > - > > Key: TEZ-1895 > URL: https://issues.apache.org/jira/browse/TEZ-1895 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: TEZ-1895-1.patch > > > Vertex reRunning should decrease successfulMembers of VertexGroupInfo, > otherwise commit may happen when vertex is still in rerunning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1897) Allow higher concurrency in AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Attachment: TEZ-1897.2.patch > Allow higher concurrency in AsyncDispatcher > --- > > Key: TEZ-1897 > URL: https://issues.apache.org/jira/browse/TEZ-1897 > Project: Apache Tez > Issue Type: Task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch > > > Currently, it processes events on a single thread. For events that can be > executed in parallel, e.g. vertex manager events, allowing higher concurrency > may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Allow higher concurrency in AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262003#comment-14262003 ] Bikas Saha commented on TEZ-1897: - bq. So at least there should be only one thread for the central dispatcher, the multiple thread can only been used internally in registerAndCreateDispatcher That is the way it is. Central dispatcher has 1 thread and uses the original constructor. If numThreads is not specified then it defaults to 1. The new constructor is package private. I can change the public to package. bq. This patch cause TestExceptionPropagation#testExceptionPropagationNonSession fail. Yes. I noticed this failure too. The javadoc says that it will wait for timeout or completion of existing tasks, whichever is smaller. However, what is happening is that when there are 0 pending tasks then it waits for the timeout period. I am going to match existing behavior by removing the awaitTermination(). What is interesting is that the AM has unregistered from the RM in both cases. That is where the diagnostic is sent to the RM. Not sure how sleeping after that will change the result of the test querying the RM for that final status/diagnostic. bq. that AsyncDispatcher would sleep for 1 second when draining event, That is existing logic that I have not changed. In real execution, we do not set drain events to true. So that code does not get exercised. Attaching updated patch. Please review. Thanks! > Allow higher concurrency in AsyncDispatcher > --- > > Key: TEZ-1897 > URL: https://issues.apache.org/jira/browse/TEZ-1897 > Project: Apache Tez > Issue Type: Task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch > > > Currently, it processes events on a single thread. For events that can be > executed in parallel, e.g. vertex manager events, allowing higher concurrency > may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1909) Remove need to copy over all events from attempt 1 to attempt 2 dir
[ https://issues.apache.org/jira/browse/TEZ-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262006#comment-14262006 ] Hitesh Shah commented on TEZ-1909: -- Actually both. Today, we end up copying over data from the previous attempt into the current attempt's directory. It might be better to just have a chain of partial files to reduce the copy overhead. > Remove need to copy over all events from attempt 1 to attempt 2 dir > --- > > Key: TEZ-1909 > URL: https://issues.apache.org/jira/browse/TEZ-1909 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > > Use of file versions should prevent the need for copying over data into a > second attempt dir. Care needs to be taken to handle "last corrupt record" > handling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1909) Remove need to copy over all events from attempt 1 to attempt 2 dir
[ https://issues.apache.org/jira/browse/TEZ-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262006#comment-14262006 ] Hitesh Shah edited comment on TEZ-1909 at 12/31/14 7:23 AM: Actually both. Today, we end up copying over data from the previous attempt into the current attempt's directory. ( the attempt specific directly already exists hence covers part 1 of your comment ). It might be better to just have a chain of partial files to reduce the copy overhead. was (Author: hitesh): Actually both. Today, we end up copying over data from the previous attempt into the current attempt's directory. It might be better to just have a chain of partial files to reduce the copy overhead. > Remove need to copy over all events from attempt 1 to attempt 2 dir > --- > > Key: TEZ-1909 > URL: https://issues.apache.org/jira/browse/TEZ-1909 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > > Use of file versions should prevent the need for copying over data into a > second attempt dir. Care needs to be taken to handle "last corrupt record" > handling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
