Success: TEZ-3428 PreCommit Build #1993
Jira: https://issues.apache.org/jira/browse/TEZ-3428
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1993/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4808 lines...]
[INFO] Tez SUCCESS [ 0.026 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56:43 min
[INFO] Finished at: 2016-09-28T23:03:26+00:00
[INFO] Final Memory: 90M/1424M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830789/TEZ-3428.1.patch
against master revision 5c2f893.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1993//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1993//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7a6176e3643c75cf3eab5e2085a0441c6645c7a3 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3428
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3428) Tez UI: First Tab not needed for few entries in DAG listings
[
https://issues.apache.org/jira/browse/TEZ-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531147#comment-15531147
]
TezQA commented on TEZ-3428:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830789/TEZ-3428.1.patch
against master revision 5c2f893.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1993//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1993//console
This message is automatically generated.
> Tez UI: First Tab not needed for few entries in DAG listings
>
>
> Key: TEZ-3428
> URL: https://issues.apache.org/jira/browse/TEZ-3428
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3428.1.patch
>
>
> All DAGs get listed in a table in the Tez view (navigated through Hive View's
> Query Tab) which allows for pagination. But when there are few DAGs ( less
> than 5 e.g.), then the navigation buttons still show Tab "First" and Tab "1"
> which can be misleading.
> It would be better to show only the page number when there is only one page
> of DAGs. The "First" tab could become enabled/visible when there are more
> entries into the table and the user navigates to subsequent pages.
> Suggested by ritticheria.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3362: - Attachment: TEZ-3362.004.patch Adding updated patch addressing [~hitesh]'s comments 1) fully and comment 2) partially. Moved all the deletion code to AMNodeTracker. We want to have deletion service work for dags and vertices, which would mean that we could pass a path to delete and no translation would be needed when the HTTP request is received. This implementation does not address that generic api and I was thinking of addressing that as a separate JIRA. Long term having a plugin instead of AMNodeTracker for DAG/vertex deletion would be better. Appreciate more comments/review from [~jeagles] and [~hitesh]. Thanks again! > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch, TEZ-3362.004.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[
https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530961#comment-15530961
]
TezQA commented on TEZ-3419:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830788/YARN%20%26%20Tez%20data%20missing.png
against master revision 5c2f893.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1992//console
This message is automatically generated.
> Tez UI: Applications page shows error, for users with only DAG level ACL
> permission.
>
>
> Key: TEZ-3419
> URL: https://issues.apache.org/jira/browse/TEZ-3419
> Project: Apache Tez
> Issue Type: Sub-task
>Affects Versions: 0.7.0
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3419.1.patch, TEZ-3419.2.patch,
> TEZ-3419.wip.1.patch, Tez data missing.png, YARN & Tez data missing.png, YARN
> data missing.png
>
>
> Follow this logic and display better message:
> On loading app details page, send a request to
> /ws/v1/timeline/TEZ_APPLICATION/tez_
> - If it succeed, display the details page as we do now.
> - If it fails, send a request to
> /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A
> -- If it succeed, then we know that DAGs under the app are available and
> assume that the user doesn't have permission to access app level data.
> --- If AHS is accessible, display application data from there in the details
> page.
> --- else if AHS is not accessible, display a message in app details tab,
> something like "Data is not available. Check if you are authorized to access
> application data!".
> --- Also display the DAGs tab, for the user to see DAGs under that app.
> -- If it fails, display error message as we do now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3419 PreCommit Build #1992
Jira: https://issues.apache.org/jira/browse/TEZ-3419
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1992/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 27 lines...]
Testing patch for TEZ-3419.
==
==
HEAD is now at 5c2f893 TEZ-3429. Set reconfigureDoneTime on
VertexConfigurationDoneEvent properly. (mingma)
Previous HEAD position was 5c2f893... TEZ-3429. Set reconfigureDoneTime on
VertexConfigurationDoneEvent properly. (mingma)
Switched to branch 'master'
Your branch is behind 'origin/master' by 6 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to 5c2f893ab850e077f1c007bb7291c7abe9914b4d.
TEZ-3419 patch is being downloaded at Wed Sep 28 21:40:45 UTC 2016 from
http://issues.apache.org/jira/secure/attachment/12830788/YARN%20%26%20Tez%20data%20missing.png
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830788/YARN%20%26%20Tez%20data%20missing.png
against master revision 5c2f893.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1992//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
f1122a8695f4c3baa972a6ea9268f6b7c542bba2 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-3428) Tez UI: First Tab not needed for few entries in DAG listings
[ https://issues.apache.org/jira/browse/TEZ-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3428: Attachment: TEZ-3428.1.patch [~hitesh] [~harishjp] Please help in reviewing the patch. > Tez UI: First Tab not needed for few entries in DAG listings > > > Key: TEZ-3428 > URL: https://issues.apache.org/jira/browse/TEZ-3428 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3428.1.patch > > > All DAGs get listed in a table in the Tez view (navigated through Hive View's > Query Tab) which allows for pagination. But when there are few DAGs ( less > than 5 e.g.), then the navigation buttons still show Tab "First" and Tab "1" > which can be misleading. > It would be better to show only the page number when there is only one page > of DAGs. The "First" tab could become enabled/visible when there are more > entries into the table and the user navigates to subsequent pages. > Suggested by ritticheria. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3419) Tez UI: Applications page shows error, for users with only DAG level ACL permission.
[ https://issues.apache.org/jira/browse/TEZ-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3419: Attachment: YARN & Tez data missing.png YARN data missing.png Tez data missing.png TEZ-3419.2.patch Attaching a fresh patch. Idea is to display a simple message regarding the configurations to be checked. Also attaching screenshots of the UI in 3 possible cases of missing data. [~hitesh] [~harishjp] Please help in the review. > Tez UI: Applications page shows error, for users with only DAG level ACL > permission. > > > Key: TEZ-3419 > URL: https://issues.apache.org/jira/browse/TEZ-3419 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.7.0 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3419.1.patch, TEZ-3419.2.patch, > TEZ-3419.wip.1.patch, Tez data missing.png, YARN & Tez data missing.png, YARN > data missing.png > > > Follow this logic and display better message: > On loading app details page, send a request to > /ws/v1/timeline/TEZ_APPLICATION/tez_ > - If it succeed, display the details page as we do now. > - If it fails, send a request to > /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A > -- If it succeed, then we know that DAGs under the app are available and > assume that the user doesn't have permission to access app level data. > --- If AHS is accessible, display application data from there in the details > page. > --- else if AHS is not accessible, display a message in app details tab, > something like "Data is not available. Check if you are authorized to access > application data!". > --- Also display the DAGs tab, for the user to see DAGs under that app. > -- If it fails, display error message as we do now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530792#comment-15530792 ] Kuhu Shukla commented on TEZ-3362: -- Thanks [~hitesh] for the comments. bq. 1) Current impl seems to be a blocking call. This can potentiallyblock submission of a new dag for a long time if any node is not reachable or takes a long time to respond ( or if there are too many nodes to cycle through). The cleanup should happen asynchronously. I am working on that at the moment and will have a patch up shortly today. > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530780#comment-15530780 ] Hitesh Shah edited comment on TEZ-3362 at 9/28/16 8:27 PM: --- 2 main concerns with the current impl: 1) Current impl seems to be a blocking call. This can potentiallyblock submission of a new dag for a long time if any node is not reachable or takes a long time to respond ( or if there are too many nodes to cycle through). The cleanup should happen asynchronously. 2) The cleanup code probably should not be in DAGAppMaster. Wouldn't the right place be something like the AMNodeTracker or the appropriate service plugin be the right layer to hand off this cleanup action? Also please consider a potential impl of how vertex level data could be cleaned up in terms of where this code should ideally belong. was (Author: hitesh): 2 main concerns with the current impl: 1) Current impl seems to be a blocking call. This can potentiallyblock submission of a new dag for a long time if any node is not reachable or takes a long time to respond ( or if there are too many nodes to cycle through). The cleanup should happen asynchronously. 2) The cleanup code probably should not be in DAGAppMaster. Wouldn't the right place be something like the AMNodeTracker or the appropriate service plugin be the right layer to hand off this cleanup action? > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530780#comment-15530780 ] Hitesh Shah commented on TEZ-3362: -- 2 main concerns with the current impl: 1) Current impl seems to be a blocking call. This can potentiallyblock submission of a new dag for a long time if any node is not reachable or takes a long time to respond ( or if there are too many nodes to cycle through). The cleanup should happen asynchronously. 2) The cleanup code probably should not be in DAGAppMaster. Wouldn't the right place be something like the AMNodeTracker or the appropriate service plugin be the right layer to hand off this cleanup action? > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3432) Add unit test for session timeout
[
https://issues.apache.org/jira/browse/TEZ-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530507#comment-15530507
]
TezQA commented on TEZ-3432:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830745/TEZ-3432.2.patch
against master revision 5c2f893.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1991//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1991//console
This message is automatically generated.
> Add unit test for session timeout
> -
>
> Key: TEZ-3432
> URL: https://issues.apache.org/jira/browse/TEZ-3432
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sushmitha Sreenivasan
>Assignee: Sreenath Somarajapuram
> Labels: newbie
> Attachments: TEZ-3432.1.patch, TEZ-3432.2.patch
>
>
> Add unit test which sets tez.session.am.dag.submit.timeout.secs to say 5 secs
> and checking if dag submission timeouts after the configured time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3432 PreCommit Build #1991
Jira: https://issues.apache.org/jira/browse/TEZ-3432
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1991/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4806 lines...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:02 h
[INFO] Finished at: 2016-09-28T18:44:32+00:00
[INFO] Final Memory: 75M/1086M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12830745/TEZ-3432.2.patch
against master revision 5c2f893.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1991//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1991//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
0cddb3517820c038e7b89d691de042533c4be2ab logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.37 MB of artifacts by 27.8% relative to #1988
[description-setter] Description set: TEZ-3432
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530293#comment-15530293 ] Kuhu Shukla commented on TEZ-3362: -- Apologies for the accidental Summary line change. > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3362: - Summary: Delete intermediate data at DAG level for Shuffle Handler (was: hbuktnugghkute) > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3432) Add unit test for session timeout
[ https://issues.apache.org/jira/browse/TEZ-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530269#comment-15530269 ] Sreenath Somarajapuram edited comment on TEZ-3432 at 9/28/16 5:17 PM: -- Attaching a fresh patch with the comments addressed. bq. the test is timing dependent - was this run 50-100 times in a row to verify that it is not flaky? Have added tests for the same, as it was taking too long the iterations have been limited to 10. [~hitesh] [~harishjp] Please help in review. was (Author: sreenath): Attaching a fresh patch with the comments addressed. [~hitesh] [~harishjp] Please review. > Add unit test for session timeout > - > > Key: TEZ-3432 > URL: https://issues.apache.org/jira/browse/TEZ-3432 > Project: Apache Tez > Issue Type: Bug >Reporter: Sushmitha Sreenivasan >Assignee: Sreenath Somarajapuram > Labels: newbie > Attachments: TEZ-3432.1.patch, TEZ-3432.2.patch > > > Add unit test which sets tez.session.am.dag.submit.timeout.secs to say 5 secs > and checking if dag submission timeouts after the configured time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3432) Add unit test for session timeout
[ https://issues.apache.org/jira/browse/TEZ-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3432: Attachment: TEZ-3432.2.patch Attaching a fresh patch with the comments addressed. [~hitesh] [~harishjp] Please review. > Add unit test for session timeout > - > > Key: TEZ-3432 > URL: https://issues.apache.org/jira/browse/TEZ-3432 > Project: Apache Tez > Issue Type: Bug >Reporter: Sushmitha Sreenivasan >Assignee: Sreenath Somarajapuram > Labels: newbie > Attachments: TEZ-3432.1.patch, TEZ-3432.2.patch > > > Add unit test which sets tez.session.am.dag.submit.timeout.secs to say 5 secs > and checking if dag submission timeouts after the configured time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3437) Improve synchronization and the progress report behavior for Inputs from TEZ-3317
[ https://issues.apache.org/jira/browse/TEZ-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530140#comment-15530140 ] Kuhu Shukla commented on TEZ-3437: -- [~sseth], Request for comments/review. Thank you! > Improve synchronization and the progress report behavior for Inputs from > TEZ-3317 > - > > Key: TEZ-3437 > URL: https://issues.apache.org/jira/browse/TEZ-3437 > Project: Apache Tez > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: TEZ-3437.001.patch, TEZ-3437.002.patch, > TEZ-3437.003.patch, TEZ-3437.004.patch, TEZ-3437.005.patch, TEZ-3437.006.patch > > > Follow up from TEZ-3317 to improve the getProgress thread synchronization and > replace timerTasks with ScheduledExecutorService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3361) Fetch Multiple Partitions from the Shuffle Handler
[
https://issues.apache.org/jira/browse/TEZ-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529928#comment-15529928
]
Kuhu Shukla commented on TEZ-3361:
--
[~jeagles], Thank you for the latest patch!
{code}
UnorderedKVInput.java:
boolean compositeFetch = ShuffleUtils.isTezShuffleHandler(conf);
{code}
Config key for composite fetch should be part of the static confKeys Set.
{code}
Fetcher.java
/*
if (shouldRetry(srcAttemptId, ioe)) {
//release mem/file handles
cleanupFetchedInput(fetchedInput);
throw new FetcherReadTimeoutException(ioe);
}
{code}
It would be nice if fetchInputs() error handling could retry all if we fail to
fetch even one of the inputs. Similar addition for FetcherOrderedGrouped.
As we discussed offline :
{code}
if (header.getCompressedLength() == 0) {
// Empty partitions are already accounted for
continue;
}
{code}
Similar change is needed in OrderedGrouped case when partLength is zero.
One other follow up would be to remove empty partitions altogether from the
shuffle header.
> Fetch Multiple Partitions from the Shuffle Handler
> --
>
> Key: TEZ-3361
> URL: https://issues.apache.org/jira/browse/TEZ-3361
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3361.1.patch, TEZ-3361.2.patch, TEZ-3361.3.patch
>
>
> Provide an API that allows for fetching multiple partitions at once from a
> single upstream task. This is to better support auto-reduce parallelism where
> a single downstream task is impersonating several (possibly?) consecutive
> downstream tasks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
