[
https://issues.apache.org/jira/browse/TEZ-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved TEZ-4381.
-------------------------------
Resolution: Fixed
> Speed up TestSecureShuffle
> --------------------------
>
> Key: TEZ-4381
> URL: https://issues.apache.org/jira/browse/TEZ-4381
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: László Bodor
> Assignee: Mark Bathori
> Priority: Major
> Fix For: 0.9.3, 0.10.2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> TestSecureShuffle is very slow. We need to experiment with it.
> There is already an optimization like:
> {code}
>
> conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT,
> 2);
> {code}
> Maybe it is worth checking if lowering the number of failed attempts can
> help. In case of shuffle failures, sooner or later the reducer and mapper
> task is retried, so lowering:
> {code}
> TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
> {code}
> might help, theoretically, as we want the DAG to fail as fast as possible.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
