[jira] [Updated] (YARN-11054) Alleviate LocalJobRunnerMetricName Conflicts
[ https://issues.apache.org/jira/browse/YARN-11054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-11054: -- Target Version/s: 3.5.0 (was: 3.4.0) > Alleviate LocalJobRunnerMetricName Conflicts > > > Key: YARN-11054 > URL: https://issues.apache.org/jira/browse/YARN-11054 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.1 >Reporter: Xingjun Hao >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 20m > Remaining Estimate: 0h > > In some scenarios, Sqoop will use LocalJobRuner (YarnLocal mode) to run a lot > of jobs, assuming 2 million jobs have been run, LocalJobRunner MetricName > generated by nextInt function is in the range of (0, 2147483647), > Then the probability of conflict is about 2000/2147483647 = 1/1000, which > means that an average of 1 task will fail for every 1000 jobs run. > If LocalJobRunner MetricName is generated by nextLong() whose range is (0, > 9223372036854775807), considering that Long's range is 1 billion times that > of Int, the probability of a new MetricName conflicts is also reduced by one > trillionth times. > the probability of conflict also goes to 1/5 from 1/1000. ( under the > situation that there are about 200million jobs have been run) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11054) Alleviate LocalJobRunnerMetricName Conflicts
[ https://issues.apache.org/jira/browse/YARN-11054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11054: -- Labels: pull-request-available (was: ) > Alleviate LocalJobRunnerMetricName Conflicts > > > Key: YARN-11054 > URL: https://issues.apache.org/jira/browse/YARN-11054 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.1 >Reporter: Xingjun Hao >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 10m > Remaining Estimate: 0h > > In some scenarios, Sqoop will use LocalJobRuner (YarnLocal mode) to run a lot > of jobs, assuming 2 million jobs have been run, LocalJobRunner MetricName > generated by nextInt function is in the range of (0, 2147483647), > Then the probability of conflict is about 2000/2147483647 = 1/1000, which > means that an average of 1 task will fail for every 1000 jobs run. > If LocalJobRunner MetricName is generated by nextLong() whose range is (0, > 9223372036854775807), considering that Long's range is 1 billion times that > of Int, the probability of a new MetricName conflicts is also reduced by one > trillionth times. > the probability of conflict also goes to 1/5 from 1/1000. ( under the > situation that there are about 200million jobs have been run) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org