[ https://issues.apache.org/jira/browse/YARN-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Lu updated YARN-2673: ------------------------ Attachment: YARN-2673-101414.patch Upload a patch for this issue. TimelineClient will by default retry for a given amount of time before throw the exception on posting to server. There are a few notes: 1. Retrying vs. discarding timeline data: If we do not adding this retry, timeline client will drop the posted data if the first attempt has failed. Had a offline discussion with [~vinodkv]. We agreed that blocking the timeline client for a short while is better, since we may not want to drop some critical timeline data. 2. Retry behavior configurations: Users can define maximum retry counts, and time interval between consecutive retries. We may want to have two levels of retry settings: a cluster global settings, managed by yarn-site.xml, and a per-application customize setting. For the cluster setting, I've added two configuration properties, yarn.timeline-service.client.max-retries (default 30) and yarn.timeline-service.client.retry-interval-ms (default 1000). I've also provide a customizeRetrySettings method for application specific retry settings. 3. Retry implementation: timeline client does not use RPC, but uses RESTful APIs. I'm implementing retry as a jersey filter in this patch. 4. Tests: I added two new unit tests, one to test the customizeRetrySettings API and the other to test if the retry has actually happened when we try to post timeline entities. > Add retry for timeline client > ----------------------------- > > Key: YARN-2673 > URL: https://issues.apache.org/jira/browse/YARN-2673 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Li Lu > Assignee: Li Lu > Attachments: YARN-2673-101414.patch > > > Timeline client now does not handle the case gracefully when the server is > down. Jobs from distributed shell may fail due to ATS restart. We may need to > add some retry mechanisms to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)