[ https://issues.apache.org/jira/browse/YARN-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937364#comment-15937364 ]
Haibo Chen commented on YARN-6376: ---------------------------------- bq. We should synchronize these two operations. Agreed. We may need to create a TimelineWriter wrap for this purpose {code} public class TimelineWriterSynchronizedOnPutEntitiesSync { private TimelineWriter writer; // lock for serializing putEntitiesSync() and flush() private final ReentrantLock lock = new ReentrantLock(); TimelineWriteResponse putEntitesSync() { lock.lock(); // block until condition holds try { writer.write(); writer.flush(); } finally { lock.unlock() } } void putEntitiesAsync() { writer.write(); } void flush() { lock.lock(); try { writer.flush(); } finally { lock.unlock(); } } } {code} However, this quickly gets our of control if there is flush() internal to TimelineWriter, buffer-size based flush for instance, because we can no longer synchronize outside of TimelineWriter > Exceptions caused by synchronous putEntities requests can be swallowed in > TimelineCollector > ------------------------------------------------------------------------------------------- > > Key: YARN-6376 > URL: https://issues.apache.org/jira/browse/YARN-6376 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 > Affects Versions: 3.0.0-alpha2 > Reporter: Haibo Chen > Priority: Critical > Labels: yarn-5355-merge-blocker > > TimelineCollector.putEntitities() is currently implemented by calling > TimelineWriter.write() followed by TimelineWriter.flush(). Given > HBaseTimelineWriter.write() is an asynchronous operation, it is possible that > TimelineClient sends a synchronous putEntities() request for critical data, > but never gets back an exception even though the HBase write request to store > the entities may have failed. > This is due to a race condition between the WriterFlushThread in > TimelineCollectorManager and web threads handling synchronous putEntities() > requests. Entities are first put into the buffer by the web thread, it is > possible that before the web thread invokes writer.flush(), WriterFlushThread > is fired up to flush the writer. If the entities were not successfully > written to the backend during flush, the WriterFlushThread would just simply > log an error, whereas the web thread would never get an exception out from > its writer.flush() invocation. This is bad because the reason of > TimelineClient sending synchronously putEntities() is to retry upon any > exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org