comnetwork commented on PR #4721: URL: https://github.com/apache/hbase/pull/4721#issuecomment-1228380175
After I read the `FSHLog` code in current master,2.x, 1.x and 0.98, I find there are always existing a single-thread consumer to call blocking `ProtobufLog.append`(eg. in master `FSHLog.RingBufferEventHandler` corresponding to `AsyncFSWAL.consumeExecutor` calls `ProtobufLog.append` in its single-thread `onEvent` method), the problem you describe already exists in `FSHLog`. One possible solution I can think of is to wrap `FSDataOutputStream` with a new `BufProtobufLogWriter` in a similar way to `AsyncProtobufLogWriter`, we just write to the internal `ByteBuffer` of `BufProtobufLogWriter` in `consumeExecutor`. When `BufProtobufLogWriter.sync` is called, we use a single-thread `FSWriter` to invoke `FSOutputStream.write`and this single-thread `FSWriter` use multiple SyncRunners to invoke `FSDataOutputstream.hflush`. `FSDataOutputstream.write` may also block this single single-thread `FSWriter`,so the queue between `consumeExecutor` and `FSWriter` is also likely to be full, once the queue is full, we could fail the sync requests and then execute syncFailed logic to retry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
