snvijaya commented on a change in pull request #1907: HADOOP-16854 Fix to prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common across all AbfsOutputStream instances URL: https://github.com/apache/hadoop/pull/1907#discussion_r399030377
########## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java ########## @@ -100,19 +99,62 @@ public AbfsOutputStream( this.appendBlob = appendBlob; this.lastError = null; this.lastFlushOffset = 0; - this.bufferSize = bufferSize; - this.buffer = byteBufferPool.getBuffer(false, bufferSize).array(); this.bufferIndex = 0; - this.writeOperations = new ConcurrentLinkedDeque<>(); - - this.maxConcurrentRequestCount = 4 * Runtime.getRuntime().availableProcessors(); - this.threadExecutor - = new ThreadPoolExecutor(maxConcurrentRequestCount, - maxConcurrentRequestCount, - 10L, - TimeUnit.SECONDS, - new LinkedBlockingQueue<>()); - this.completionService = new ExecutorCompletionService<>(this.threadExecutor); + + init(abfsConfiguration); + buffer = new byte[bufferSize]; + } + + private void init(final AbfsConfiguration conf) { + if (isCommonPoolsInitialised()) { + return; + } + + initWriteBufferPool(conf); Review comment: As discussed, all implementations not tied to the AbfsOutputStream should reside outside this class. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org