[ https://issues.apache.org/jira/browse/HADOOP-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087838#comment-17087838 ]
Steve Loughran commented on HADOOP-16998: ----------------------------------------- can tag this with the specific version of hadoop you are having problems with. Try hadoop branch-3/trunk if not already done -that is, something with HADOOP-16785 in, which tried to harden thos close work. If that's not enough, at least it has a start with where to begin testing this. Patches up on github as PRs for review, thanks. > WASB : NativeAzureFsOutputStream#close() throwing > java.lang.IllegalArgumentException instead of IOE which causes HBase RS to > get aborted > ---------------------------------------------------------------------------------------------------------------------------------------- > > Key: HADOOP-16998 > URL: https://issues.apache.org/jira/browse/HADOOP-16998 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure > Reporter: Anoop Sam John > Assignee: Anoop Sam John > Priority: Major > Attachments: HADOOP-16998.patch > > > During HFile create, at the end when called close() on the OutputStream, > there is some pending data to get flushed. When this flush happens, an > Exception is thrown back from Storage. The Azure-storage SDK layer will throw > back IOE. (Even if it is a StorageException thrown from the Storage, the SDK > converts it to IOE.) But at HBase, we end up getting IllegalArgumentException > which causes the RS to get aborted. If we get back IOE, the flush will get > retried instead of aborting RS. > The reason is this > NativeAzureFsOutputStream uses Azure-storage SDK's BlobOutputStreamInternal. > But the BlobOutputStreamInternal is wrapped within a SyncableDataOutputStream > which is a FilterOutputStream. During the close op, NativeAzureFsOutputStream > calls close on SyncableDataOutputStream and it uses below method from > FilterOutputStream > {code} > public void close() throws IOException { > try (OutputStream ostream = out) { > flush(); > } > } > {code} > Here the flush call caused an IOE to be thrown to here. The finally will > issue close call on ostream (Which is an instance of BlobOutputStreamInternal) > When BlobOutputStreamInternal#close() is been called, if there was any > exception already occured on that Stream, it will throw back the same > Exception > {code} > public synchronized void close() throws IOException { > try { > // if the user has already closed the stream, this will throw a > STREAM_CLOSED exception > // if an exception was thrown by any thread in the > threadExecutor, realize it now > this.checkStreamState(); > ... > } > private void checkStreamState() throws IOException { > if (this.lastError != null) { > throw this.lastError; > } > } > {code} > So here both try and finally block getting Exceptions and Java uses > Throwable#addSuppressed() > Within this method if both Exceptions are same objects, it throws back > IllegalArgumentException > {code} > public final synchronized void addSuppressed(Throwable exception) { > if (exception == this) > throw new > IllegalArgumentException(SELF_SUPPRESSION_MESSAGE, exception); > .... > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org