[
https://issues.apache.org/jira/browse/HADOOP-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738228#comment-17738228
]
ASF GitHub Bot commented on HADOOP-18781:
-----------------------------------------
mukund-thakur commented on PR #5780:
URL: https://github.com/apache/hadoop/pull/5780#issuecomment-1611846911
> This problem I guess would happen when job opens filesystem and opens an
OutputStream and the job is writing data into it, But after outputStream open,
there is no reference to AzureBlobFileSystem object in the job.
Yes
> Shouldn't jobs have logic to close the filesystem. Kindly point if there
are other usecases where in this problem arises. Thanks!
No, close() on an FS you got from FileSystem.get shouldn't be called as they
are shared. Things like FileSystem.closeAllForUGI() do the cleanup
> ABFS Output stream thread pools getting shutdown during GC.
> -----------------------------------------------------------
>
> Key: HADOOP-18781
> URL: https://issues.apache.org/jira/browse/HADOOP-18781
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
>
> Applications using AzureBlobFileSystem to create the AbfsOutputStream can use
> the AbfsOutputStream for the purpose of writing, however, the OutputStream
> doesn't hold any reference to the fs instance that created it, which can make
> the FS instance eligible for GC, when this occurs, AzureblobFileSystem's
> `finalize()` method gets called which in turn closes the FS, and in turn call
> the close for AzureBlobFileSystemStore, which uses the same Threadpool that
> is used by the AbfsOutputStream. This leads to the closing of the thread pool
> while the writing is happening in the background and leads to hanging while
> writing.
>
> *Solution:*
> Pass a backreference of AzureBlobFileSystem into AzureBlobFileSystemStore and
> AbfsOutputStream as well.
>
> Same should be done for AbfsInputStream as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
