Steve Loughran created HADOOP-18526:
---------------------------------------
Summary: Leak of S3AInstrumentation instances via hadoop Metrics
references
Key: HADOOP-18526
URL: https://issues.apache.org/jira/browse/HADOOP-18526
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.4
Reporter: Steve Loughran
A heap dump of a process running OOM shows that if a process creates then
destroys lots of S3AFS instances, you seem to run out of heap due to references
to S3AInstrumentation and the IOStatisticsStore kept via the hadoop metrics
registry
It doesn't look like S3AInstrumentation.close() is being invoked in
S3AFS.close(). it should -with the IOStats being snapshotted to a local
reference before this happens. This allows for stats of a closed fs to be
examined.
If you look at org.apache.hadoop.ipc.DecayRpcScheduler.MetricsProxy it uses a
WeakReference to refer back to the larger object. we should do the same for
abfs/s3a bindings. ideally do some template proxy class in hadoop common they
can both use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
