[ https://issues.apache.org/jira/browse/FLINK-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262343#comment-16262343 ]
Stefan Richter edited comment on FLINK-8125 at 11/22/17 11:27 AM: ------------------------------------------------------------------ Maybe we can add this monitoring to the already existing {{SafetyNetWrapperFileSystem}}, which already returns wrapped streams to ensure that they are closed? was (Author: srichter): Maybe we can add this monitoring to the already existing `SafetyNetWrapperFileSystem`, which already returns wrapped streams to ensure that they are closed? > Support limiting the number of open FileSystem connections > ---------------------------------------------------------- > > Key: FLINK-8125 > URL: https://issues.apache.org/jira/browse/FLINK-8125 > Project: Flink > Issue Type: Improvement > Components: Core > Reporter: Stephan Ewen > Assignee: Stephan Ewen > Fix For: 1.5.0, 1.4.1 > > > We need a way to limit the number of streams that Flink FileSystems > concurrently open. > For example, for very small HDFS clusters with few RPC handlers, a large > Flink job trying to build up many connections during a checkpoint causes > failures due to rejected connections. > I propose to add a file system that can wrap another existing file system The > file system may track the progress of streams and close streams that have > been inactive for too long, to avoid locked streams of taking up the complete > pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)