[
https://issues.apache.org/jira/browse/FLINK-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262343#comment-16262343
]
Stefan Richter edited comment on FLINK-8125 at 11/22/17 11:27 AM:
------------------------------------------------------------------
Maybe we can add this monitoring to the already existing
{{SafetyNetWrapperFileSystem}}, which already returns wrapped streams to ensure
that they are closed?
was (Author: srichter):
Maybe we can add this monitoring to the already existing
`SafetyNetWrapperFileSystem`, which already returns wrapped streams to ensure
that they are closed?
> Support limiting the number of open FileSystem connections
> ----------------------------------------------------------
>
> Key: FLINK-8125
> URL: https://issues.apache.org/jira/browse/FLINK-8125
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Fix For: 1.5.0, 1.4.1
>
>
> We need a way to limit the number of streams that Flink FileSystems
> concurrently open.
> For example, for very small HDFS clusters with few RPC handlers, a large
> Flink job trying to build up many connections during a checkpoint causes
> failures due to rejected connections.
> I propose to add a file system that can wrap another existing file system The
> file system may track the progress of streams and close streams that have
> been inactive for too long, to avoid locked streams of taking up the complete
> pool.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
