Etienne CARRIERE created FLINK-9560:
---------------------------------------
Summary: RateLimiting for FileSystem
Key: FLINK-9560
URL: https://issues.apache.org/jira/browse/FLINK-9560
Project: Flink
Issue Type: Improvement
Components: FileSystem
Affects Versions: 1.5.0
Reporter: Etienne CARRIERE
*Pain*: On our system, we see that during checkpoint , all the bandwidth is
take to send the checkpoint to object storage (s3 in our case)
*Proposal* : After the creation of some limitation on Filesystem (mostly number
of connections with the tickets FLINK-8125/FLINK-8198/FLINK-9468), I propose
to add ratelimiting "per Filesystem" .
Proposal of implementation : Modify LimitedConnectionsFileSystem to add a
ratelimiter on both Input and OutputStream.
Current issue : I would like to use the guava RateLimiter that is a good
ratelimiter but it is guava and so not included in flink (dependency clash with
hadoop) if I am right. What would be a right strategy in this case ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
