Hello Devs, Wanted to reach out and see if anyone has thought about ability to throttle data transfer within HDFS. One option we have been thinking is to throttle on a per FileSystem basis, similar to Statistics in FileSystem. This would mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. Right value to come up for this would be based on type of hardware we use and how many tasks/clients we allow.
On the other hand doing something like this at FileSystem layer would mean many other tasks such as Job jar copy, DistributedCache copy and any hidden data movement would also be throttled. We wanted to know if anyone has had such requirement on their clusters in the past and what was the thinking around it. Appreciate your inputs/comments -- Have a Nice Day! Lohit
