Hi Adam, Thanks for the reply. The changes I was referring was in FileSystem.java layer which should not affect HDFS Replication/NameNode operations. To give better idea this would affect clients something like this
Configuration conf = new Configuration(); conf.setInt("read.bandwitdh.mbpersec", 20); // 20MB/s FileSystem fs = FileSystem.get(conf); FSDataInputStream fis = fs.open("/path/to/file.xt"); fis.read(); // <-- This would be max of 20MB/s 2013/11/11 Adam Muise <amu...@hortonworks.com> > See https://issues.apache.org/jira/browse/HDFS-3475 > > Please note that this has met with many unexpected impacts on workload. Be > careful and be mindful of your Datanode memory and network capacity. > > > > > On Mon, Nov 11, 2013 at 1:59 PM, lohit <lohit.vijayar...@gmail.com> wrote: > > > Hello Devs, > > > > Wanted to reach out and see if anyone has thought about ability to > throttle > > data transfer within HDFS. One option we have been thinking is to > throttle > > on a per FileSystem basis, similar to Statistics in FileSystem. This > would > > mean anyone with handle to HDFS/Hftp will be throttled globally within > JVM. > > Right value to come up for this would be based on type of hardware we use > > and how many tasks/clients we allow. > > > > On the other hand doing something like this at FileSystem layer would > mean > > many other tasks such as Job jar copy, DistributedCache copy and any > hidden > > data movement would also be throttled. We wanted to know if anyone has > had > > such requirement on their clusters in the past and what was the thinking > > around it. Appreciate your inputs/comments > > > > -- > > Have a Nice Day! > > Lohit > > > > > > -- > * Adam Muise * Solutions Engineer > ------------------------------ > > Phone: 416-417-4037 > Email: amu...@hortonworks.com > Website: http://www.hortonworks.com/ > > * Follow Us: * > < > http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature > > > < > http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature > > > < > http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature > > > > [image: photo] > > Latest From Our Blog: How to use R and other non-Java languages in > MapReduce and Hive > < > http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Have a Nice Day! Lohit