Thank you very much Anu, This is very useful
Manuel From: Anu Engineer [mailto:aengin...@hortonworks.com] Sent: Tuesday, January 30, 2018 12:25 PM To: Manuel Sopena Ballesteros; user@hadoop.apache.org Subject: Re: HDFS latency and bandwidth/speed Hi Manuel, Depending on your use case: There are several tools. Unfortunately, most of them need some familiarity with HDFS. Here is a quick set of links that google returns. https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/Benchmarking.html An old blog, but most of these applications work. There are a set of applications that get shipped with Hadoop. Both DFSIO and Terragen are useful benchmarks. http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/ If this is the first time you are using HDFS, you might want to take this as an opportunity to write a small program that reads the local files and puts them on to HDFS. When you start working against the cluster, it is the apps that matter, and having some familiarity with how applications are written will be very useful. Thanks Anu From: Manuel Sopena Ballesteros <manuel...@garvan.org.au<mailto:manuel...@garvan.org.au>> Date: Monday, January 29, 2018 at 5:08 PM To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>> Subject: HDFS latency and bandwidth/speed Hi all, I am going to start working on HDFS, how could I test HDFS latency and speed? Is there an ioping command and or hdpram or fio I can use in HDFS? Thank you very much Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel...@garvan.org.au<mailto:manuel...@garvan.org.au> NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
