----- Original Message ----- From: Mark question <markq2...@gmail.com> Date: Saturday, October 22, 2011 5:57 am Subject: Remote Blocked Transfer count To: common-user <common-user@hadoop.apache.org>
> Hello, > > I wonder if there is a way to measure how many of the data blocks > havetransferred over the network? Or more generally, how many times > where there > a connection/contact between different machines? > There is a metrics available in Hadoop. Did you check them. The simplest way to configure Hadoop metrics is to funnel them into a user-configurable file on the machine running the daemon. Metrics are organized into “contexts” (Hadoop currently uses “jvm”, “dfs”, “mapred”, and “rpc”), and each context is independently configured http://www.cloudera.com/blog/2009/03/hadoop-metrics/ You can view them by JMX. > I thought of checking the Namenode log file which usually shows > blk_....from src= to dst ... but I'm not sure if it's correct to > count those lines. > I wont recommend to depend on logs. Because if some one changes the log, then it will effect your application. > Any ideas are helpful. > Mark > Regards, Uma
