Ok grepping the RS logs I see nothing with 'local' in any of them. Thanks for that hint.
For the test I was using, I know it is data local. Every map task launched data local, and no regions were moving recently. I think I've hijacked this thread enough, I'll move my issues to another. ;-) On Thu, Jan 31, 2013 at 11:51 PM, Anoop Sam John <anoo...@huawei.com> wrote: > Hi Robert > When HDFS is doing the local short circuit read, it will use > BlockReaderLocal class for reading. There should be some logs at the DFS > client side (RS) which tells abt creating new BlockReaderLocal . If you > can see this then sure the local read is happening. > > Also check DN log. If local read happening, then you will not see read > request related logs for the HFile at the DN side. > You check your no# of HFiles and names for checking the logs > > Are you sure that when you tested, u have data locality? Region movements > across RSs can break the full data locality. > > -Anoop- > ________________________________________ > From: Robert Dyer [psyb...@gmail.com] > Sent: Friday, February 01, 2013 11:10 AM > To: Hbase-User > Subject: Re: HBase Checksum > > Not trying to hijack your thread here... > > But can you verify via logs that the shortcircuit is working? Because I > enabled shortcircuit but I sure didn't see any performance increase. > > I haven't tried enabling hbase checksum yet but I'd like to be able to > verify that works too. > > > On Thu, Jan 31, 2013 at 9:55 PM, Anoop Sam John <anoo...@huawei.com> > wrote: > > > You can check with HDFS level logs whether the checksum meta file is > > getting read to the DFS client? In the HBase handled checksum, this > should > > not happen. > > Have you noticed any perf gain when you configure the HBase handled > > checksum option? > > > > -Anoop- > > ________________________________________ > > From: Jean-Marc Spaggiari [jean-m...@spaggiari.org] > > Sent: Friday, February 01, 2013 4:16 AM > > To: user > > Subject: HBase Checksum > > > > Hi, > > > > I have activated shortcircuit and checksum and I would like to get a > > confirmation that it's working fine. > > > > So I have activated short circuit first and saw a 40% improvement of > > the MR rowcount job. So I guess it's working fine. > > > > Now, I'm configuring the checksum option, and I'm wondering how I can > > do to validate that it's taken into consideration and used, or not. Is > > there a way to see that? > > > > Thanks, > > > > JM > > > -- Robert Dyer rd...@iastate.edu
