We were also caught by this - we're running CentOS 6. Likewise, once we reset the date/time, HBase was happy again. I wonder why Java processes seemed to be affected more than other processes?
On Jul 2, 2012, at 8:34 AM, Dean Banks wrote: > We were caught by this issue. It impacted all of our JVM processes, > including the full hbase/zookeeper/HDFS stack. > > Before we had determined root cause, we somewhat hastily bounced some of our > services (ZK, HM), and had to forcefully kill a few RSes in an attempt to get > things back online. > > At the end of the day (with root cause addressed), I was thrilled to see that > our hbase cluster came back online, split logs for 10 minutes, then passed > hbck as soon as all the regions were available. It is very impressive that > hbase is capable of withstanding what amounted to a pretty nasty scenario > without data loss. > > Nice work, guys. > > -Dean
