Kazuki-san, Setting the ZK timeout to a large value will stop the expirations but may not provide sufficiently fast failure detection for your use case of course.
However if even Ganglia stops working during a large mapreduce job, I think you need to question the adequacy of the network hardware. - Andy > From: Kazuki Ohta <kazuki.o...@gmail.com> > Subject: massive zk expirations under heavy network load > To: user@hbase.apache.org > Cc: kazuki.o...@gmail.com > Date: Wednesday, April 20, 2011, 11:41 AM > Hi, > > I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15). > The configuraiton is below. > > hdp0: zk + master + region + nn + dn + jt + tt > hdp1: zk + master + region + snn + dn + tt > hdp2: zk + region + dn + tt > hdp3 to hdp15: region + dn + tt > > Usually, it works really well. But once the user throws MapReduce > job which requires massive network transfer in the shuffle phase, > the master got the zk session timeout exception, and fails-over to > another master. > > The problem is that shuffle network transfer dominates the switch, > and important zk packets are not transferred properly at > that time. > > Even ganglia monitoring seems to stop at that time. And mr task > attempts also got zk session timeouts and dies altogether (about > 100 tasks dies at the same time. input and output are both hbase). > > This is the potential problem running MapReduce job alongside > with HBase. Does anyone know any good solution for this > phenomenon?
