Hi Pankil, Are you sure your hbase is running with external zookeeper ensemble ?
As per documentation on Hbase Replication http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-2-0/CDH4-Installation-Guide/cdh4ig_topic_20_11.html zookeeper must not be managed by HBase,.But i havent tried this On Fri, Oct 23, 2015 at 9:55 AM, Ashish Singhi < ashish.singhi.apa...@gmail.com> wrote: > Hi Pankil. > > A similar issue was reported few days back ( > > http://search-hadoop.com/m/YGbbknQt52rKBDS1&subj=HRegionServer+failed+due+to+replication > ). > > May be this is due to hbase-indexer code ? > One more Q, did you upgrade hbase from 0.94 and you see this issue ? > > Regards, > Ashish Singhi > > On Fri, Oct 23, 2015 at 2:47 AM, Pankil Doshi <forpan...@gmail.com> wrote: > > > Hi, > > > > I am using hbase-0.98.15-hadoop2 and hbase-indexer from lily ( > > http://ngdata.github.io/hbase-indexer/). > > > > I am seeing below error when I add my indexer: > > > > > > 2015-10-22 14:08:27,468 INFO [regionserver60020-EventThread] > > replication.ReplicationTrackerZKImpl: /hbase/replication/peers znode > > expired, triggering peerListChanged event > > > > 2015-10-22 14:08:27,473 ERROR [regionserver60020-EventThread] > > regionserver.ReplicationSourceManager: Error while adding a new peer > > > > org.apache.hadoop.hbase.replication.ReplicationException: Error adding > peer > > with id=Indexer_newtest2 > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:386) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.peerAdded(ReplicationPeersZKImpl.java:358) > > > > at > > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.peerListChanged(ReplicationSourceManager.java:514) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$PeersWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:189) > > > > at > > > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:468) > > > > at > > > > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) > > > > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > > > > Caused by: org.apache.hadoop.hbase.replication.ReplicationException: > Error > > starting the peer state tracker for peerId=Indexer_newtest2 > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:454) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:384) > > > > ... 6 more > > > > Caused by: > org.apache.zookeeper.KeeperException$DataInconsistencyException: > > KeeperErrorCode = DataInconsistency > > > > at org.apache.hadoop.hbase.zookeeper.ZKUtil.convert(ZKUtil.java:2063) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:85) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:452) > > > > ... 7 more > > > > Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: > > Missing pb magic PBUF prefix > > > > at > > > > > org.apache.hadoop.hbase.protobuf.ProtobufUtil.expectPBMagicPrefix(ProtobufUtil.java:270) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.parseStateFrom(ReplicationPeerZKImpl.java:243) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.isStateEnabled(ReplicationPeerZKImpl.java:232) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.readPeerStateZnode(ReplicationPeerZKImpl.java:90) > > > > at > > > > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:83) > > > > ... 8 more > > > > > > > > My Hbase-site.xml: > > > > > > <?xml version="1.0"?> > > > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > > > <!-- > > > > /** > > > > * Copyright 2010 The Apache Software Foundation > > > > * > > > > * Licensed to the Apache Software Foundation (ASF) under one > > > > * or more contributor license agreements. See the NOTICE file > > > > * distributed with this work for additional information > > > > * regarding copyright ownership. The ASF licenses this file > > > > * to you under the Apache License, Version 2.0 (the > > > > * "License"); you may not use this file except in compliance > > > > * with the License. You may obtain a copy of the License at > > > > * > > > > * http://www.apache.org/licenses/LICENSE-2.0 > > > > * > > > > * Unless required by applicable law or agreed to in writing, software > > > > * distributed under the License is distributed on an "AS IS" BASIS, > > > > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > implied. > > > > * See the License for the specific language governing permissions and > > > > * limitations under the License. > > > > */ > > > > --> > > > > <configuration> > > > > <!--<property> > > > > <name>zookeeper.znode.parent</name> > > > > <value>/hbase-unsecure</value> > > > > </property>--> > > > > > > <property> > > > > <name>hbase.cluster.distributed</name> > > > > <value>true</value> > > > > </property> > > > > //Here you have to set the path where you want HBase to store its files. > > > > <property> > > > > <name>hbase.rootdir</name> > > > > <value>file:/tmp/HBase/HFiles</value> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.clientPort</name> > > > > <value>2181</value> > > > > <description>Property from ZooKeeper's config zoo.cfg. > > > > The port at which the clients will connect. > > > > </description> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.quorum</name> > > > > <value>localhost</value> > > > > <description>Comma separated list of servers in the ZooKeeper > Quorum. > > > > For example, "host1.mydomain.com,host2.mydomain.com, > > host3.mydomain.com > > ". > > > > By default this is set to localhost for local and > pseudo-distributed > > modes > > > > of operation. For a fully-distributed setup, this should be set to > a > > full > > > > list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in > > hbase-env.sh > > > > this is the list of servers which we will start/stop ZooKeeper on. > > > > </description> > > > > </property> > > > > <property> > > > > <name>hbase.zookeeper.property.dataDir</name> > > > > <value>/tmp/zookeeper</value> > > > > <description>Property from ZooKeeper config zoo.cfg. > > > > The directory where the snapshot is stored. > > > > </description> > > > > </property> > > > > // FOR HBASE INDEXER > > > > > > <!-- SEP is basically replication, so enable it --> > > > > <property> > > > > <name>hbase.replication</name> > > > > <value>true</value> > > > > </property> > > > > <!-- Source ratio of 100% makes sure that each SEP consumer is actually > > > > used (otherwise, some can sit idle, especially with small > clusters) > > --> > > > > <property> > > > > <name>replication.source.ratio</name> > > > > <value>1.0</value> > > > > </property> > > > > <!-- Maximum number of hlog entries to replicate in one go. If this is > > > > large, and a consumer takes a while to process the events, the > > > > HBase rpc call will time out. --> > > > > <property> > > > > <name>replication.source.nb.capacity</name> > > > > <value>1000</value> > > > > </property> > > > > <!-- A custom replication source that fixes a few things and adds > > > > some functionality (doesn't interfere with normal replication > > > > usage). --> > > > > <property> > > > > <name>replication.replicationsource.implementation</name> > > > > <value>com.ngdata.sep.impl.SepReplicationSource</value> > > > > </property> > > > > </configuration> > > > > > > All research so far shows me one can face problem while upgrading hbase. > > But in my case, I am not upgrading. > > > > Any ideas/suggestions will be very helpful. > > > > Thanks, > > Pankil > > > --
