Hi Pankil. A similar issue was reported few days back ( http://search-hadoop.com/m/YGbbknQt52rKBDS1&subj=HRegionServer+failed+due+to+replication ).
May be this is due to hbase-indexer code ? One more Q, did you upgrade hbase from 0.94 and you see this issue ? Regards, Ashish Singhi On Fri, Oct 23, 2015 at 2:47 AM, Pankil Doshi <[email protected]> wrote: > Hi, > > I am using hbase-0.98.15-hadoop2 and hbase-indexer from lily ( > http://ngdata.github.io/hbase-indexer/). > > I am seeing below error when I add my indexer: > > > 2015-10-22 14:08:27,468 INFO [regionserver60020-EventThread] > replication.ReplicationTrackerZKImpl: /hbase/replication/peers znode > expired, triggering peerListChanged event > > 2015-10-22 14:08:27,473 ERROR [regionserver60020-EventThread] > regionserver.ReplicationSourceManager: Error while adding a new peer > > org.apache.hadoop.hbase.replication.ReplicationException: Error adding peer > with id=Indexer_newtest2 > > at > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:386) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.peerAdded(ReplicationPeersZKImpl.java:358) > > at > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.peerListChanged(ReplicationSourceManager.java:514) > > at > > org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$PeersWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:189) > > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:468) > > at > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) > > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) > > Caused by: org.apache.hadoop.hbase.replication.ReplicationException: Error > starting the peer state tracker for peerId=Indexer_newtest2 > > at > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:454) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createAndAddPeer(ReplicationPeersZKImpl.java:384) > > ... 6 more > > Caused by: org.apache.zookeeper.KeeperException$DataInconsistencyException: > KeeperErrorCode = DataInconsistency > > at org.apache.hadoop.hbase.zookeeper.ZKUtil.convert(ZKUtil.java:2063) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:85) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.createPeer(ReplicationPeersZKImpl.java:452) > > ... 7 more > > Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: > Missing pb magic PBUF prefix > > at > > org.apache.hadoop.hbase.protobuf.ProtobufUtil.expectPBMagicPrefix(ProtobufUtil.java:270) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.parseStateFrom(ReplicationPeerZKImpl.java:243) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.isStateEnabled(ReplicationPeerZKImpl.java:232) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.readPeerStateZnode(ReplicationPeerZKImpl.java:90) > > at > > org.apache.hadoop.hbase.replication.ReplicationPeerZKImpl.startStateTracker(ReplicationPeerZKImpl.java:83) > > ... 8 more > > > > My Hbase-site.xml: > > > <?xml version="1.0"?> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- > > /** > > * Copyright 2010 The Apache Software Foundation > > * > > * Licensed to the Apache Software Foundation (ASF) under one > > * or more contributor license agreements. See the NOTICE file > > * distributed with this work for additional information > > * regarding copyright ownership. The ASF licenses this file > > * to you under the Apache License, Version 2.0 (the > > * "License"); you may not use this file except in compliance > > * with the License. You may obtain a copy of the License at > > * > > * http://www.apache.org/licenses/LICENSE-2.0 > > * > > * Unless required by applicable law or agreed to in writing, software > > * distributed under the License is distributed on an "AS IS" BASIS, > > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > > * See the License for the specific language governing permissions and > > * limitations under the License. > > */ > > --> > > <configuration> > > <!--<property> > > <name>zookeeper.znode.parent</name> > > <value>/hbase-unsecure</value> > > </property>--> > > > <property> > > <name>hbase.cluster.distributed</name> > > <value>true</value> > > </property> > > //Here you have to set the path where you want HBase to store its files. > > <property> > > <name>hbase.rootdir</name> > > <value>file:/tmp/HBase/HFiles</value> > > </property> > > <property> > > <name>hbase.zookeeper.property.clientPort</name> > > <value>2181</value> > > <description>Property from ZooKeeper's config zoo.cfg. > > The port at which the clients will connect. > > </description> > > </property> > > <property> > > <name>hbase.zookeeper.quorum</name> > > <value>localhost</value> > > <description>Comma separated list of servers in the ZooKeeper Quorum. > > For example, "host1.mydomain.com,host2.mydomain.com, > host3.mydomain.com > ". > > By default this is set to localhost for local and pseudo-distributed > modes > > of operation. For a fully-distributed setup, this should be set to a > full > > list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in > hbase-env.sh > > this is the list of servers which we will start/stop ZooKeeper on. > > </description> > > </property> > > <property> > > <name>hbase.zookeeper.property.dataDir</name> > > <value>/tmp/zookeeper</value> > > <description>Property from ZooKeeper config zoo.cfg. > > The directory where the snapshot is stored. > > </description> > > </property> > > // FOR HBASE INDEXER > > > <!-- SEP is basically replication, so enable it --> > > <property> > > <name>hbase.replication</name> > > <value>true</value> > > </property> > > <!-- Source ratio of 100% makes sure that each SEP consumer is actually > > used (otherwise, some can sit idle, especially with small clusters) > --> > > <property> > > <name>replication.source.ratio</name> > > <value>1.0</value> > > </property> > > <!-- Maximum number of hlog entries to replicate in one go. If this is > > large, and a consumer takes a while to process the events, the > > HBase rpc call will time out. --> > > <property> > > <name>replication.source.nb.capacity</name> > > <value>1000</value> > > </property> > > <!-- A custom replication source that fixes a few things and adds > > some functionality (doesn't interfere with normal replication > > usage). --> > > <property> > > <name>replication.replicationsource.implementation</name> > > <value>com.ngdata.sep.impl.SepReplicationSource</value> > > </property> > > </configuration> > > > All research so far shows me one can face problem while upgrading hbase. > But in my case, I am not upgrading. > > Any ideas/suggestions will be very helpful. > > Thanks, > Pankil >
