Hi Rushabh, >From the logs, whether we can find anything wrong with the system. The Zookeeper cluster becomes unstable due to increase in znode.
Anyone, please help to fix this. Regards, Roshan On Wed, 28 Apr 2021 at 10:34, Roshan <jlks...@gmail.com> wrote: > Below is the exception in the RS related to the replication > > Exception in one cluster where active writing takes place > > 2021-04-26 13:12:53,917 INFO > [regionserver//10.XX.235.XX:16020-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.212.XXX/10.XX.212.XXX:2171, initiating session >> 2021-04-26 13:12:53,941 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10011, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:54,608 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.212.XXX/10.XX.212.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-26 13:12:54,631 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.212.XXX/10.XX.212.XXX:2171, initiating session >> 2021-04-26 13:12:54,655 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10012, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:54,848 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.232.XXX/10.XX.232.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-26 13:12:54,869 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.232.XXX/10.XX.232.XXX:2171, initiating session >> 2021-04-26 13:12:55,018 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.23.XXX/10.XX.23.XXX:2171. Will not attempt to authenticate using SASL >> (unknown error) >> 2021-04-26 13:12:55,041 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.23.XXX/10.XX.23.XXX:2171, initiating session >> 2021-04-26 13:12:55,064 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10011, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:55,112 WARN >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Session 0x1573dd4199c10012 for server >> 10.XX.232.XXX/10.XX.232.XXX:2171, unexpected error, closing socket >> connection and attempting reconnect >> java.io.IOException: Connection reset by peer >> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) >> at sun.nio.ch.IOUtil.read(IOUtil.java:192) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) >> 2021-04-26 13:12:55,918 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.23.XXX/10.XX.23.XXX:2171. Will not attempt to authenticate using SASL >> (unknown error) >> 2021-04-26 13:12:55,941 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.23.XXX/10.XX.23.XXX:2171, initiating session >> 2021-04-26 13:12:55,964 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10012, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:56,047 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.232.XXX/10.XX.232.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-26 13:12:56,068 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.232.XXX/10.XX.232.XXX:2171, initiating session >> 2021-04-26 13:12:56,090 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.232.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10011, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:57,250 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.212.XXX/10.XX.212.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-26 13:12:57,273 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.212.XXX/10.XX.212.XXX:2171, initiating session >> 2021-04-26 13:12:57,297 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10011, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:57,853 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.212.XXX/10.XX.212.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-26 13:12:57,877 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.212.XXX/10.XX.212.XXX:2171, initiating session >> 2021-04-26 13:12:57,900 INFO >> [regionserver//10.XX.235.XX:16020.replicationSource,1-SendThread(10.XX.212.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x1573dd4199c10012, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-26 13:12:57,933 INFO >> [regionserver//10.XX.235.XX:16020-SendThread(10.XX.23.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.23.XXX/10.XX.23.XXX:2171. Will not attempt to authenticate using SASL >> (unknown error) > > > > Below is the exception in another cluster where currently no active writes > in Hbase and but we enabled replication there too > > 2021-04-28 10:30:06,723 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.239.XXX/10.XX.239.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-28 10:30:06,724 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.239.XXX/10.XX.239.XXX:2171, initiating session >> 2021-04-28 10:30:06,725 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.239.XXX/10.XX.239.XXX:2171, sessionid = 0x15791182d1b10003, >> negotiated timeout = 40000 >> 2021-04-28 10:30:07,944 WARN [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Session 0x15791182d1b10003 for server >> 10.XX.239.XXX/10.XX.239.XXX:2171, unexpected error, closing socket >> connection and attempting reconnect >> java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> at sun.nio.ch.IOUtil.write(IOUtil.java:65) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) >> 2021-04-28 10:30:08,185 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.29.XXX/10.XX.29.XXX:2171. Will not attempt to authenticate using SASL >> (unknown error) >> 2021-04-28 10:30:08,186 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.29.XXX/10.XX.29.XXX:2171, initiating session >> 2021-04-28 10:30:08,187 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.29.XXX/10.XX.29.XXX:2171, sessionid = 0x15791182d1b10003, negotiated >> timeout = 40000 >> 2021-04-28 10:30:12,074 WARN [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Session 0x15791182d1b10003 for server >> 10.XX.29.XXX/10.XX.29.XXX:2171, unexpected error, closing socket connection >> and attempting reconnect >> java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> at sun.nio.ch.IOUtil.write(IOUtil.java:65) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) >> 2021-04-28 10:30:12,751 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.242.XXX/10.XX.242.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-28 10:30:12,752 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.242.XXX/10.XX.242.XXX:2171, initiating session >> 2021-04-28 10:30:12,754 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.242.XXX/10.XX.242.XXX:2171, sessionid = 0x15791182d1b10003, >> negotiated timeout = 40000 >> 2021-04-28 10:30:20,202 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x15791182d1b10003, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-28 10:30:20,303 ERROR [ReplicationExecutor-0] >> zookeeper.RecoverableZooKeeper: ZooKeeper multi failed after 4 attempts >> 2021-04-28 10:30:20,303 WARN [ReplicationExecutor-0] >> replication.ReplicationQueuesZKImpl: Got exception in >> copyQueuesFromRSUsingMulti: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at >> org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:992) >> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:672) >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1685) >> at >> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.moveQueueUsingMulti(ReplicationQueuesZKImpl.java:437) >> at >> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueue(ReplicationQueuesZKImpl.java:257) >> at >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:697) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 2021-04-28 10:30:21,295 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.239.XXX/10.XX.239.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-28 10:30:21,295 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.239.XXX/10.XX.239.XXX:2171, initiating session >> 2021-04-28 10:30:21,296 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.239.XXX/10.XX.239.XXX:2171, sessionid = 0x15791182d1b10003, >> negotiated timeout = 40000 >> 2021-04-28 10:30:35,305 INFO [ReplicationExecutor-0] >> replication.ReplicationQueuesZKImpl: Atomically moving >> 10.XX.22.XXX,16020,1597146947282/1's WALs to my queue >> 2021-04-28 10:30:37,260 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x15791182d1b10003, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-28 10:30:37,805 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.29.XXX/10.XX.29.XXX:2171. Will not attempt to authenticate using SASL >> (unknown error) >> 2021-04-28 10:30:37,806 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.29.XXX/10.XX.29.XXX:2171, initiating session >> 2021-04-28 10:30:37,808 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.29.XXX/10.XX.29.XXX:2171, sessionid = 0x15791182d1b10003, negotiated >> timeout = 40000 >> 2021-04-28 10:30:38,390 INFO [main-SendThread(10.XX.29.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x15791182d1b10003, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-28 10:30:38,731 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.242.XXX/10.XX.242.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-28 10:30:38,732 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.242.XXX/10.XX.242.XXX:2171, initiating session >> 2021-04-28 10:30:38,734 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.242.XXX/10.XX.242.XXX:2171, sessionid = 0x15791182d1b10003, >> negotiated timeout = 40000 >> 2021-04-28 10:30:40,522 INFO [main-SendThread(10.XX.242.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x15791182d1b10003, likely server has closed socket, closing socket >> connection and attempting reconnect >> 2021-04-28 10:30:40,765 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Opening socket connection to server >> 10.XX.239.XXX/10.XX.239.XXX:2171. Will not attempt to authenticate using >> SASL (unknown error) >> 2021-04-28 10:30:40,766 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Socket connection established to >> 10.XX.239.XXX/10.XX.239.XXX:2171, initiating session >> 2021-04-28 10:30:40,767 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Session establishment complete on server >> 10.XX.239.XXX/10.XX.239.XXX:2171, sessionid = 0x15791182d1b10003, >> negotiated timeout = 40000 >> 2021-04-28 10:30:44,651 INFO [main-SendThread(10.XX.239.XXX:2171)] >> zookeeper.ClientCnxn: Unable to read additional data from server sessionid >> 0x15791182d1b10003, likely server has closed socket, closing socket >> connection and attempting reconnect > > > Regards, > Roshan > > On Tue, 27 Apr 2021 at 23:48, Rushabh Shah > <rushabh.s...@salesforce.com.invalid> wrote: > >> Hi Roshan, >> Are you seeing any replication related exception in your RS logs ? >> >> >> >> >> On Tue, Apr 27, 2021 at 1:59 PM Roshan <jlks...@gmail.com> wrote: >> >> > Hi, >> > >> > In the hbase-1.4.10, I have enabled replication for all tables and >> > configured the peer_id. the list_peers provide the below result: >> > >> > hbase(main):001:0> list_peers >> > > PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE TABLE_CFS BANDWIDTH >> > > 1 10.XX.221.XX,10.XX.234.XX,10.XX.212.XX:2171:/hbase nil ENABLED nil >> 0 >> > > 1 row(s) in 0.1430 seconds >> > >> > >> > But the status_replication shows replication lag >> > >> > hbase(main):002:0> status 'replication' >> > > version 1.4.10 >> > > 3 live servers >> > > 10.XX.232.XX: >> > > SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1, >> > > TimeStampsOfLastShippedOp=Thu Jan 01 05:30:00 IST 1970, Replication >> Lag= >> > > *1619545264329* >> > > SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Tue >> Apr 27 >> > > 23:09:23 IST 2021 >> > > 10.XX.118.XX: >> > > SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1, >> > > TimeStampsOfLastShippedOp=Thu Jan 01 05:30:00 IST 1970, Replication >> Lag= >> > > *1619545264663* >> > > SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Tue >> Apr 27 >> > > 18:53:23 IST 2021 >> > > 10.XX.138.XX: >> > > SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1, >> > > TimeStampsOfLastShippedOp=Thu Jan 01 05:30:00 IST 1970, Replication >> Lag= >> > > *1619545263509* >> > > SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Tue >> Apr 27 >> > > 10:31:05 IST 2021 >> > >> > >> > >> > But all the data are replicated properly to the defined cluster. I have >> > checked the table in both clusters. >> > >> > I have verified using VerifyReplication Mapreduce to check unreplicated >> > rows. But there are no rows in the unreplicated one. All are good Rows. >> > >> > ./hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication >> 1 >> > > tablename >> > >> > >> > >> > >> org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters >> > > GOODROWS=45 >> > > File Input Format Counters >> > > Bytes Read=0 >> > > File Output Format Counters >> > > Bytes Written=0 >> > >> > >> > Due to this issue, I have Zknodes under replication is growing >> > exponentially which causes issues in running ZK cluster which eventually >> > affects the Hbase Connection too. Below exception occurs in ZK >> > >> > *ERROR java.io.IOException: Len error* >> > >> > Increasing jute.maxbuffer in ZK will not solve the problem as >> replication >> > znode is increasing though the data are replicated properly to the given >> > cluster Peer_id. >> > >> > I have enabled two-way replication between the cluster. It happens in >> both >> > the cluster. >> > >> > hbase version - 1.4.10 >> > ZK Version - 3.4.10 >> > Hadoop version - 2.7.3 >> > >> > Please help to fix this. >> > >> > Regards, >> > Roshan >> > >> >
