Ok. Was this host (broker id:1,host:c-ccp-tk1-a58,port:9091) up when the
controller had SocketTimeoutException?
Thanks,
Jun
On Mon, Jun 9, 2014 at 10:11 PM, Bongyeon Kim <bongyeon....@gmail.com>
wrote:
> No, I can see any ZK session expiration log.
>
> What I have to do to prevent this? Increasing '
> zookeeper.session.timeout.ms'
> can help?
>
>
> On Tue, Jun 10, 2014 at 12:58 PM, Jun Rao <jun...@gmail.com> wrote:
>
> > This is probably related to kafka-1382. The root cause is likely ZK
> session
> > expiration in the broker. Did you see any?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jun 9, 2014 at 8:11 PM, Bongyeon Kim <bongyeon....@gmail.com>
> > wrote:
> >
> > > Hi, team.
> > >
> > > I’m using 0.8.1.
> > > I found some strange log repeatedly on server.log in one of my brokers
> > and
> > > it keeps logging until now.
> > >
> > > server.log
> > >
> >
> ======================================================================================
> > > ...
> > > [2014-06-09 10:41:47,402] ERROR Conditional update of path
> > > /brokers/topics/topicTRACE/partitions/1/state with data
> > >
> >
> {"controller_epoch":19,"leader":2,"version":1,"leader_epoch":43,"isr":[4,2]}
> > > and expected version 439 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode =
> > > BadVersion for /brokers/topics/topicTRACE/partitions/1/state
> > > (kafka.utils.ZkUtils$)
> > > [2014-06-09 10:41:47,402] INFO Partition [topicTRACE,1] on broker 2:
> > > Cached zkVersion [439] not equal to that in zookeeper, skip updating
> ISR
> > > (kafka.cluster.Partition)
> > > [2014-06-09 10:41:47,402] INFO Partition [topicDEBUG,0] on broker 2:
> > > Shrinking ISR for partition [topicDEBUG,0] from 1,3,2 to 2
> > > (kafka.cluster.Partition)
> > > [2014-06-09 10:41:47,416] ERROR Conditional update of path
> > > /brokers/topics/topicDEBUG/partitions/0/state with data
> > >
> >
> {"controller_epoch":19,"leader":2,"version":1,"leader_epoch":43,"isr":[2]}
> > > and expected version 1424 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode =
> > > BadVersion for /brokers/topics/topicDEBUG/partitions/0/state
> > > (kafka.utils.ZkUtils$)
> > > [2014-06-09 10:41:47,432] INFO Partition [topicDEBUG,0] on broker 2:
> > > Cached zkVersion [1424] not equal to that in zookeeper, skip updating
> ISR
> > > (kafka.cluster.Partition)
> > > [2014-06-09 10:41:47,432] INFO Partition [topicCDR,3] on broker 2:
> > > Shrinking ISR for partition [topicCDR,3] from 4,1,2 to 2
> > > (kafka.cluster.Partition)
> > > [2014-06-09 10:41:47,435] ERROR Conditional update of path
> > > /brokers/topics/topicCDR/partitions/3/state with data
> > >
> >
> {"controller_epoch":19,"leader":2,"version":1,"leader_epoch":46,"isr":[2]}
> > > and expected version 541 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode =
> > > BadVersion for /brokers/topics/topicCDR/partitions/3/state
> > > (kafka.utils.ZkUtils$)
> > > [2014-06-09 10:41:47,435] INFO Partition [topicCDR,3] on broker 2:
> Cached
> > > zkVersion [541] not equal to that in zookeeper, skip updating ISR
> > > (kafka.cluster.Partition)
> > > [2014-06-09 10:41:48,426] INFO Partition [topicTRACE,1] on broker 2:
> > > Shrinking ISR for partition [topicTRACE,1] from 4,3,2 to 4,2
> > > (kafka.cluster.Partition)
> > > ...
> > >
> > >
> >
> =================================================================================================
> > >
> > > and found some error and warning in controller.log
> > >
> > >
> > > controller.log
> > >
> >
> ======================================================================================
> > > ...
> > > [2014-06-09 10:42:03,962] WARN [Controller-3-to-broker-1-send-thread],
> > > Controller 3 fails to send a request to broker
> > > id:1,host:c-ccp-tk1-a58,port:9091 (kafka.controller.RequestSendThread)
> > > java.net.SocketTimeoutException
> > > at
> > > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> > > at
> > sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> > > at
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > at kafka.utils.Utils$.read(Utils.scala:375)
> > > at
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > > at
> > > kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > at
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at
> > kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > at
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146)
> > > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > [2014-06-09 10:42:03,964] ERROR [Controller-3-to-broker-1-send-thread],
> > > Controller 3 epoch 21 failed to send UpdateMetadata request with
> > > correlation id 1 to broker id:1,host:c-ccp-tk1-a58,port:9091.
> > Reconnecting
> > > to broker. (kafka.controller.RequestSendThread)
> > > java.nio.channels.ClosedChannelException
> > > at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > at
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > at
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >
> > > ...
> > >
> > > [2014-06-09 10:42:38,064] WARN [OfflinePartitionLeaderSelector]: No
> > broker
> > > in ISR is alive for [topicTRACE,0]. Elect leader 3 from live brokers 3.
> > > There's potential data loss.
> > > (kafka.controller.OfflinePartitionLeaderSelector)
> > > ...
> > >
> > >
> >
> =================================================================================================
> > >
> > > Why is this happen? Is there any possibilities data loss?
> > > To normalize my brokers, What I have to do? Do I have to restart this
> > > broker?
> > >
> > >
> > > Thanks in advance.
> > >
> > >
> > >
> >
>
>
>
> --
> *Sincerely*
> *,**Bongyeon Kim*
>
> Java Developer & Engineer
> Seoul, Korea
> Mobile: +82-10-9369-1314
> Email: bongyeon...@gmail.com
> Twitter: http://twitter.com/tigerby
> Facebook: http://facebook.com/tigerby
> Wiki: http://tigerby.com
>
