Hi Nathan, Yes, One of my bolt takes lot of time to prepare because it contains some data preparation Operation. So what are solutions for such problem.
On Thu, Aug 20, 2015 at 4:19 PM, Nathan Leung <[email protected]> wrote: > Do you have long running gc? I've seen this cause zk connection loss. > On Aug 20, 2015 2:30 AM, "swapnil joshi" <[email protected]> > wrote: > >> Thanks!!! For giving me response. I had change configuration in storm.yml >> file. But still I got following error >> >> *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >> retry gave up* >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >> at >> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >> at >> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >> retry gave up >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >> at >> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >> at >> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation >> retry gave up >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >> at >> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >> at >> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >> at >> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST >> >> >> On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <[email protected]> wrote: >> >>> Hi, >>> Try to change worker timeout abd zookeeper timeout. >>> Currently i am using below.. >>> >>> Configuration is : >>> storm.zookeeper.session.timeout: 250000 >>> storm.zookeeper.connection.timeout: 250000 >>> nimbus.task.timeout.secs: 300 >>> nimbus.monitor.freq.secs: 10 >>> storm.local.dir: "/mnt/storm" >>> supervisor.worker.timeout.secs: 250 >>> worker.heartbeat.frequency.secs: 5 >>> supervisor.monitor.frequency.secs: 10 >>> supervisor.heartbeat.frequency.secs: 5 >>> >>> - satya >>> On Aug 19, 2015 8:12 PM, "swapnil joshi" <[email protected]> >>> wrote: >>> >>>> Hi Friends, >>>> >>>> In my topology one bolt required time to prepare. Because, It's >>>> doing lots of data loading operation before preparing this bolt storm gives >>>> me following error. >>>> >>>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, timeout of >>>> 20000ms exceeded* >>>> >>>> What is problem? >>>> How will I increase zookeeper Timeout? >>>> >>>> >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd001b, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd001a, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd001e, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd0019, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd001b >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd001d, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session >>>> 0x14f4657e8dd001c, timeout of 20000ms exceeded >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd001a >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd001e >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd0019 >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd001d >>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session >>>> termination for sessionid: 0x14f4657e8dd001c >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background operation >>>> retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change: LOST >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no >>>> ConnectionStateListeners registered. >>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper error >>>> Background operation retry gave up >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>>> KeeperErrorCode = ConnectionLoss >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>>> at >>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380) >>>> at >>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) >>>> at >>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) >>>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable >>>> Zookeeper error") >>>> >>>> -- >>>> Regards, >>>> Swapnil K. Joshi >>>> >>> >> >> >> -- >> Regards, >> Swapnil K. Joshi >> > -- Regards, Swapnil K. Joshi
