Yes, You are right. This is GC problem. Now to avoid this issue. I have
increased my JVM Heap size. Now the new problem rises. After some time bolt
start prepare method again and again.
Currently, My System have 16GB RAM and SWAP area is Approx. 28GB. And As
per basic calculation, I need 26GB memory. I have run this topology in
local cluster mode.
On Fri, Aug 21, 2015 at 12:18 AM, Nathan Leung <ncle...@gmail.com> wrote:
> Prepare shouldn't cause a timeout. When I said gc I meant something like
> stop the world garbage collection.
>
> On Thu, Aug 20, 2015 at 6:52 AM, swapnil joshi <
> webmaster.swap...@gmail.com> wrote:
>
>> Hi Nathan,
>> Yes, One of my bolt takes lot of time to prepare because it contains some
>> data preparation Operation. So what are solutions for such problem.
>>
>> On Thu, Aug 20, 2015 at 4:19 PM, Nathan Leung <ncle...@gmail.com> wrote:
>>
>>> Do you have long running gc? I've seen this cause zk connection loss.
>>> On Aug 20, 2015 2:30 AM, "swapnil joshi" <webmaster.swap...@gmail.com>
>>> wrote:
>>>
>>>> Thanks!!! For giving me response. I had change configuration in
>>>> storm.yml file. But still I got following error
>>>>
>>>> *15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background
>>>> operation retry gave up*
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>> at
>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>> at
>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation
>>>> retry gave up
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>> at
>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>> at
>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST
>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST
>>>> 15/08/20 11:44:17 ERROR imps.CuratorFrameworkImpl: Background operation
>>>> retry gave up
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>> at
>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>> at
>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>> 15/08/20 11:44:17 INFO state.ConnectionStateManager: State change: LOST
>>>>
>>>>
>>>> On Wed, Aug 19, 2015 at 8:16 PM, satyavrat <bit.sa...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Try to change worker timeout abd zookeeper timeout.
>>>>> Currently i am using below..
>>>>>
>>>>> Configuration is :
>>>>> storm.zookeeper.session.timeout: 250000
>>>>> storm.zookeeper.connection.timeout: 250000
>>>>> nimbus.task.timeout.secs: 300
>>>>> nimbus.monitor.freq.secs: 10
>>>>> storm.local.dir: "/mnt/storm"
>>>>> supervisor.worker.timeout.secs: 250
>>>>> worker.heartbeat.frequency.secs: 5
>>>>> supervisor.monitor.frequency.secs: 10
>>>>> supervisor.heartbeat.frequency.secs: 5
>>>>>
>>>>> - satya
>>>>> On Aug 19, 2015 8:12 PM, "swapnil joshi" <webmaster.swap...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Friends,
>>>>>>
>>>>>> In my topology one bolt required time to prepare. Because, It's
>>>>>> doing lots of data loading operation before preparing this bolt storm
>>>>>> gives
>>>>>> me following error.
>>>>>>
>>>>>> *server.ZooKeeperServer: Expiring session 0x14f4657e8dd001b, timeout
>>>>>> of 20000ms exceeded*
>>>>>>
>>>>>> What is problem?
>>>>>> How will I increase zookeeper Timeout?
>>>>>>
>>>>>>
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd001b, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd001a, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd001e, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd0019, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd001b
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd001d, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.ZooKeeperServer: Expiring session
>>>>>> 0x14f4657e8dd001c, timeout of 20000ms exceeded
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd001a
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd001e
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd0019
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd001d
>>>>>> 15/08/19 20:05:06 INFO server.PrepRequestProcessor: Processed session
>>>>>> termination for sessionid: 0x14f4657e8dd001c
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 ERROR imps.CuratorFrameworkImpl: Background
>>>>>> operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO state.ConnectionStateManager: State change:
>>>>>> LOST
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 WARN state.ConnectionStateManager: There are no
>>>>>> ConnectionStateListeners registered.
>>>>>> 15/08/19 20:05:06 ERROR storm.zookeeper: Unrecoverable Zookeeper
>>>>>> error Background operation retry gave up
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss
>>>>>> at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:380)
>>>>>> at
>>>>>> com.netflix.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:49)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>>>>>> 15/08/19 20:05:06 INFO storm.util: Halting process: ("Unrecoverable
>>>>>> Zookeeper error")
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Swapnil K. Joshi
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Swapnil K. Joshi
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Swapnil K. Joshi
>>
>
>
--
Regards,
Swapnil K. Joshi
