Hi Danijel,
We are able to reproduce this issue with 0.9.2 as well.
We have two worker setup to run the trident topology.
When we kill one of the worker and again when that killed worker spawn on
same port(same slot) then that worker not able to communicate with 2nd
worker.
only transaction attempts are increasing continuously.
But if the killed worker spawn on new slot(new communication port) then it
working fine. Same behavior as in storm 9.0.1.
Please update me if you get any new development.
Regards
Tarkeshwar
On Thu, Jul 3, 2014 at 7:06 PM, Danijel Schiavuzzi <dani...@schiavuzzi.com>
wrote:
> Hi Bobby,
>
> Just an update on the stuck Trident transactional topology issue -- I've
> upgraded to Storm 0.9.2-incubating (from 0.9.1-incubating) and can't
> reproduce the bug anymore. Will keep you posted if any issues arise.
>
> Regards,
>
> Danijel
>
>
> On Mon, Jun 16, 2014 at 7:56 PM, Bobby Evans <ev...@yahoo-inc.com> wrote:
>
>> I have not seen this before, if you could file a JIRA on this that
>> would be great.
>>
>> - Bobby
>>
>> From: Danijel Schiavuzzi <dani...@schiavuzzi.com>
>> Reply-To: "user@storm.incubator.apache.org" <
>> user@storm.incubator.apache.org>
>> Date: Wednesday, June 4, 2014 at 10:30 AM
>> To: "user@storm.incubator.apache.org" <user@storm.incubator.apache.org>,
>> "d...@storm.incubator.apache.org" <d...@storm.incubator.apache.org>
>> Subject: Trident transactional topology stuck re-emitting batches with
>> Netty, but running fine with ZMQ (was Re: Topology is stuck)
>>
>> Hi all,
>>
>> I've managed to reproduce the stuck topology problem and it seems it's
>> due to the Netty transport. Running with ZMQ transport enabled now and I
>> haven't been able to reproduce this.
>>
>> The problem is basically a Trident/Kafka transactional topology getting
>> stuck, i.e. re-emitting the same batches over and over again. This happens
>> after the Storm workers restart a few times due to Kafka spout throwing
>> RuntimeExceptions (because of the Kafka consumer in the spout timing out
>> with a SocketTimeoutException due to some temporary network problems).
>> Sometimes the topology is stuck after just one worker is restarted, and
>> sometimes a few worker restarts are needed to trigger the problem.
>>
>> I simulated the Kafka spout socket timeouts by blocking network access
>> from Storm to my Kafka machines (with an iptables firewall rule). Most of
>> the time the spouts (workers) would restart normally (after re-enabling
>> access to Kafka) and the topology would continue to process batches, but
>> sometimes the topology would get stuck re-emitting batches after the
>> crashed workers restarted. Killing and re-submitting the topology manually
>> fixes this always, and processing continues normally.
>>
>> I haven't been able to reproduce this scenario after reverting my Storm
>> cluster's transport to ZeroMQ. With Netty transport, I can almost always
>> reproduce the problem by causing a worker to restart a number of times
>> (only about 4-5 worker restarts are enough to trigger this).
>>
>> Any hints on this? Anyone had the same problem? It does seem a serious
>> issue as it affect the reliability and fault tolerance of the Storm cluster.
>>
>> In the meantime, I'll try to prepare a reproducible test case for this.
>>
>> Thanks,
>>
>> Danijel
>>
>>
>> On Mon, Mar 31, 2014 at 4:39 PM, Danijel Schiavuzzi <
>> dani...@schiavuzzi.com> wrote:
>>
>>> To (partially) answer my own question -- I still have no idea on the
>>> cause of the stuck topology, but re-submitting the topology helps -- after
>>> re-submitting my topology is now running normally.
>>>
>>>
>>> On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi <
>>> dani...@schiavuzzi.com> wrote:
>>>
>>>> Also, I did have multiple cases of my IBackingMap workers dying
>>>> (because of RuntimeExceptions) but successfully restarting afterwards (I
>>>> throw RuntimeExceptions in the BackingMap implementation as my strategy in
>>>> rare SQL database deadlock situations to force a worker restart and to
>>>> fail+retry the batch).
>>>>
>>>> From the logs, one such IBackingMap worker death (and subsequent
>>>> restart) resulted in the Kafka spout re-emitting the pending tuple:
>>>>
>>>> 2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emitting
>>>> batch, attempt 29698959:736
>>>>
>>>> This is of course the normal behavior of a transactional topology,
>>>> but this is the first time I've encountered a case of a batch retrying
>>>> indefinitely. This is especially suspicious since the topology has been
>>>> running fine for 20 days straight, re-emitting batches and restarting
>>>> IBackingMap workers quite a number of times.
>>>>
>>>> I can see in my IBackingMap backing SQL database that the batch with
>>>> the exact txid value 29698959 has been committed -- but I suspect that
>>>> could come from another BackingMap, since there are two BackingMap
>>>> instances running (paralellismHint 2).
>>>>
>>>> However, I have no idea why the batch is being retried indefinitely
>>>> now nor why it hasn't been successfully acked by Trident.
>>>>
>>>> Any suggestions on the area (topology component) to focus my research
>>>> on?
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi <
>>>> dani...@schiavuzzi.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm having problems with my transactional Trident topology. It has
>>>>> been running fine for about 20 days, and suddenly is stuck processing a
>>>>> single batch, with no tuples being emitted nor tuples being persisted by
>>>>> the TridentState (IBackingMap).
>>>>>
>>>>> It's a simple topology which consumes messages off a Kafka queue. The
>>>>> spout is an instance of storm-kafka-0.8-plus
>>>>> TransactionalTridentKafkaSpout
>>>>> and I use the trident-mssql transactional TridentState implementation to
>>>>> persistentAggregate() data into a SQL database.
>>>>>
>>>>> In Zookeeper I can see Storm is re-trying a batch, i.e.
>>>>>
>>>>> "/transactional/<myTopologyName>/coordinator/currattempts" is
>>>>> "{"29698959":6487}"
>>>>>
>>>>> ... and the attempt count keeps increasing. It seems the batch with
>>>>> txid 29698959 is stuck, as the attempt count in Zookeeper keeps increasing
>>>>> -- seems like the batch isn't being acked by Trident and I have no idea
>>>>> why, especially since the topology has been running successfully the last
>>>>> 20 days.
>>>>>
>>>>> I did rebalance the topology on one occasion, after which it
>>>>> continued running normally. Other than that, no other modifications were
>>>>> done. Storm is at version 0.9.0.1.
>>>>>
>>>>> Any hints on how to debug the stuck topology? Any other useful info
>>>>> I might provide?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> Danijel Schiavuzzi
>>>>>
>>>>> E: dani...@schiavuzzi.com
>>>>> W: www.schiavuzzi.com
>>>>> T: +385989035562
>>>>> Skype: danijel.schiavuzzi
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Danijel Schiavuzzi
>>>>
>>>> E: dani...@schiavuzzi.com
>>>> W: www.schiavuzzi.com
>>>> T: +385989035562
>>>> Skype: danijel.schiavuzzi
>>>>
>>>
>>>
>>>
>>> --
>>> Danijel Schiavuzzi
>>>
>>> E: dani...@schiavuzzi.com
>>> W: www.schiavuzzi.com
>>> T: +385989035562
>>> Skype: danijels7
>>>
>>
>>
>>
>> --
>> Danijel Schiavuzzi
>>
>> E: dani...@schiavuzzi.com
>> W: www.schiavuzzi.com
>> T: +385989035562
>> Skype: danijels7
>>
>
>
>
> --
> Danijel Schiavuzzi
>
> E: dani...@schiavuzzi.com
> W: www.schiavuzzi.com
> T: +385989035562
> Skype: danijels7
>
