Hi Tarkeshwar, Could you provide a code sample of your topology? Do you have any special configs enabled?
Thanks, Danijel On Mon, Jul 7, 2014 at 9:01 AM, M.Tarkeshwar Rao <tarkeshwa...@gmail.com> wrote: > Hi Danijel, > > We are able to reproduce this issue with 0.9.2 as well. > We have two worker setup to run the trident topology. > > When we kill one of the worker and again when that killed worker spawn on > same port(same slot) then that worker not able to communicate with 2nd > worker. > > only transaction attempts are increasing continuously. > > But if the killed worker spawn on new slot(new communication port) then it > working fine. Same behavior as in storm 9.0.1. > > Please update me if you get any new development. > > Regards > Tarkeshwar > > > On Thu, Jul 3, 2014 at 7:06 PM, Danijel Schiavuzzi <dani...@schiavuzzi.com > > wrote: > >> Hi Bobby, >> >> Just an update on the stuck Trident transactional topology issue -- I've >> upgraded to Storm 0.9.2-incubating (from 0.9.1-incubating) and can't >> reproduce the bug anymore. Will keep you posted if any issues arise. >> >> Regards, >> >> Danijel >> >> >> On Mon, Jun 16, 2014 at 7:56 PM, Bobby Evans <ev...@yahoo-inc.com> wrote: >> >>> I have not seen this before, if you could file a JIRA on this that >>> would be great. >>> >>> - Bobby >>> >>> From: Danijel Schiavuzzi <dani...@schiavuzzi.com> >>> Reply-To: "user@storm.incubator.apache.org" < >>> user@storm.incubator.apache.org> >>> Date: Wednesday, June 4, 2014 at 10:30 AM >>> To: "user@storm.incubator.apache.org" <user@storm.incubator.apache.org>, >>> "d...@storm.incubator.apache.org" <d...@storm.incubator.apache.org> >>> Subject: Trident transactional topology stuck re-emitting batches with >>> Netty, but running fine with ZMQ (was Re: Topology is stuck) >>> >>> Hi all, >>> >>> I've managed to reproduce the stuck topology problem and it seems it's >>> due to the Netty transport. Running with ZMQ transport enabled now and I >>> haven't been able to reproduce this. >>> >>> The problem is basically a Trident/Kafka transactional topology >>> getting stuck, i.e. re-emitting the same batches over and over again. This >>> happens after the Storm workers restart a few times due to Kafka spout >>> throwing RuntimeExceptions (because of the Kafka consumer in the spout >>> timing out with a SocketTimeoutException due to some temporary network >>> problems). Sometimes the topology is stuck after just one worker is >>> restarted, and sometimes a few worker restarts are needed to trigger the >>> problem. >>> >>> I simulated the Kafka spout socket timeouts by blocking network access >>> from Storm to my Kafka machines (with an iptables firewall rule). Most of >>> the time the spouts (workers) would restart normally (after re-enabling >>> access to Kafka) and the topology would continue to process batches, but >>> sometimes the topology would get stuck re-emitting batches after the >>> crashed workers restarted. Killing and re-submitting the topology manually >>> fixes this always, and processing continues normally. >>> >>> I haven't been able to reproduce this scenario after reverting my >>> Storm cluster's transport to ZeroMQ. With Netty transport, I can almost >>> always reproduce the problem by causing a worker to restart a number of >>> times (only about 4-5 worker restarts are enough to trigger this). >>> >>> Any hints on this? Anyone had the same problem? It does seem a serious >>> issue as it affect the reliability and fault tolerance of the Storm cluster. >>> >>> In the meantime, I'll try to prepare a reproducible test case for this. >>> >>> Thanks, >>> >>> Danijel >>> >>> >>> On Mon, Mar 31, 2014 at 4:39 PM, Danijel Schiavuzzi < >>> dani...@schiavuzzi.com> wrote: >>> >>>> To (partially) answer my own question -- I still have no idea on the >>>> cause of the stuck topology, but re-submitting the topology helps -- after >>>> re-submitting my topology is now running normally. >>>> >>>> >>>> On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi < >>>> dani...@schiavuzzi.com> wrote: >>>> >>>>> Also, I did have multiple cases of my IBackingMap workers dying >>>>> (because of RuntimeExceptions) but successfully restarting afterwards (I >>>>> throw RuntimeExceptions in the BackingMap implementation as my strategy in >>>>> rare SQL database deadlock situations to force a worker restart and to >>>>> fail+retry the batch). >>>>> >>>>> From the logs, one such IBackingMap worker death (and subsequent >>>>> restart) resulted in the Kafka spout re-emitting the pending tuple: >>>>> >>>>> 2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emitting >>>>> batch, attempt 29698959:736 >>>>> >>>>> This is of course the normal behavior of a transactional topology, >>>>> but this is the first time I've encountered a case of a batch retrying >>>>> indefinitely. This is especially suspicious since the topology has been >>>>> running fine for 20 days straight, re-emitting batches and restarting >>>>> IBackingMap workers quite a number of times. >>>>> >>>>> I can see in my IBackingMap backing SQL database that the batch with >>>>> the exact txid value 29698959 has been committed -- but I suspect that >>>>> could come from another BackingMap, since there are two BackingMap >>>>> instances running (paralellismHint 2). >>>>> >>>>> However, I have no idea why the batch is being retried indefinitely >>>>> now nor why it hasn't been successfully acked by Trident. >>>>> >>>>> Any suggestions on the area (topology component) to focus my research >>>>> on? >>>>> >>>>> Thanks, >>>>> >>>>> On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi < >>>>> dani...@schiavuzzi.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm having problems with my transactional Trident topology. It has >>>>>> been running fine for about 20 days, and suddenly is stuck processing a >>>>>> single batch, with no tuples being emitted nor tuples being persisted by >>>>>> the TridentState (IBackingMap). >>>>>> >>>>>> It's a simple topology which consumes messages off a Kafka queue. The >>>>>> spout is an instance of storm-kafka-0.8-plus >>>>>> TransactionalTridentKafkaSpout >>>>>> and I use the trident-mssql transactional TridentState implementation to >>>>>> persistentAggregate() data into a SQL database. >>>>>> >>>>>> In Zookeeper I can see Storm is re-trying a batch, i.e. >>>>>> >>>>>> "/transactional/<myTopologyName>/coordinator/currattempts" is >>>>>> "{"29698959":6487}" >>>>>> >>>>>> ... and the attempt count keeps increasing. It seems the batch with >>>>>> txid 29698959 is stuck, as the attempt count in Zookeeper keeps >>>>>> increasing >>>>>> -- seems like the batch isn't being acked by Trident and I have no idea >>>>>> why, especially since the topology has been running successfully the last >>>>>> 20 days. >>>>>> >>>>>> I did rebalance the topology on one occasion, after which it >>>>>> continued running normally. Other than that, no other modifications were >>>>>> done. Storm is at version 0.9.0.1. >>>>>> >>>>>> Any hints on how to debug the stuck topology? Any other useful info >>>>>> I might provide? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> Danijel Schiavuzzi >>>>>> >>>>>> E: dani...@schiavuzzi.com >>>>>> W: www.schiavuzzi.com >>>>>> T: +385989035562 >>>>>> Skype: danijel.schiavuzzi >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Danijel Schiavuzzi >>>>> >>>>> E: dani...@schiavuzzi.com >>>>> W: www.schiavuzzi.com >>>>> T: +385989035562 >>>>> Skype: danijel.schiavuzzi >>>>> >>>> >>>> >>>> >>>> -- >>>> Danijel Schiavuzzi >>>> >>>> E: dani...@schiavuzzi.com >>>> W: www.schiavuzzi.com >>>> T: +385989035562 >>>> Skype: danijels7 >>>> >>> >>> >>> >>> -- >>> Danijel Schiavuzzi >>> >>> E: dani...@schiavuzzi.com >>> W: www.schiavuzzi.com >>> T: +385989035562 >>> Skype: danijels7 >>> >> >> >> >> -- >> Danijel Schiavuzzi >> >> E: dani...@schiavuzzi.com >> W: www.schiavuzzi.com >> T: +385989035562 >> Skype: danijels7 >> > > -- Danijel Schiavuzzi E: dani...@schiavuzzi.com W: www.schiavuzzi.com T: +385989035562 Skype: danijels7