*@Spico: *The code as promised: http://nrecursions.blogspot.in/2016/05/more-concepts-of-apache-storm-you-need.html#morecreativetopologystructures *@Matthias:* Still no luck. I tried this in the bolt code: Map conf = Utils.readStormConfig(); conf.put("nimbus.leader", "localhost");
Also tried altering the storm.yaml file to have this: ########### These MUST be filled in for a storm configuration storm.zookeeper.servers: - "localhost" # - "server2" nimbus.seeds: ["localhost"] Am running this on LocalCluster, and strangely, the storm.yaml file is in my ~/eclipseworkspace/apache-storm-1.0.0_release/conf/ folder, although my project is in the ~/eclipseworkspace/MyStorm folder. Placed a copy of storm.yaml in my project folder and in the build/libs folder. Still no luck. For this person <http://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts>, it was a port issue. I don't think that's the case for me. Is there anything else that could be tried out? On Mon, May 9, 2016 at 6:18 PM, Matthias J. Sax <mj...@apache.org> wrote: > Utils.readStormConfig() tries to read "./storm.yaml" from local disc > (ie, supervisor machine that executes the bolt) -- as it is using > "working-directory" a guess it does not find the file, and thus value > "nimbus.host" is not set. > > Make sure that storm.yaml is found be the worker, or set nimbus.host > manually in your bolt code: > > conf.put("nimbus.host", "<your-nimbus-host-name>"); > > (or "nimbus.leader" that replaces "nimbus.host" in Storm 1.0.0 > > > -Matthias > > On 05/09/2016 12:31 PM, Navin Ipe wrote: > > @Spico: Will share. > > > > The streams implementation is working beautifully. > > Only the topology killing is failing. > > > > *Tried:* > > Map conf = Utils.readStormConfig(); > > NimbusClient cc = > > NimbusClient.getConfiguredClient(conf); > > Nimbus.Client client = cc.getClient(); > > client.killTopology("myStorm"); > > > > *I get these errors:* > > 29442 [Thread-32-topologyKillerBolt-executor[16 16]] WARN > > o.a.s.u.NimbusClient - Ignoring exception while trying to get leader > > nimbus info from localhost. will retry with a different seed host. > > java.lang.RuntimeException: > > org.apache.storm.thrift.transport.TTransportException: > > java.net.ConnectException: Connection refused > > Caused by: org.apache.storm.thrift.transport.TTransportException: > > java.net.ConnectException: Connection refused > > Caused by: java.net.ConnectException: Connection refused > > 29445 [Thread-32-topologyKillerBolt-executor[16 16]] ERROR o.a.s.util - > > Async loop died! > > java.lang.RuntimeException: > > org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find > > leader nimbus from seed hosts [localhost]. Did you specify a valid list > > of nimbus hosts for config nimbus.seeds? > > Caused by: org.apache.storm.utils.NimbusLeaderNotFoundException: Could > > not find leader nimbus from seed hosts [localhost]. Did you specify a > > valid list of nimbus hosts for config nimbus.seeds? > > 29462 [Thread-32-topologyKillerBolt-executor[16 16]] ERROR o.a.s.util - > > Halting process: ("Worker died") > > java.lang.RuntimeException: ("Worker died") > > > > The error is apparently on this line: NimbusClient cc = > > NimbusClient.getConfiguredClient(conf); > > > > > > > > On Mon, May 9, 2016 at 3:15 PM, Spico Florin <spicoflo...@gmail.com > > <mailto:spicoflo...@gmail.com>> wrote: > > > > Hi! > > You welcome Navine. I'm also interested in the solution. Can you > > please share your remarks and (some code :)) after the > implementation? > > Thanks. > > Regards,\ > > Florin > > > > On Mon, May 9, 2016 at 7:20 AM, Navin Ipe > > <navin....@searchlighthealth.com > > <mailto:navin....@searchlighthealth.com>> wrote: > > > > @Matthias: That's genius! I didn't know streams and allGroupings > > could be used like that. > > In the way Storm introduced tick tuples, it'd have been nice if > > Storm had a native technique of doing all this, but the ideas > > you've come up with are extremely good. Am going to try > > implementing them right away. > > Thank you too Florin! > > > > On Mon, May 9, 2016 at 12:48 AM, Matthias J. Sax > > <mj...@apache.org <mailto:mj...@apache.org>> wrote: > > > > To synchronize this, use an additional "shut down bolt" that > > used > > parallelism of one. "shut down bolt" must be notified by all > > parallel > > DbBolts after they performed the flush. If all notifications > are > > received, there are not in-flight message and thus "shut > > down bolt" can > > kill the topology safely. > > > > -Matthias > > > > > > > > On 05/08/2016 07:27 PM, Spico Florin wrote: > > > hi! > > > there is this solution of sending a poison pill message > from the > > > spout. on bolt wil receiv your poison pill and will kill > topology via > > > storm storm nimbus API. one potentential issue whith this > approach is > > > that due to your topology structure regarding the > parralelism of your > > > bolts nd the time required by themto excute their > bussineess logic, is > > > that the poison pill to be swallowed by the one bolt > responsilble for > > > killing the topology, before all the other messages that > are in-flight > > > to be processed. the conseuence is that you cannot be sure > that all the > > > messagess sent by the spout were processed. also sharing > the total > > > number of sent messages between the excutors in order to > shutdown when > > > all messages were processed coul be error prone since > tuple can be > > > processed many times (depending on your guaranteee message > processing) > > > or they could be failed. > > > i coul not find a solution for this. storm is intended > to run > > > forunbounded data. > > > i hope that thrse help, > > > regard, > > > florin > > > > > > > > > On Sunday, May 8, 2016, Matthias J. Sax <mj...@apache.org > <mailto:mj...@apache.org> > > > <mailto:mj...@apache.org <mailto:mj...@apache.org>>> > wrote: > > > > > > You can get the number of bolt instances from > > TopologyContext that is > > > provided in Bolt.prepare() > > > > > > Furthermore, you could put a loop into your topology, > > ie, a bolt reads > > > it's own output; if you broadcast (ie, allGrouping) > this > > > feedback-loop-stream you can let bolt instances talk > > to each other. > > > > > > builder.setBolt("DbBolt", new MyDBBolt()) > > > .shuffleGrouping("spout") > > > .allGrouping("flush-stream", "DbBolt"); > > > > > > where "flush-stream" is a second output stream of > > MyDBBolt() sending a > > > notification tuple after it received the end-of-stream > > from spout; > > > furthermore, if a bolt received the signal via > > "flush-stream" from > > > **all** parallel bolt instances, it can flush to DB. > > > > > > Or something like this... Be creative! :) > > > > > > > > > -Matthias > > > > > > > > > On 05/08/2016 02:26 PM, Navin Ipe wrote: > > > > @Matthias: I agree about the batch processor, but my > > superior took the > > > > decision to use Storm, and he visualizes more > > complexity later for > > > which > > > > he needs Storm. > > > > I had considered the "end of stream" tuple earlier > > (my idea was to > > > emit > > > > 10 consecutive nulls), but then the question was how > > do I know how > > > many > > > > bolt instances have been created, and how do I > > notify all the bolts? > > > > Because it's only after the last bolt finishes > > writing to DB, that I > > > > have to shut down the topology. > > > > > > > > @Jason: Thanks. I had seen storm signals earlier (I > > think from one of > > > > your replies to someone else) and I had a look at > > the code too, > > > but am a > > > > bit wary because it's no longer being maintained and > > because of the > > > > issues: > https://github.com/ptgoetz/storm-signals/issues > > > > > > > > On Sun, May 8, 2016 at 5:40 AM, Jason Kusar > > <ja...@kusar.net <mailto:ja...@kusar.net> > > > <javascript:;> > > > > <mailto:ja...@kusar.net <mailto:ja...@kusar.net> > <javascript:;>>> wrote: > > > > > > > > You might want to check out Storm Signals. > > > > https://github.com/ptgoetz/storm-signals > > > > > > > > It might give you what you're looking for. > > > > > > > > > > > > On Sat, May 7, 2016, 11:59 AM Matthias J. Sax > > > <mj...@apache.org <mailto:mj...@apache.org> > <javascript:;> > > > > <mailto:mj...@apache.org > > <mailto:mj...@apache.org> <javascript:;>>> wrote: > > > > > > > > As you mentioned already: Storm is designed > > to run topologies > > > > forever ;) > > > > If you have finite data, why do you not use > > a batch > > > processor??? > > > > > > > > As a workaround, you can embed "control > > messages" in your > > > stream > > > > (or use > > > > an additional stream for them). > > > > > > > > If you want a topology to shut down itself, > > you could use > > > > > > > > > > `NimbusClient.getConfiguredClient(conf).getClient().killTopology(name);` > > > > in your spout/bolt code. > > > > > > > > Something like: > > > > - Spout emit all tuples > > > > - Spout emit special "end of stream" > > control tuple > > > > - Bolt1 processes everything > > > > - Bolt1 forward "end of stream" control > > tuple (when it > > > received it) > > > > - Bolt2 processed everything > > > > - Bolt2 receives "end of stream" control > > tuple => flush to DB > > > > => kill > > > > topology > > > > > > > > But I guess, this is kinda weird pattern. > > > > > > > > -Matthias > > > > > > > > On 05/05/2016 06:13 AM, Navin Ipe wrote: > > > > > Hi, > > > > > > > > > > I know Storm is designed to run forever. I > > also know about > > > > Trident's > > > > > technique of aggregation. But shouldn't > > Storm have a way to > > > > let bolts > > > > > know that a certain bunch of processing > > has been completed? > > > > > > > > > > Consider this topology: > > > > > Spout------>Bolt-A------>Bolt-B > > > > > | |--->Bolt-B > > > > > | \--->Bolt-B > > > > > |--->Bolt-A------>Bolt-B > > > > > | |--->Bolt-B > > > > > | \--->Bolt-B > > > > > \--->Bolt-A------>Bolt-B > > > > > |--->Bolt-B > > > > > \--->Bolt-B > > > > > > > > > > * From Bolt-A to Bolt-B, it is a > > FieldsGrouping. > > > > > * Spout emits only a few tuples and then > > stops emitting. > > > > > * Bolt A takes those tuples and > > generates millions of > > > tuples. > > > > > > > > > > > > > > > *Bolt-B accumulates tuples that Bolt A > > sends and needs > > > to know > > > > when > > > > > Spout finished emitting. Only then can > > Bolt-B start > > > writing to > > > > SQL.* > > > > > > > > > > *Questions:* > > > > > 1. How can all Bolts B be notified that it > > is time to > > > write to > > > > SQL? > > > > > 2. After all Bolts B have written to SQL, > > how to know > > > that all > > > > Bolts B > > > > > have completed writing? > > > > > 3. How to stop the topology? I know of > > > > > localCluster.killTopology("HelloStorm"), > > but shouldn't there > > > > be a way to > > > > > do it from the Bolt? > > > > > > > > > > -- > > > > > Regards, > > > > > Navin > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Navin > > > > > > > > > > > > > -- > > Regards, > > Navin > > > > > > > > > > > > -- > > Regards, > > Navin > > -- Regards, Navin