Thanks Srikanth, you were right, was using pieces of sample codes and the shutdown was at the end of the topology.
On 3 May 2015, at 17:44, Srikanth <srikanth...@gmail.com> wrote: > Is this your own code or did you copy it from somewhere? Most sample code you > find online will run in local cluster only for short duration. > > See if your code does something like this > > cluster.submitTopology(); > Thread.sleep(1000); > cluster.shutdown(); ---> This will bring your process down > > Srikanth > > On Sun, May 3, 2015 at 12:05 PM, Joan Perez Esteban <corleon...@gmail.com> > wrote: > Thanks ageing Jungtaek, > Do you happen to know in where should I look at?, or if there are any > parameters I can tune to improve efficiency? maybe is shutting down the > process to save resources. > > rgds, > > Juan > > > On 3 May 2015, at 15:29, 임정택 <kabh...@gmail.com> wrote: > >> I don't find any suspicious things from logs. >> Maybe nimbus was being shutdown gracefully (I mean normally), so you would >> like to check it. >> >> Jungtaek Lim (HeartSaVioR) >> >> 2015-05-03 20:26 GMT+09:00 Joan Perez Esteban <corleon...@gmail.com>: >> Hi Jungtaek, >> >> Storm version is 0.9.2-incubating, I am running it locally in a virtual >> machine is just test environment. >> About the logs from Nimbus where are they stored? >> >> This is all I have in the console from other test: >> >> 36863 [main] INFO backtype.storm.daemon.nimbus - Shutting down master >> 36868 [main] INFO backtype.storm.daemon.nimbus - Shut down master >> 36872 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN >> org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception >> org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read >> additional data from client sessionid 0x14d15efbfbf0005, likely client has >> closed socket >> at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) >> ~[zookeeper-3.4.5.jar:3.4.5-1392090] >> at >> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) >> [zookeeper-3.4.5.jar:3.4.5-1392090] >> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55] >> 36873 [main] INFO backtype.storm.daemon.supervisor - Shutting down >> 9bd1c53b-9583-484d-9c0e-6b9208bbe130:a958afaf-04da-472f-a595-969ed155ccf2 >> 36874 [main] INFO backtype.storm.process-simulator - Killing process >> 8b0be90a-a805-4f46-9fe5-402b16eda600 >> 36874 [main] INFO backtype.storm.daemon.worker - Shutting down worker >> sentence-count-1-1430592148 9bd1c53b-9583-484d-9c0e-6b9208bbe130 1024 >> 36874 [main] INFO backtype.storm.daemon.worker - Shutting down receive >> thread >> 36875 [main] INFO backtype.storm.messaging.loader - Shutting down >> receiving-thread: [sentence-count-1-1430592148, 1024] >> 36875 [main] INFO backtype.storm.messaging.loader - Waiting for >> receiving-thread:[sentence-count-1-1430592148, 1024] to die >> 36875 [Thread-25-worker-receiver-thread-0] INFO >> backtype.storm.messaging.loader - >> Receiving-thread:[sentence-count-1-1430592148, 1024] received shutdown notice >> 36876 [main] INFO backtype.storm.messaging.loader - Shutdown >> receiving-thread: [sentence-count-1-1430592148, 1024] >> 36876 [main] INFO backtype.storm.daemon.worker - Shut down receive thread >> 36876 [main] INFO backtype.storm.daemon.worker - Terminating messaging >> context >> 36877 [main] INFO backtype.storm.daemon.worker - Shutting down executors >> 36877 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> count-bolt:[2 2] >> 36877 [Thread-7-disruptor-executor[2 2]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36877 [Thread-8-count-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36878 [main] INFO backtype.storm.daemon.executor - Shut down executor >> count-bolt:[2 2] >> 36878 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> count-bolt:[3 3] >> 36878 [Thread-10-count-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36878 [Thread-9-disruptor-executor[3 3]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36879 [main] INFO backtype.storm.daemon.executor - Shut down executor >> count-bolt:[3 3] >> 36879 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> count-bolt:[4 4] >> 36879 [Thread-11-disruptor-executor[4 4]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36880 [Thread-12-count-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36880 [main] INFO backtype.storm.daemon.executor - Shut down executor >> count-bolt:[4 4] >> 36880 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> report-bolt:[5 5] >> 36881 [Thread-13-disruptor-executor[5 5]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36881 [Thread-14-report-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36881 [main] INFO backtype.storm.daemon.executor - Shut down executor >> report-bolt:[5 5] >> 36882 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> report-bolt:[6 6] >> 36882 [Thread-15-disruptor-executor[6 6]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36882 [Thread-16-report-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36883 [main] INFO backtype.storm.daemon.executor - Shut down executor >> report-bolt:[6 6] >> 36883 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> report-bolt:[7 7] >> 36883 [Thread-18-report-bolt] INFO backtype.storm.util - Async loop >> interrupted! >> 36883 [Thread-17-disruptor-executor[7 7]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36884 [main] INFO backtype.storm.daemon.executor - Shut down executor >> report-bolt:[7 7] >> 36884 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> sentence-spout:[8 8] >> 36884 [Thread-20-sentence-spout] INFO backtype.storm.util - Async loop >> interrupted! >> 36884 [Thread-19-disruptor-executor[8 8]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36885 [main] INFO backtype.storm.daemon.executor - Shut down executor >> sentence-spout:[8 8] >> 36885 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> __system:[-1 -1] >> 36885 [Thread-22-__system] INFO backtype.storm.util - Async loop >> interrupted! >> 36885 [Thread-21-disruptor-executor[-1 -1]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36886 [main] INFO backtype.storm.daemon.executor - Shut down executor >> __system:[-1 -1] >> 36886 [main] INFO backtype.storm.daemon.executor - Shutting down executor >> __acker:[1 1] >> 36886 [Thread-24-__acker] INFO backtype.storm.util - Async loop interrupted! >> 36886 [Thread-23-disruptor-executor[1 1]-send-queue] INFO >> backtype.storm.util - Async loop interrupted! >> 36887 [main] INFO backtype.storm.daemon.executor - Shut down executor >> __acker:[1 1] >> 36887 [main] INFO backtype.storm.daemon.worker - Shut down executors >> 36887 [main] INFO backtype.storm.daemon.worker - Shutting down transfer >> thread >> 36887 [Thread-26-disruptor-worker-transfer-queue] INFO backtype.storm.util >> - Async loop interrupted! >> 36888 [main] INFO backtype.storm.daemon.worker - Shut down transfer thread >> 36889 [main] INFO backtype.storm.daemon.worker - Shutting down default >> resources >> 36889 [main] INFO backtype.storm.daemon.worker - Shut down default resources >> 36896 [main] INFO backtype.storm.daemon.worker - Disconnecting from storm >> cluster state context >> 36898 [main] INFO backtype.storm.daemon.worker - Shut down worker >> sentence-count-1-1430592148 9bd1c53b-9583-484d-9c0e-6b9208bbe130 1024 >> 36901 [main] INFO backtype.storm.daemon.supervisor - Shut down >> 9bd1c53b-9583-484d-9c0e-6b9208bbe130:a958afaf-04da-472f-a595-969ed155ccf2 >> 36902 [main] INFO backtype.storm.daemon.supervisor - Shutting down >> supervisor 9bd1c53b-9583-484d-9c0e-6b9208bbe130 >> 36902 [Thread-3] INFO backtype.storm.event - Event manager interrupted >> 36903 [Thread-4] INFO backtype.storm.event - Event manager interrupted >> 36905 [main] INFO backtype.storm.daemon.supervisor - Shutting down >> supervisor 10e953ec-1832-4f3d-b8f4-b2c1f46857ab >> 36906 [Thread-5] INFO backtype.storm.event - Event manager interrupted >> 36906 [Thread-6] INFO backtype.storm.event - Event manager interrupted >> 36908 [main] INFO backtype.storm.testing - Shutting down in process >> zookeeper >> 36909 [main] INFO backtype.storm.testing - Done shutting down in process >> zookeeper >> 36909 [main] INFO backtype.storm.testing - Deleting temporary path >> /tmp/d5babc7f-2c83-4e2f-9489-05fe73157fed >> 36910 [main] INFO backtype.storm.testing - Deleting temporary path >> /tmp/1ee9e61c-4496-4e05-ae5b-305d88e3c294 >> 36911 [main] INFO backtype.storm.testing - Deleting temporary path >> /tmp/35c83b74-5dce-4efd-9be5-73ee625d3098 >> 36912 [main] INFO backtype.storm.testing - Deleting temporary path >> /tmp/edd7365c-334e-40b8-ac29-4eb2812c945f >> >> >> Thanks for your help, >> >> Juan >> >> >> On 3 May 2015, at 00:17, 임정택 <kabh...@gmail.com> wrote: >> >>> Hi. >>> >>> Could you elavorate it? >>> >>> - What's your Storm version? >>> - Did you run your topology with local mode or cluster mode? >>> - Is there no Exception and Stack trace logs from Nimbus? >>> - Could you share more log lines, or full Nimbus log file? >>> >>> Regards >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> 2015-05-03 2:23 GMT+09:00 Joan Perez Esteban <corleon...@gmail.com>: >>> Hi and thanks in advance, >>> >>> I am trying to perform a log analyser with storm, starting from the >>> beginning I am just reading a file and sending lines (spout) receiving bolt >>> just save the lines to a new file. >>> As simple as that by now, I want to compare origin and destination files >>> and check they are the same. >>> Problem is that when about 52K are written storm stops with following error: >>> >>> >>> 34870 [main] INFO backtype.storm.daemon.nimbus - Shutting down master >>> 34876 [main] INFO backtype.storm.daemon.nimbus - Shut down master >>> 34880 [main] INFO backtype.storm.daemon.supervisor - Shutting down >>> 74037900-6997-4732-a987-a5f996db5928:b5e20ba0-33eb-4236-bc38-c323a840cac1 >>> 34880 [main] INFO backtype.storm.process-simulator - Killing process >>> db74c56d-6e27-4cc6-91d8-b4ac66a552ae >>> 34881 [main] INFO backtype.storm.daemon.worker - Shutting down worker >>> sentence-count-1-1430586398 74037900-6997-4732-a987-a5f996db5928 1024 >>> 34881 [main] INFO backtype.storm.daemon.worker - Shutting down receive >>> thread >>> 34881 [main] INFO backtype.storm.messaging.loader - Shutting down >>> receiving-thread: [sentence-count-1-1430586398, 1024] >>> 34882 [Thread-19-worker-receiver-thread-0] INFO >>> backtype.storm.messaging.loader - >>> Receiving-thread:[sentence-count-1-1430586398, 1024] received shutdown >>> notice >>> 34882 [main] INFO backtype.storm.messaging.loader - Waiting for >>> receiving-thread:[sentence-count-1-1430586398, 1024] to die >>> 34883 [main] INFO backtype.storm.messaging.loader - Shutdown >>> receiving-thread: [sentence-count-1-1430586398, 1024] >>> 34883 [main] INFO backtype.storm.daemon.worker - Shut down receive thread >>> 34883 [main] INFO backtype.storm.daemon.worker - Terminating messaging >>> context >>> 34884 [main] INFO backtype.storm.daemon.worker - Shutting down executors >>> 34884 [main] INFO backtype.storm.daemon.executor - Shutting down executor >>> report-bolt:[2 2] >>> 34884 [Thread-8-report-bolt] INFO backtype.storm.util - Async loop >>> interrupted! >>> 34884 [Thread-7-disruptor-executor[2 2]-send-queue] INFO >>> backtype.storm.util - Async loop interrupted! >>> 34885 [main] INFO backtype.storm.daemon.executor - Shut down executor >>> report-bolt:[2 2] >>> 34885 [main] INFO backtype.storm.daemon.executor - Shutting down executor >>> report-bolt:[3 3] >>> 34885 [Thread-9-disruptor-executor[3 3]-send-queue] INFO >>> backtype.storm.util - Async loop interrupted! >>> 34886 [Thread-10-report-bolt] INFO backtype.storm.util - Async loop >>> interrupted! >>> 34886 [main] INFO backtype.storm.daemon.executor - Shut down executor >>> report-bolt:[3 3] >>> 34886 [main] INFO backtype.storm.daemon.executor - Shutting down executor >>> report-bolt:[4 4] >>> 34886 [Thread-11-disruptor-executor[4 4]-send-queue] INFO >>> backtype.storm.util - Async loop interrupted! >>> 34887 [Thread-12-report-bolt] INFO backtype.storm.util - Async loop >>> interrupted! >>> 34887 [main] INFO backtype.storm.daemon.executor - Shut down executor >>> report-bolt:[4 4] >>> 34887 [main] INFO backtype.storm.daemon.executor - Shutting down executor >>> sentence-spout:[5 5] >>> 34887 [Thread-14-sentence-spout] INFO backtype.storm.util - Async loop >>> interrupted! >>> >>> >>> >>> Any idea why this may be happening? >>> >>> Rgds, >>> >>> Juan >>> >>> >>> >>> -- >>> Name : 임 정택 >>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >>> Twitter : http://twitter.com/heartsavior >>> LinkedIn : http://www.linkedin.com/in/heartsavior >> >> >> >> >> -- >> Name : 임 정택 >> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >> Twitter : http://twitter.com/heartsavior >> LinkedIn : http://www.linkedin.com/in/heartsavior > >
