Are you using LocalCluster.shutdown or killTopology in the main method once you submit the topology? From the logs it looks like that…
Thanks & Regards Dintu Alex John From: sam mohel [mailto:[email protected]] Sent: Thursday, July 27, 2017 2:54 PM To: [email protected]; [email protected] Subject: Re: Setting heap size parameters by workers.childopts and supervisor.childopts i forgot to mention that i tried to increase topology.message.timeout.secs to 180 but didn't work too On Thu, Jul 27, 2017 at 9:52 PM, sam mohel <[email protected]<mailto:[email protected]>> wrote: i tried to use debug . got in the worker.log.err 2017-07-27 21:47:48,868 FATAL Unable to register shutdown hook because JVM is shutting down. and this lines from worker.log 2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-1:27, stream: __ack_ack, id: {}, [3247365064986003851 -431522470795602124] 2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-1:27, stream: __ack_ack, id: {}, [3247365064986003851 -431522470795602124] 2017-07-27 21:47:48.811 b.s.d.executor [INFO] Execute done TUPLE source: b-1:27, stream: __ack_ack, id: {}, [3247365064986003851 -431522470795602124] TASK: 1 DELTA: 0 2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-1:29, stream: __ack_ack, id: {}, [3247365064986003851 -6442207219333745818] 2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-1:29, stream: __ack_ack, id: {}, [3247365064986003851 -6442207219333745818] 2017-07-27 21:47:48.811 b.s.d.executor [INFO] Execute done TUPLE source: b-1:29, stream: __ack_ack, id: {}, [3247365064986003851 -6442207219333745818] TASK: 1 DELTA: 0 2017-07-27 21:47:48.811 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 5263752373603294688] 2017-07-27 21:47:48.811 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 5263752373603294688] 2017-07-27 21:47:48.868 b.s.d.worker [INFO] Shutting down worker top-1-1501184820 9adf5f4c-dc5b-47b5-a458-40defe84fe9e 6703 2017-07-27 21:47:48.868 b.s.d.worker [INFO] Shutting down receive thread 2017-07-27 21:47:48.869 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-1:31, stream: __ack_ack, id: {}, [3247365064986003851 4288963968930353157] 2017-07-27 21:47:48.872 b.s.d.executor [INFO] Execute done TUPLE source: b-1:31, stream: __ack_ack, id: {}, [3247365064986003851 4288963968930353157] TASK: 1 DELTA: 60 2017-07-27 21:47:48.872 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 5240959063117469257] 2017-07-27 21:47:48.872 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 5240959063117469257] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 5240959063117469257] TASK: 1 DELTA: 1 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 7583382518734849127] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 7583382518734849127] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 7583382518734849127] TASK: 1 DELTA: 0 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 6840644970823833210] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 6840644970823833210] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Execute done TUPLE source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 6840644970823833210] TASK: 1 DELTA: 0 2017-07-27 21:47:48.873 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 -6463368911496394080] 2017-07-27 21:47:48.873 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 -6463368911496394080] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 -6463368911496394080] TASK: 1 DELTA: 1 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 764549587969230513] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 764549587969230513] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source: b-3:33, stream: __ack_ack, id: {}, [3247365064986003851 764549587969230513] TASK: 1 DELTA: 0 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 -4632707886455738545] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 -4632707886455738545] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Execute done TUPLE source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 -4632707886455738545] TASK: 1 DELTA: 0 2017-07-27 21:47:48.874 b.s.d.executor [INFO] Processing received message FOR 1 TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 2993206175355277727] 2017-07-27 21:47:48.874 b.s.d.executor [INFO] BOLT ack TASK: 1 TIME: 0 TUPLE: source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 2993206175355277727] 2017-07-27 21:47:48.875 b.s.d.executor [INFO] Execute done TUPLE source: b-5:35, stream: __ack_ack, id: {}, [3247365064986003851 2993206175355277727] TASK: 1 DELTA: 1 2017-07-27 21:47:48.898 b.s.m.n.Client [INFO] creating Netty Client, connecting to lenovo:6703, bufferSize: 5242880 2017-07-27 21:47:48.902 b.s.m.loader [INFO] Shutting down receiving-thread: [top-1-1501184820, 6703] 2017-07-27 21:47:48.902 b.s.m.n.Client [INFO] closing Netty Client Netty-Client-lenovo/192.168.1.5:6703<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.1.5%3A6703&data=01%7C01%7CDintu.John%40searshc.com%7C4f35423d6f0a407ca6a808d4d52939e0%7C27e4c16803234463acad7e124b566726%7C0&sdata=TNXTB4ML7m%2BccFEk%2BIFdkIxZ1QUD7PTNs0i%2F%2Bj4HH%2BA%3D&reserved=0> 2017-07-27 21:47:48.902 b.s.m.n.Client [INFO] waiting up to 600000 ms to send 0 pending messages to Netty-Client-lenovo/192.168.1.5:6703<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.1.5%3A6703&data=01%7C01%7CDintu.John%40searshc.com%7C4f35423d6f0a407ca6a808d4d52939e0%7C27e4c16803234463acad7e124b566726%7C0&sdata=TNXTB4ML7m%2BccFEk%2BIFdkIxZ1QUD7PTNs0i%2F%2Bj4HH%2BA%3D&reserved=0> 2017-07-27 21:47:48.902 b.s.m.loader [INFO] Waiting for receiving-thread:[top-1-1501184820, 6703] to die 2017-07-27 21:47:48.903 b.s.m.loader [INFO] Shutdown receiving-thread: [top-1-1501184820, 6703] 2017-07-27 21:47:48.904 b.s.d.worker [INFO] Shut down receive thread 2017-07-27 21:47:48.904 b.s.d.worker [INFO] Terminating messaging context 2017-07-27 21:47:48.904 b.s.d.worker [INFO] Shutting down executors 2017-07-27 21:47:48.904 b.s.d.executor [INFO] Shutting down executor b-0:[8 8] 2017-07-27 21:47:48.905 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.905 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.906 b.s.d.executor [INFO] Shut down executor b-0:[8 8] 2017-07-27 21:47:48.906 b.s.d.executor [INFO] Shutting down executor b-8:[47 47] 2017-07-27 21:47:48.907 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.907 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shut down executor b-8:[47 47] 2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shutting down executor b-0:[12 12] 2017-07-27 21:47:48.908 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.908 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shut down executor b-0:[12 12] 2017-07-27 21:47:48.908 b.s.d.executor [INFO] Shutting down executor b-8:[54 54] 2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shut down executor b-8:[54 54] 2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shutting down executor b-0:[2 2] 2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shut down executor b-0:[2 2] 2017-07-27 21:47:48.909 b.s.d.executor [INFO] Shutting down executor b-2:[32 32] 2017-07-27 21:47:48.909 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.910 b.s.util [INFO] Async loop interrupted! 2017-07-27 21:47:48.910 b.s.d.executor [INFO] Shut down executor b-2:[32 32] 2017-07-27 21:47:48.910 b.s.d.executor [INFO] Shutting down executor b-8:[41 41] 2017-07-27 21:47:48.910 b.s.util [INFO] Asy On Thu, Jul 27, 2017 at 3:11 PM, Stig Rohde Døssing <[email protected]<mailto:[email protected]>> wrote: Yes, there is topology.message.timeout.secs for setting how long the topology has to process a message after it is emitted from the spout, and topology.enable.message.timeouts if you want to disable timeouts entirely. I'm assuming that's what you're asking? 2017-07-27 15:03 GMT+02:00 sam mohel <[email protected]<mailto:[email protected]>>: Thanks for your patience and time. I will use debug now . But is there any settings or configurations about the time for spout? How can I increase it to try ? On Thursday, July 27, 2017, Stig Rohde Døssing <[email protected]<mailto:[email protected]>> wrote: > Last message accidentally went to you directly instead of the mailing list. > > Never mind what I wrote about worker slots. I think you should check that all > tuples are being acked first. Then you might want to try enabling debug > logging. You should also verify that your spout is emitting all the expected > tuples. Since you're talking about a result file, I'm assuming your spout > output is limited. > > 2017-07-27 10:36 GMT+02:00 Stig Rohde Døssing > <[email protected]<mailto:[email protected]>>: >> >> Okay. Unless you're seeing out of memory errors or know that your garbage >> collector is thrashing, I don't know why changing your xmx would help. >> Without knowing more about your topology it's hard to say what's going >> wrong. I think your best bet is to enable debug logging and try to figure >> out what happens when the topology stops writing to your result file. When >> you run your topology on a distributed cluster, you can use Storm UI to >> verify that all your tuples are being acked, maybe your tuple trees are not >> being acked correctly? >> >> Multiple topologies shouldn't be interfering with each other, the only thing >> I can think of is if you have too few worker slots and some of your >> topology's components are not being assigned to a worker. You can see this >> as well in Storm UI. >> >> 2017-07-27<tel:20%2017%2007%2027> 8:11 GMT+02:00 sam mohel >> <[email protected]<mailto:[email protected]>>: >>> >>> Yes I tried 2048 and 4096 to make worker more size but same problem . >>> >>> I have result file . It should contains the result of my processing . The >>> size of this file should be 7 mb but what I got after sumbit the topology >>> 50 kb only . >>> >>> I submitted this toplogy before . Since 4 months . But when I submitted it >>> now I got this problem . >>> >>> How the toplogy working well before but now not ? >>> >>> Silly question and sorry for that >>> I submitted three topology except that one . Is that make memory weak ? Or >>> should I clean something after that >>> >>> On Thursday, July 27, 2017, Stig Rohde Døssing >>> <[email protected]<mailto:[email protected]>> wrote: >>> > As far as I can tell the default xmx for workers in 0.10.2 is 768 megs >>> > (https://github.com/apache/storm/blob/v0.10.2/conf/defaults.yaml#L134<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fstorm%2Fblob%2Fv0.10.2%2Fconf%2Fdefaults.yaml%23L134&data=01%7C01%7CDintu.John%40searshc.com%7C4f35423d6f0a407ca6a808d4d52939e0%7C27e4c16803234463acad7e124b566726%7C0&sdata=2FbjUiuZzBiLXlGerVR5zVn6bKBhSqu763vHKE8XnpA%3D&reserved=0>), >>> > your supervisor logs shows the following: >>> > "Launching worker with command: <snip> -Xmx2048m". Is this the right >>> > configuration? >>> > >>> > Regarding the worker log, it looks like the components are initialized >>> > correctly, all the bolts report that they're done running prepare(). >>> > Could you explain what you expect the logs to look like and what you >>> > expect to happen when you run the topology? >>> > >>> > It's sometimes helpful to enable debug logging if your topology acts >>> > strange, consider trying that by setting >>> > Config conf = new Config(); >>> > conf.setDebug(true); >>> > >>> > 2017-07-27<tel:20%2017%2007%2027> 1:43 GMT+02:00 sam mohel >>> > <[email protected]<mailto:[email protected]>>: >>> >> >>> >> Same problem with distributed mode . I tried to submit toplogy in >>> >> distributed with localhost and attached log files of worker and >>> >> supervisor >>> >> >>> >> >>> >> >>> >> On Thursday, July 27, 2017, sam mohel >>> >> <[email protected]<mailto:[email protected]>> wrote: >>> >> > I submit my topology by this command >>> >> > mvn package >>> >> > mvn compile exec:java -Dexec.classpathScope=compile >>> >> > -Dexec.mainClass=trident.Topology >>> >> > and i copied those lines >>> >> > 11915 [Thread-47-b-4] INFO b.s.d.executor - Prepared bolt b-4:(40) >>> >> > 11912 [Thread-111-b-2] INFO b.s.d.executor - Prepared bolt b-2:(14) >>> >> > 11934 [Thread-103-b-5] INFO b.s.d.executor - Prepared bolt b-5:(45) >>> >> > sam@lenovo:~/first-topology$ >>> >> > from what i saw in terminal . I checked the size of the result file >>> >> > and found it's 50 KB each time i submit it . >>> >> > what should i check ? >>> >> > On Wed, Jul 26, 2017 at 9:05 PM, Bobby Evans >>> >> > <[email protected]<mailto:[email protected]>> wrote: >>> >> >> >>> >> >> Local mode is totally separate and there are no processes launched >>> >> >> except the original one. Those values are ignored in local mode. >>> >> >> >>> >> >> >>> >> >> - Bobby >>> >> >> >>> >> >> >>> >> >> On Wednesday, July 26, 2017, 2:01:52 PM CDT, sam mohel >>> >> >> <[email protected]<mailto:[email protected]>> wrote: >>> >> >> >>> >> >> Thanks so much for replying , i tried to submit topology in local >>> >> >> mode ... i increased size of worker like >>> >> >> conf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx4096m" ); >>> >> >> >>> >> >> but got in terminal >>> >> >> 11920 [Thread-121-b-4] INFO b.s.d.executor - Preparing bolt b-4:(25) >>> >> >> 11935 [Thread-121-b-4] INFO b.s.d.executor - Prepared bolt b-4:(25) >>> >> >> 11920 [Thread-67-b-5] INFO b.s.d.executor - Preparing bolt b-5:(48) >>> >> >> 11936 [Thread-67-b-5] INFO b.s.d.executor - Prepared bolt b-5:(48) >>> >> >> 11919 [Thread-105-b-2] INFO b.s.d.executor - Prepared bolt b-2:(10) >>> >> >> 11915 [Thread-47-b-4] INFO b.s.d.executor - Prepared bolt b-4:(40) >>> >> >> 11912 [Thread-111-b-2] INFO b.s.d.executor - Prepared bolt b-2:(14) >>> >> >> 11934 [Thread-103-b-5] INFO b.s.d.executor - Prepared bolt b-5:(45) >>> >> >> sam@lenovo:~/first-topology$ >>> >> >> and didn't complete processing . the size of the result is 50 KB . >>> >> >> This topology was working well without any problems . But when i >>> >> >> tried to submit it now , i didn't get the full result >>> >> >> >>> >> >> On Wed, Jul 26, 2017 at 8:35 PM, Bobby Evans >>> >> >> <[email protected]<mailto:[email protected]>> wrote: >>> >> >> >>> >> >> worker.childops is the default value that is set by the system >>> >> >> administrator in storm.yaml on each of the supervisor nodes. >>> >> >> topology.worker.childopts is what you set in your topology conf if >>> >> >> you want to add something more to the command line. >>> >> >> >>> >> >> >>> >> >> - Bobby >>> >> >> >>> >> >> >>> >> >> On Tuesday, July 25, 2017, 11:50:04 PM CDT, sam mohel >>> >> >> <[email protected]<mailto:[email protected]>> wrote: >>> >> >> >>> >> >> i'm using 0.10.2 version . i tried to write in the code >>> >> >> conf.put(Config.WORKER_ CHILDOPTS, "-Xmx4g"); >>> >> >> conf.put(Config.SUPERVISOR_ CHILDOPTS, "-Xmx4g"); >>> >> >> >>> >> >> but i didn't touch any affect . Did i write the right configurations ? >>> >> >> Does this value is the largest ? >>> >> >> >>> >> > >>> >> > >>> > > > This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
