Thanks for all the pointers guys. The solution turned out simple - Had to increase the ulimit to acceptable limits. :)
Thanks Kashyap On Mon, Jan 18, 2016 at 12:33 AM, Anishek Agarwal <anis...@gmail.com> wrote: > i think the worker has to be changed using "worker.childopts" , we are > currently changing this via /opt/storm/conf/storm.yaml > > i am not sure if there is something we have to do for Open JDK , you might > want to look that up a bit :) > > On Sun, Jan 10, 2016 at 11:32 AM, Kashyap Mhaisekar <kashya...@gmail.com> > wrote: > >> Some interesting observations - >> >> I can create only have parallelism around 415 without getting the >> java.lang.OutOfMemoryError >> : unable to create new native Thread This is despite - >> 1. Heap being 2g and above >> 2. system memory and swap were still under control (Under 30% >> utilization). CPU was under 10% >> 3. ulimit -a shows open files and no. of processers at 65336 >> 4. XSS has no impact -> Tried 104K (default of 64bit Open JDK 7) to all >> numbers high and low -> has no affect >> >> Question now I have is - Who creates the supervisors? Is it the worker >> thread or the supervisor thread or nimbus? I think it is the worker thread. >> If I need to change the JVM params to see what is teh max parallelism hint >> I can give, do I change this for supervisor child opts, worker child opts? >> Or storm itself in general? >> >> Thanks >> Kashyap >> >> On Thu, Jan 7, 2016 at 11:58 PM, Anishek Agarwal <anis...@gmail.com> >> wrote: >> >>> More stack size per thread will definitely require more memory for sure, >>> its just that i didnt spend too much time investigating the cause. and for >>> oracle default is 228k so we cant go lower than that. >>> >>> Please do let us know your observations / inferences from the test. >>> >>> thanks >>> anishek >>> >>> On Thu, Jan 7, 2016 at 6:43 PM, Kashyap Mhaisekar <kashya...@gmail.com> >>> wrote: >>> >>>> That's one area that is confusing... Xss indicates stack size at each >>>> thread level. Increasing it means that each thread effectively has more >>>> memory for stack and hence overall requirements for memory increase. Am not >>>> really sure how increasing stack size allows for more thread creation. Will >>>> test this part and confirm back >>>> >>>> Thanks >>>> Kashyap >>>> On Jan 7, 2016 02:58, "Anishek Agarwal" <anis...@gmail.com> wrote: >>>> >>>>> Yeh I have encountered the same error, and as you said its because >>>>> there is a limit to the number of threads with a specific stack size, that >>>>> can be created for a given heap, there are a few discussions on >>>>> Stackoverflow for the same , I also tried to verify this via a standalone >>>>> program but it was behaving very non intuitively on the local machine, for >>>>> ex it was stopping after creating 980 threads always with the same stack >>>>> size of 256k and Heap size varying from default to 1 GB. >>>>> >>>>> we are using oracle jdk and hence in our case the the stack size >>>>> cannont below 228k we have it as 256k with about 4 GB of -Xmx its working >>>>> fine with increased number of threads. >>>>> >>>>> On Wed, Jan 6, 2016 at 7:14 PM, Kashyap Mhaisekar <kashya...@gmail.com >>>>> > wrote: >>>>> >>>>>> Thanks Anishek. >>>>>> I read that OOM due to lack of native threads is because of stack >>>>>> size or because of OS running out of threads. >>>>>> >>>>>> Did you specifically encounter this error in your use case? >>>>>> >>>>>> Thanks >>>>>> Kashyap >>>>>> On Jan 6, 2016 02:32, "Anishek Agarwal" <anis...@gmail.com> wrote: >>>>>> >>>>>>> Hey Kashyap, >>>>>>> >>>>>>> There seems to be a lot of threads that are created per bolt thread >>>>>>> within storm for processing. for ex we have apporx 100 parallelism per >>>>>>> worker(with all bolts in a topology) and we had to specify the -Xmx to >>>>>>> 4 GB >>>>>>> -- internally looked like the process was having abour 3.5-4K threads. >>>>>>> The >>>>>>> number of instances that can be created will depend on the heap size per >>>>>>> worker gets so larger it is higher number of threads, from what you have >>>>>>> published it seems it would be better to have less workers per box say >>>>>>> 4-8 >>>>>>> and higher max heap per worker so you can create more threads per >>>>>>> worker. >>>>>>> >>>>>>> anishek >>>>>>> >>>>>>> On Wed, Jan 6, 2016 at 6:33 AM, Kashyap Mhaisekar < >>>>>>> kashya...@gmail.com> wrote: >>>>>>> >>>>>>>> Priyank, >>>>>>>> Already seen this. My question was more on practical limitations on >>>>>>>> parallelism than doing it right. >>>>>>>> >>>>>>>> Thanks anyways. >>>>>>>> >>>>>>>> Regards >>>>>>>> Kashyap >>>>>>>> On Jan 5, 2016 4:47 PM, "Priyank Shah" <ps...@hortonworks.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Kashyap, >>>>>>>>> >>>>>>>>> Please check out >>>>>>>>> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 >>>>>>>>> >>>>>>>>> >>>>>>>>> Priyank >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Kashyap Mhaisekar >>>>>>>>> Reply-To: "user@storm.apache.org" >>>>>>>>> Date: Tuesday, January 5, 2016 at 1:21 PM >>>>>>>>> To: "user@storm.apache.org" >>>>>>>>> Subject: Storm bolt instances - limits >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> I had a question on the no. of instances of bolts that can be >>>>>>>>> defined on a topology. Here is my use case - >>>>>>>>> 1. 32 GB machine >>>>>>>>> 2. 8 Processors >>>>>>>>> 3. 12 Workers for each box (6700 to 6711) >>>>>>>>> 4. Each worker has been defined with 2G heap and 64mb of XSS. >>>>>>>>> Ulimit for storm is at 1024 >>>>>>>>> >>>>>>>>> My topology has 5 bolts and a spout and If the combined >>>>>>>>> parallelism is above 1000, i see OOM errrors indicating - >>>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>>>>>> >>>>>>>>> When this error occurs, topology is getting initialized and has >>>>>>>>> not started processing. >>>>>>>>> >>>>>>>>> Is there a guideline from resource perspective as to how many >>>>>>>>> instances can be created? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Kashyap >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >> >
