I've not seen this, (un)fortunately. :) Are there any other relevant details you might be able to provide?
Or better yet can you distill it down to a bare bones topology that reproduces it and share the code? -Taylor > On Jun 17, 2014, at 6:27 PM, Michael Rose <mich...@fullcontact.com> wrote: > > I've run into similar leaks with one of our topologies. ZMQ vs. Netty didn't > make any difference for us. We'd been looking into the Netty-based HTTP > client we're using as a suspect, but maybe it is Storm. > > 8 workers, 1.5GB heap, CMS collector, Java 1.7.0_25-b15, Storm 0.9.0.1 > > What kinds of things do your topologies do? > > One thing we'd observed is a bump in direct buffers. Usually starts around > 100. Java can't account for the memory used, but the size & count of the > allocations as shown by pmap is suspicious. > > ... > 00007f30ac1bc000 63760K ----- [ anon ] > 00007f30b0000000 864K rw--- [ anon ] > 00007f30b00d8000 64672K ----- [ anon ] > 00007f30b4000000 620K rw--- [ anon ] > 00007f30b409b000 64916K ----- [ anon ] > 00007f30b8000000 1780K rw--- [ anon ] > 00007f30b81bd000 63756K ----- [ anon ] > 00007f30bc000000 1376K rw--- [ anon ] > 00007f30bc158000 64160K ----- [ anon ] > 00007f30c0000000 1320K rw--- [ anon ] > ... > > "buffers":{ > "direct":{ > "count":721, > "memoryUsed":16659150, > "totalCapacity":16659150 > }, > "mapped":{ > "count":0, > "memoryUsed":0, > "totalCapacity":0 > } > }, > > Do you have a similar bump in direct buffer counts? > > Michael > > Michael Rose (@Xorlev) > Senior Platform Engineer, FullContact > mich...@fullcontact.com > > > >> On Tue, Jun 17, 2014 at 11:15 AM, Indra Nath Bardhan >> <indranath.bard...@gmail.com> wrote: >> Hi All, >> >> We have a topology which is running on 16 workers with 2GB heap each. >> >> However we see that the topology worker RES memory usage keeps on piling up >> i.e., starting at 1.1 G and keeps growing over and beyond the 2G mark till >> it overwhelms the entire node. >> >> This possibly indicates that >> >> 1) we either have slowly consuming bolts and thus need throttling in spout >> 2) OR a memory leak in the ZMQ buffer allocation or some of the JNI code. >> >> Based on responses in certain other discussions, we tried making our >> topology reliable and make use of the MAX_SPOUT_PENDING to throttle the >> spouts. However this did not yield us much value, trying with a value of >> 1000 & 100, we see the same growth in the memory usage, although a bit >> slower in the later case. >> >> We also did a pmap of the offending pids and did not see much memory usage >> by the native lib*so files. >> >> Is there any way to identify the source of this native leak OR fix this ? We >> need some urgent help on this. >> >> [NOTE: Using Storm - 0.9.0_wip21] >> >> Thanks, >> Indra >> >> >> >