yes, you were right. i the update and now it looks much better! thanks for your help!
On Friday, July 6, 2012 1:05:36 PM UTC+2, Vyacheslav Egorov wrote: > > Yeah this does look like a fragmentation/lazyness issue. > > Heap has only 8MB of live data, but keeps 50mb reserved for old > pointer space alone. This does not look bright. > > First thing I would recommend is to try and roll V8 forward for your > build of node from 3.11.10.12 to 3.11.10.13 and remeasure. I strongly > suspect that memory usage will go down. > > -- > Vyacheslav Egorov > > > On Fri, Jul 6, 2012 at 12:32 PM, chusi wrote: > > Ok, i waited till ps showed around 50MB (still out-of-the box compiled > node > > 0.8.1 without arguments) > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > nodejs 12655 0.0 2.4 731204 50528 pts/0 Sl+ Jul05 0:13 node > > /opt/turntable/bin/hook_luncher.js --hook stats > > > > at the same time i see debug output from nodes gc like this: > > > > 46965999 ms: Mark-sweep 8.7 (95.0) -> 8.3 (95.0) MB, 0 ms (+ 7 ms in 1 > steps > > since start of marking, biggest step 7.866211 ms) [idle notification: > > finalize incremental] [GC in old space requested]. > > Memory allocator, used: 99598336, available: 1435516928 > > New space, used: 0, available: 1048576 > > Old pointers, used: 50052944, available: 4272, waste: 0 > > Old data space, used: 6388224, available: 0, waste: 0 > > Code space, used: 2606944, available: 1472672, waste: 0 > > Map space, used: 252056, available: 910952, waste: 0 > > Cell space, used: 98240, available: 1032000, waste: 0 > > Large object space, used: 0, available: 1434451712 > > > > 46966009 ms: Mark-sweep 8.7 (95.0) -> 8.3 (95.0) MB, 0 ms (+ 7 ms in 1 > steps > > since start of marking, biggest step 7.776855 ms) [idle notification: > > finalize incremental] [GC in old space requested]. > > Memory allocator, used: 99598336, available: 1435516928 > > New space, used: 0, available: 1048576 > > Old pointers, used: 50052944, available: 4272, waste: 0 > > Old data space, used: 6388224, available: 0, waste: 0 > > Code space, used: 2606944, available: 1472672, waste: 0 > > Map space, used: 252056, available: 910952, waste: 0 > > Cell space, used: 98240, available: 1032000, waste: 0 > > Large object space, used: 0, available: 1434451712 > > > > how does this output reflect the 50MB RSS? Can you see something that is > > wrong? > > > > i also posted the whole trace (from the beginning (-13h) till now) at > > http://pastebin.com/K1RmPKUC > > > > > > > > On Thursday, July 5, 2012 12:37:09 PM UTC+2, Vyacheslav Egorov wrote: > >> > >> --trace-gc-verbose should be used together with --trace-gc. > >> > >> When enabled should produce something like: > >> > >> 1087 ms: Scavenge 2.6 (20.4) -> 1.6 (20.4) MB, 0 ms [allocation > >> failure]. > >> Memory allocator, used: 21409792, available: 746147840 > >> New space, used: 0, available: 1048576 > >> Old pointers, used: 1277824, available: 0, waste: > 0 > >> Old data space, used: 1173980, available: 38308, waste: > 0 > >> Code space, used: 1396000, available: 736, waste: > 0 > >> Map space, used: 131072, available: 0, waste: > 0 > >> Cell space, used: 98304, available: 0, waste: > 0 > >> Large object space, used: 0, available: 745066368 > >> > >> I need to see this kind of fine grained statistics to make a guess. > >> > >> You can also try to update V8 inside node from 3.11.10.12 to 3.11.10.13 > so > >> that > >> CL improving memory usage > >> (https://chromiumcodereview.appspot.com/10699051) is included. > >> > >> -- > >> Vyacheslav Egorov > >> > >> > >> On Thu, Jul 5, 2012 at 9:40 AM, chusi wrote: > >> > > >> > the --trace-gc-verbose parameter didn't print anything (at least not > in > >> > the > >> > minute i waited) and until i reach the 50 MB (monitoring using > >> > --trace-gc) > >> > at least some hours have to pass by. I will try it later. > >> > > >> > here's a pic of running the same code once with node v0.6.17 (chris > >> > lea's > >> > packet) and node v.0.8.1 (my own compiled version). > >> > > >> > http://troll.ws/i/nJQJFr.png > >> > > >> > before I updated to the new node version i was fighting against some > >> > memory > >> > leaks in my code, but was able to fix them. when i switched to > v.0.8.1 I > >> > was > >> > worried that they seemed to be back (when i saw the graph). besides > the > >> > "red > >> > line" that goes crazy, i guess the rest can be explained by the new > more > >> > lazy garbage collection approach. > >> > > >> > btw, the app is almost idle. the different modules shown in the graph > >> > just > >> > communicate from time to time with each other and only exchange tiny > >> > bits of > >> > data. > >> > > >> > > >> > > >> > On Wednesday, July 4, 2012 5:33:34 PM UTC+2, Vyacheslav Egorov wrote: > >> >> > >> >> If you run you app with --trace-gc what does it print (I am > interested > >> >> in GC statistics for the time when app reaches 50MB)? > >> >> > >> >> I am also curious in --trace-gc-verbose output to see how memory is > >> >> distributed between different spaces in V8? > >> >> > >> >> It might be that the fragmentation in the heap became higher due to > >> >> lazyness of the new collector (it is tunes for interactive > performance > >> >> so it tries to do things in small increments). > >> >> > >> >> Also I am not sure what the current situation with idle > notifications > >> >> in node is but they are very important for the new collector to > >> >> perform bigger steps when app is idle. > >> >> > >> >> Additionally there were some changes recently that increased > >> >> aggressiveness with which new collector sweeps and compacts things > out > >> >> from almost empty pages. You might be being bitten by this issues. > >> >> > >> >> -- > >> >> Vyacheslav Egorov > >> >> > >> >> > >> >> On Wed, Jul 4, 2012 at 12:42 PM, Ben Noordhuis <i...@bnoordhuis.nl> > >> >> wrote: > >> >> > On Wed, Jul 4, 2012 at 11:39 AM, chusi wrote: > >> >> >> On Tuesday, July 3, 2012 7:53:26 PM UTC+2, Ben Noordhuis wrote: > >> >> >>> Sanity check - what do `file /path/to/old/node` and `file > >> >> >>> /path/to/new/node` print? > >> >> >> > >> >> >> old 0.6.17: > >> >> >> ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically > >> >> >> linked > >> >> >> (uses shared libs), for GNU/Linux 2.6.15, stripped > >> >> >> > >> >> >> new 0.8.1 > >> >> >> ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically > >> >> >> linked > >> >> >> (uses shared libs), for GNU/Linux 2.6.15, stripped > >> >> >> > >> >> >> node is installed using chis-lea's package as described here: > >> >> >> > >> >> >> > >> >> >> > https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager > >> >> > > >> >> > Output from `file` looks good. If one binary was built for ia32 > and > >> >> > the other for x64, you'd expect to see different memory profiles, > but > >> >> > that's not the case here. > >> >> > > >> >> > I don't know if Chris lands patches on top in a way that changes > >> >> > memory consumption. Do you see the same behavior with a (tarball, > >> >> > git) > >> >> > source build? > >> >> > > >> >> > One thing to note is that V8 has a new and improved (but also more > >> >> > lazy) garbage collector. Run your app with --expose-gc and call > gc() > >> >> > from time to time if memory pressure is a real issue - but I would > >> >> > generally advise against that. > >> >> > > >> >> > -- > >> >> > Job Board: http://jobs.nodejs.org/ > >> >> > Posting guidelines: > >> >> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >> >> > You received this message because you are subscribed to the Google > >> >> > Groups "nodejs" group. > >> >> > To post to this group, send email to nodejs@googlegroups.com > >> >> > To unsubscribe from this group, send email to > >> >> > nodejs+unsubscr...@googlegroups.com > >> >> > For more options, visit this group at > >> >> > http://groups.google.com/group/nodejs?hl=en?hl=en > >> > > >> > -- > >> > Job Board: http://jobs.nodejs.org/ > >> > Posting guidelines: > >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >> > You received this message because you are subscribed to the Google > >> > Groups "nodejs" group. > >> > To post to this group, send email to nodejs@googlegroups.com > >> > To unsubscribe from this group, send email to > >> > nodejs+unsubscr...@googlegroups.com > >> > For more options, visit this group at > >> > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to nodejs@googlegroups.com > > To unsubscribe from this group, send email to > > nodejs+unsubscr...@googlegroups.com > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
