Thanks Ben. We're making some progress (I've been working with Ken on this). Installing binutils helped, as did slamming the CPU harder.
We're banging around trying to get a top-down tree (tweaking linux-tick-processor). +1 for filtering out epoll, or anything else that makes it so you don't have to slam the cpu as a prerequisite to profiling. A general challenge has been getting confident that we're seeing the whole picture (e.g., now i'm often seeing "syscall" at 47% with no break down) so we don't waste time optimizing a small bit. -aaron On Tuesday, October 1, 2013 9:29:30 PM UTC-7, Ben Noordhuis wrote: > > On Wed, Oct 2, 2013 at 5:34 AM, Ben Noordhuis > <in...@bnoordhuis.nl<javascript:>> > wrote: > > On Wed, Oct 2, 2013 at 12:48 AM, Kenneth Gunn > > <k...@161labs.com<javascript:>> > wrote: > >> Hi! > >> > >> > >> My team is developing a service in node. We are experiencing high CPU > >> utilization and are attempting to profile, but are having a hard time > >> getting a sufficient picture of what’s going on. We have experience > >> profiling in various other environments, but this is our first crack at > >> node. > >> > >> > >> We've tried a few different tools (including nodetime.com, which has > been > >> useful for some things), and have spent most of our time with the v8 > >> profiler. The main problem is that our viewable results only cover a > small > >> portion of the program runtime. More than 80% of the time is spent in > >> libc.so, and that time isn't rolled up by function or caller in the > node > >> program. Also, the C++ section, which I would expect to contain events > in > >> the v8 interpreter itself, is empty. (Below, I'm including an > abbreviated > >> output from the v8 tick processor.) > > > > You need to have the binutils package installed. The tick processor > > uses `nm` to map addresses to symbol. > > > > Small nomenclature nit: V8 is a just-in-time compiler, not an > interpreter. > > > >> We're aware that the v8 profiling output changes frequently, and we've > >> managed to figure out how to get the right tick processor version that > >> corresponds to the node version we are using. (Our steps are here: > >> https://gist.github.com/kennethgunn/6770664 ) We've seen very similar > >> results with versions of node ranging from v0.8.9 to v0.10.18. > >> > >> > >> Is libc actually responsible for 80+% of the CPU time? If so, how do > we > >> roll that up to the the higher level code leading to those calls? Does > it > >> sound like we're missing something here, or is there another set of > tools we > >> should consider using? Your help is greatly appreciated! > > > > That's probably node.js sleeping in the epoll_wait() system call. > > Future versions of node.js will filter out such ticks but right now > > that's not possible, you have to keep your application busy when > > profiling. > > Forgot to mention, you can get a reasonable approximation of non-idle > time by passing -j or --js to the tick processor. That filters out > samples that aren't accountable to JS land. > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
