Thanks for this Ted. Pretty interesting. On Tue, Nov 24, 2015 at 3:40 PM, Ted Wilmes <[email protected]> wrote:
> Hello, > I've started doing some profiling related to various tickets and as a > result, have attached a few flame graph visualizations to these tickets > [1]. Marko made a good suggestion that some explanation of how to read a > flame graph would be helpful. They are a bit opaque upon first look. I'm > not an expert by any means but I'll give a brief summary of how I read them > here and you guys can dig into the links if you're interested in a more in > depth explanation. > > A fellow by the name of Brendan Gregg created the flame graph visualization > to help him diagnose performance issues. Flame graphs are generated in 2 > steps. > > 1. Gather stack trace samples from your process under test (in our case, a > Gremlin query). I usually sample the JVM at 100 or 1000hz. > 2. Use Brendan Gregg's flame graph tools [2] to transform those raw stack > traces into the SVG flame graph. > > Flame graphs are not limited to the JVM. Traces can be captured from any > number of tools (jstack, DTrace, SystemTap, etc). I've seen some really > neat flame graphs that include the JVM stack and system calls all rolled > into one visualization. > > X-axis - units here are # of samples, It is not to be read as a progression > of time. In other words, say we profiled for 3 seconds at 100 hz. That > would translate to 300 stack samples. > Y-axis - as you go up, you're moving deeper into the stack. > > Putting this together, at the bottom of the flame graph, you'll see the > shallowest part of your stack. A single chunk probably takes up the full > width of the y axis. That means that most, if not all of the stack samples > include this stack frame. A simple example would be if I profiled the main > method of something, main would show up as the entry point because > everything else is originating from it, and therefore, it is in all my > stack samples. As you move upwards on the Y-axis, you get deeper into the > stack and you can start to see how things branch out. The flame graph > generator clusters the samples together so they are easier to read. The > SVG is interactive so you can hover over to see % of samples that frame > appears in and zoom in on areas. Again, do not read the progression from > left to right as a progression of time. It's somewhat analogous to a > Fourier transform in that the output of the FFT has transformed something > that had a time dimension into the frequency dimension. > > When I'm taking a look at one of these, I usually start by looking at what > is showing up frequently (large # of samples) deeper into the stack > (further up the y axis). This helps you prioritize what you want to go > after. In many cases you can identify some low hanging fruit this way that > doesn't require a more extensive refactor. If you want to try generating > your own flame graph, I put together a brief example here > https://gist.github.com/twilmes/7adb092bea2a83c26ea3 . > > I hope that sheds some light on how to read these, but please let me know > if I can clarify/expand on anything. I'm also down to hop on the old > google hangouts at some point if you guys want to walk through one live. > > Thanks, > Ted > > [1] https://issues.apache.org/jira/browse/TINKERPOP3-872 > https://issues.apache.org/jira/browse/TINKERPOP3-957 > [2] https://github.com/brendangregg/FlameGraph >
