Hello, I've started doing some profiling related to various tickets and as a result, have attached a few flame graph visualizations to these tickets [1]. Marko made a good suggestion that some explanation of how to read a flame graph would be helpful. They are a bit opaque upon first look. I'm not an expert by any means but I'll give a brief summary of how I read them here and you guys can dig into the links if you're interested in a more in depth explanation.
A fellow by the name of Brendan Gregg created the flame graph visualization to help him diagnose performance issues. Flame graphs are generated in 2 steps. 1. Gather stack trace samples from your process under test (in our case, a Gremlin query). I usually sample the JVM at 100 or 1000hz. 2. Use Brendan Gregg's flame graph tools [2] to transform those raw stack traces into the SVG flame graph. Flame graphs are not limited to the JVM. Traces can be captured from any number of tools (jstack, DTrace, SystemTap, etc). I've seen some really neat flame graphs that include the JVM stack and system calls all rolled into one visualization. X-axis - units here are # of samples, It is not to be read as a progression of time. In other words, say we profiled for 3 seconds at 100 hz. That would translate to 300 stack samples. Y-axis - as you go up, you're moving deeper into the stack. Putting this together, at the bottom of the flame graph, you'll see the shallowest part of your stack. A single chunk probably takes up the full width of the y axis. That means that most, if not all of the stack samples include this stack frame. A simple example would be if I profiled the main method of something, main would show up as the entry point because everything else is originating from it, and therefore, it is in all my stack samples. As you move upwards on the Y-axis, you get deeper into the stack and you can start to see how things branch out. The flame graph generator clusters the samples together so they are easier to read. The SVG is interactive so you can hover over to see % of samples that frame appears in and zoom in on areas. Again, do not read the progression from left to right as a progression of time. It's somewhat analogous to a Fourier transform in that the output of the FFT has transformed something that had a time dimension into the frequency dimension. When I'm taking a look at one of these, I usually start by looking at what is showing up frequently (large # of samples) deeper into the stack (further up the y axis). This helps you prioritize what you want to go after. In many cases you can identify some low hanging fruit this way that doesn't require a more extensive refactor. If you want to try generating your own flame graph, I put together a brief example here https://gist.github.com/twilmes/7adb092bea2a83c26ea3 . I hope that sheds some light on how to read these, but please let me know if I can clarify/expand on anything. I'm also down to hop on the old google hangouts at some point if you guys want to walk through one live. Thanks, Ted [1] https://issues.apache.org/jira/browse/TINKERPOP3-872 https://issues.apache.org/jira/browse/TINKERPOP3-957 [2] https://github.com/brendangregg/FlameGraph
