Adding tracing component to Ignite

[Vladimir Ozerov]

Folks,

Quite often we face some concurrency or performance issues. And what is
worse, sometimes they spread across several nodes making them hard to debug.

We do various things to debug such problems. We add System.outs, measure
latencies between algorithm steps, etc.. And once the problem is resolved,
we throw away all debugging code.

What if add a kind of tracing component to Ignite, so that convenient
debugging infrastructure wiil always be ready for use?

E.g., assume that we have performance issue in cache operation. Tracing
framework could be used as follows:

/** Register tracer. */
Cache.onStart() {
    ...
    ctx.trace().createProfile("cache_problem");
}

/** Put trace inside problematic code. */
Cache.get(...) {
    TraceEvent traceEvt = ctx.trace().start("cache_problem");
    ...
    traceEvt.checkpoint("LOCKED", [optional args and identifiers]);
    ...
    traceEvt.checkpoint("COMMITTED", ...);
}

/** In the end we dump problematic events. */
Cache.onStop() {
    ...
    ctx.profile("cache_problem").dump(System.out, [FILTER PREDICATE]);
}

Result:
23:20:01.234 [LOCKED=20ms, COMMITTED=152ms, FINISHED=153ms]
23:20:03.211 [LOCKED= 1ms, COMMITTED=  1ms, FINISHED=  2ms]

I believe lots of you already did something like this during debug. The
idea is to have some reusable component, so that we do not need to reinvent
the wheel over and over again.

Thoughts?

Vladimir.