* Avi Kivity <a...@redhat.com> wrote:
> On 01/27/2010 10:24 AM, Ingo Molnar wrote:
> >
> >
> >>>Not to mention that that process could wreck the trace data rendering it
> >>>utterly unreliable.
> >>It could, but it also might not. Are we going to deny high performance
> >>tracing to users just because it doesn't work in all cases?
> >Tracing and monitoring is foremost about being able to trust the instrument,
> >then about performance and usability. That's one of the big things about
> >ftrace and perf.
> >
> >By proposing 'user space tracing' you are missing two big aspects:
> >
> > - That self-contained, kernel-driven tracing can be replicated in
> > user-space.
> > It cannot. Sharing and global state is much harder to maintain reliably,
> > but the bigger problem is that user-space can stomp on its own tracing
> > state and can make it unreliable. Tracing is often used to figure out
> > bugs,
> > and tracers will be trusted less if they can stomp on themselves.
> >
> > - That somehow it's much faster and that this edge matters. It isnt and it
> > doesnt matter. The few places that need very very fast tracing wont use
> > any
> > of these facilities - it will use something specialized.
> >
> >So you are creating a solution for special cases that dont need it, and you
> >are also ignoring prime qualities of a good tracing framework.
>
> I see it exactly the opposite. Only a very small minority of cases will
> have such severe memory corruption that tracing will fall apart because of
> random writes to memory; especially on 64-bit where the address space is
> sparse. On the other hand, knowing that the cost is a few dozen cycles
> rather than a thousand or so means that you can trace production servers
> running full loads without worrying about whether tracing will affect
> whatever it is you're trying to observe.
>
> I'm not against slow reliable tracing, but we shouldn't ignore the need for
> speed.
I havent seen a conscise summary of your points in this thread, so let me
summarize it as i've understood them (hopefully not putting words into your
mouth): AFAICS you are arguing for some crazy fragile architecture-specific
solution that traps INT3 into ring3 just to shave off a few cycles, and then
use user-space state to trace into.
If so then you ignore the obvious solution to _that_ problem: dont use INT3 at
all, but rebuild (or re-JIT) your program with explicit callbacks. It's _MUCH_
faster than _any_ breakpoint based solution - literally just the cost of a
function call (or not even that - i've written very fast inlined tracers -
they do rock when it comes to performance). Problem solved and none of the
INT3 details matters at all.
INT3 only matters to _transparent_ probing, and for that, the cost of INT3 is
almost _by definition_ less important than the fact that we can do transparent
tracing. If performance were the overriding issue they'd use dedicated
callbacks - and the INT3 technique wouldnt matter at all.
( Also, just like we were able to extend the kprobes code with more and more
optimizations, the same can be done with any user-space probing as well, to
make it faster. But at the core of it has to be a sane design that is
transparent and controlled by the kernel, so that it has the option to apply
more and more otimizations - yours isnt such and its limitations are
designed-in. Which is neither smart nor useful. )
Ingo
