Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On 04/14/2014 05:11 PM, Ingo Molnar wrote: * Jovi Zhangwei wrote: Obviously many people love ktap nowadays even though it's not a C-family language, [...] Imagine how much more widespread it would become amongst kernel developers if it had C syntax - see PeterZ's reply for example. +1 I think it would be awesome to reuse the kernel's BPF engine with its C backend for this in the way that Alexei has started in its original set; surely there's a lot of work that still needs to be addressed, but it would allow long-term for just having that engine to maintain, and also to exploit its flexibility and speed (JIT) where we can. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Jovi Zhangwei wrote: > Obviously many people love ktap nowadays even though it's not a > C-family language, [...] Imagine how much more widespread it would become amongst kernel developers if it had C syntax - see PeterZ's reply for example. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Tue, Apr 08, 2014 at 04:40:36PM +0900, Masami Hiramatsu wrote: > (2014/04/07 22:55), Peter Zijlstra wrote: > > On Wed, Apr 02, 2014 at 09:42:03AM +0200, Ingo Molnar wrote: > >> I'd suggest using C syntax instead initially, because that's what the > >> kernel is using. > >> > >> The overwhelming majority of people probing the kernel are > >> programmers, so there's no point in inventing new syntax, we should > >> reuse existing syntax! > > > > Yes please, keep it C, I forever forget all other syntaxes. While I have > > in the past known other languages, I never use them frequently enough to > > remember them. And there's nothing more frustrating than having to fight > > a tool/language when you just want to get work done. > > Why wouldn't you write a kernel module in C directly? :) > It seems that all what you need is not a tracing language nor a bytecode > engine, but an well organized tracing APIs(library?) for writing a kernel > module for tracing... Most my kernels are CONFIG_MODULE=n :-) Also, I never can remember how to do modules. That said; what I currently do it hack the kernel with debug bits and pieces and run that, which is effectively the same. Its just that its impossible to save/share these hacks in any sane fashion. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
(2014/04/07 22:55), Peter Zijlstra wrote: > On Wed, Apr 02, 2014 at 09:42:03AM +0200, Ingo Molnar wrote: >> I'd suggest using C syntax instead initially, because that's what the >> kernel is using. >> >> The overwhelming majority of people probing the kernel are >> programmers, so there's no point in inventing new syntax, we should >> reuse existing syntax! > > Yes please, keep it C, I forever forget all other syntaxes. While I have > in the past known other languages, I never use them frequently enough to > remember them. And there's nothing more frustrating than having to fight > a tool/language when you just want to get work done. Why wouldn't you write a kernel module in C directly? :) It seems that all what you need is not a tracing language nor a bytecode engine, but an well organized tracing APIs(library?) for writing a kernel module for tracing... Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Fri, Apr 4, 2014 at 3:36 PM, Ingo Molnar wrote:
>
> * Jovi Zhangwei wrote:
>
>> I don't see any suggestion about integrating BPF before this review
>> cycle.
>
> Hm, I mentioned it at the kernel summit to folks who raised the ktap
> subject, but apparently not over email. I assumed a new ktap
> submission would come quickly.
>
> Anyway, considering how BPF is integrating with (and hopefully
> replacing) tracing filters, it makes sense to apply the same concept
> in the ktap case as well.
>
It seems that kernel developers want a C based tracing filter, BPF
rooted from C(Compiler, performance, and networking target),
but ktap designed from a different point of view, which highlights
simplicity and flexibility, this simplicity and flexibility is not only
presented by simple syntax, but also in its bytecode engine.
trace syscalls:* {
print(cpu, pid, execname, argstr, stack())
}
Each expression have own type in kernel, the type is not judged by
userspace compiler, this make print expression much much easy,
also make handle associate array(and aggregation) extremely simple.
If we force replace ktap core with a much "static" bytecode engine,
then it will lose the highlight feature: simplicity and flexibility.
ktap is not designed to be a built-in tracing filter and not purpose for
high performance bytecode engine(can use for networking), BPF is
suit for that way, but that is not a design flaw in ktap, it's designed
to be like that.
Obviously many people love ktap nowadays even though it's not a
C-family language, people can use ktap one-liner to do interesting
things(http://brendangregg.com/ktap.html), and used in real world.
Also especially people in embedded world like ktap very much,
they already included ktap into OpenEmebedded as I know.
It seems that more and more tracing related kernel modules will
come in future(like https://github.com/draios/sysdig/issues/81),
so maybe these third-part kernel modules based on tracing export
symbols should locate in drivers/, not kernel/trace/, today we have
well defined tracing kernel interface(tracepoint, kprobe, uprobe, perf
callback),
no reason to put all these external modules sit in kernel/trace one place,
especially if the module have real end users in world.
Thanks.
Jovi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Wed, Apr 02, 2014 at 09:42:03AM +0200, Ingo Molnar wrote: > I'd suggest using C syntax instead initially, because that's what the > kernel is using. > > The overwhelming majority of people probing the kernel are > programmers, so there's no point in inventing new syntax, we should > reuse existing syntax! Yes please, keep it C, I forever forget all other syntaxes. While I have in the past known other languages, I never use them frequently enough to remember them. And there's nothing more frustrating than having to fight a tool/language when you just want to get work done. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Jovi Zhangwei wrote: > On Wed, Apr 2, 2014 at 3:43 PM, Ingo Molnar wrote: > > > > * Jovi Zhangwei wrote: > > > >> So based on all these input, I suggest: > >> > >> Put all these community efforts together, figure out the proper > >> design implementation of dynamic tracing tool, ktap can be a good > >> start to build upon it, evolve to a unified kernel script engine > >> with ebpf together, finally service for dynamic tracing and > >> network(if possible). > >> > >> Our goal is same and very clearly, we really want a "simple & > >> flexible & safe" dynamic scripting tracing tool for Linux, which > >> could compare or even better than Dtrace, this is the motivation of > >> ktap project. > >> > >> Two solution may be take: > >> > >> 1). upstream ktap into core trace and evolve it step by step, and > >> finally make a integrated bytecode engine, it's a long process, > >> but I think it's worth. > >> > >> 2). move ktap back into staging, and graduate from staging after the > >> code make tracing people and ebpf people both happy. > >> > >> The benefit is the process will be under the eyes of community. > >> > >> Ingo, steven, Greg, what do you think? > > > > For now I'm opting for a third option: > > > > 3) Maintain my NAK on the ktap patches until they address the > > fundamental design concerns outlined by Alexei and others in > > their review feedback: > > > > There is no fundamental design concerns outlined about ktap, [...] Well, the concerns that were outlined do raise to fundamental in my book. But as long as they are addressed it does not matter how we classify them. > I don't see any suggestion about integrating BPF before this review > cycle. Hm, I mentioned it at the kernel summit to folks who raised the ktap subject, but apparently not over email. I assumed a new ktap submission would come quickly. Anyway, considering how BPF is integrating with (and hopefully replacing) tracing filters, it makes sense to apply the same concept in the ktap case as well. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Wed, Apr 2, 2014 at 3:43 PM, Ingo Molnar wrote: > > * Jovi Zhangwei wrote: > >> So based on all these input, I suggest: >> >> Put all these community efforts together, figure out the proper >> design implementation of dynamic tracing tool, ktap can be a good >> start to build upon it, evolve to a unified kernel script engine >> with ebpf together, finally service for dynamic tracing and >> network(if possible). >> >> Our goal is same and very clearly, we really want a "simple & >> flexible & safe" dynamic scripting tracing tool for Linux, which >> could compare or even better than Dtrace, this is the motivation of >> ktap project. >> >> Two solution may be take: >> >> 1). upstream ktap into core trace and evolve it step by step, and >> finally make a integrated bytecode engine, it's a long process, >> but I think it's worth. >> >> 2). move ktap back into staging, and graduate from staging after the >> code make tracing people and ebpf people both happy. >> >> The benefit is the process will be under the eyes of community. >> >> Ingo, steven, Greg, what do you think? > > For now I'm opting for a third option: > > 3) Maintain my NAK on the ktap patches until they address the > fundamental design concerns outlined by Alexei and others in > their review feedback: > There is no fundamental design concerns outlined about ktap, that 'loop' in ktap design is not fundamental, others review feedback is focus on patch, not fundamental design, and those "safety" concerns raised by Alexei is just bound checking. I don't see any others. > NAKed-by: Ingo Molnar > > The thing is, I've outlined some of the concerns in my previous > review. Not much happened on that front, for example ktap did not get > any closer in integrating with BPF. > I don't see any suggestion about integrating BPF before this review cycle. > Many months have passed since the > previous ktap submission, still I see no progress on the 'design' > front. That really needs to change. > I think the change need both in ktap and ebpf, not ktap one side, if you really want integrate ktap and BPF is a way. There have no proven that current ebpf low level bytecode design is suit for generic dynamic tracing use, I also raised some design concern about ebpf in previous mail but got no clear answer, like exposed too much low level stuff to end user, that would be a problem because script engine need to validate more memory references which should not need to; and also there may be expose table lock into user inf future, that would be more complex IMO. The efforts need all related people participation, not only ktap, ktap and bpf maybe need to close with each other, but not means bpf stay and let ktap move close. That's why I think the things would be more faster and effective if in public place(like staging tree), otherwise the change may be hard to happen. Thanks. Jovi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Jovi Zhangwei wrote: > So based on all these input, I suggest: > > Put all these community efforts together, figure out the proper > design implementation of dynamic tracing tool, ktap can be a good > start to build upon it, evolve to a unified kernel script engine > with ebpf together, finally service for dynamic tracing and > network(if possible). > > Our goal is same and very clearly, we really want a "simple & > flexible & safe" dynamic scripting tracing tool for Linux, which > could compare or even better than Dtrace, this is the motivation of > ktap project. > > Two solution may be take: > > 1). upstream ktap into core trace and evolve it step by step, and > finally make a integrated bytecode engine, it's a long process, > but I think it's worth. > > 2). move ktap back into staging, and graduate from staging after the > code make tracing people and ebpf people both happy. > > The benefit is the process will be under the eyes of community. > > Ingo, steven, Greg, what do you think? For now I'm opting for a third option: 3) Maintain my NAK on the ktap patches until they address the fundamental design concerns outlined by Alexei and others in their review feedback: NAKed-by: Ingo Molnar The thing is, I've outlined some of the concerns in my previous review. Not much happened on that front, for example ktap did not get any closer in integrating with BPF. Many months have passed since the previous ktap submission, still I see no progress on the 'design' front. That really needs to change. Please keep me Cc:-ed to any and all future ktap submissions so I can monitor ktap's progress and lift the NAK if the design concerns have been addressed. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Alexei Starovoitov wrote: > [...] > > It seems that your definition of 'safe ktap' is that user cannot break > kernel if he uses ktap scripting syntax. > In that sense ktap is not much different from stap. > > Overall it seems you view ktap bytecode as a continuation > of ktap syntax. > ktap language allows to read pid,uid,tid, so they were added as > separate instructions to ktap bytecode... > ktap allows dump of a table, so kernel has to do tab_histdump() > including sorting of fields and printf formatting. > What if ktap user wants a different table dump? > or new features from the language? > keep extending bytecode for every printf tweak is not a great solution. > > I think design approach to ktap needs to change. > What I'm proposing is the following: > - keep ktap syntax as-is, but remove loops > - ktap style of accessing tables is definitely less verbose then C, > so keep it, but don't let compiled program to own the memory > - keep table dump as-is, but do it in userspace instead I'd suggest using C syntax instead initially, because that's what the kernel is using. The overwhelming majority of people probing the kernel are programmers, so there's no point in inventing new syntax, we should reuse existing syntax! That is one reason why for example the (very simple!) ftrace filter language tries to mimic C syntax. Especially as C is simpler for an important category, filters: > Though I think C as a language to express filters is simpler, > ktap syntax is fine as well. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Jovi Zhangwei wrote: > > I think nothing stops ktap userspace to parse ktap language and > > generate 'internal bpf' format. gcc is unnecessary here. > > It's a big engineering problem, [...] Sorry, but it will become an even bigger engineering problem if it's merged to the upstream kernel tree! You need to solve known design and implementational issues before we can even think about any upstream merge. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Wed, Apr 2, 2014 at 12:57 PM, Alexei Starovoitov wrote: > On Mon, Mar 31, 2014 at 9:47 PM, Jovi Zhangwei > wrote: >> Hi Alexei, >> >> On Tue, Apr 1, 2014 at 5:29 AM, Alexei Starovoitov wrote: >>> On Mon, Mar 31, 2014 at 3:01 AM, Jovi Zhangwei >>> wrote: Hi Ingo, On Mon, Mar 31, 2014 at 3:17 PM, Ingo Molnar wrote: > > * Jovi Zhangwei wrote: > >> Hi All, >> >> The following set of patches add ktap tracing tool. >> >> ktap is a new script-based dynamic tracing tool for Linux. >> It uses a scripting language and lets the user trace system dynamically. >> >> Highlights features: >> * a simple but powerful scripting language >> * register-based interpreter (heavily optimized) in Linux kernel >> * small and lightweight >> * not depend on the GCC toolchain for each script run >> * easy to use in embedded environments without debugging info >> * support for tracepoint, kprobe, uprobe, function trace, timer, and more >> * supported in x86, ARM, PowerPC, MIPS >> * safety in sandbox > > I've asked this fundamental design question before but got no full > answer: how does ktap compare to the ongoing effort of improving the > BPF scripting engine? > From long experiences of ktap development, what make me really love ktap is: 1) Availability ktap is only available tool to use in small embedded platform, stap and BPF both need GCC now, stap have its own language, so it's much better than BPF. (IMO there may need several years to complete a skeleton of dynamic tracing script language, see stap and dtrace) 2) Simplicity ktap is simplest dynamic scripting trace solution now in Linux world, compare with stap/dtrace/BPF. a). It have simple syntax which make many people like it, it have b). It have simple associate array, make dynamic tracing powerful. c). It have a simple compiler which only have 87K in x86_64. d). It have a simple tracing syntax which constant with perf events. 3) Safety ktap already delivered its safety to end user, many people use ktap in their dev lab to investigate problem. But BPF need time to prove its safety, especially proved by end user, and IMO BPF safety check would be more complex if the runtime support more features as time goes. >>> >>> safety of ktap is arguable. >>> >>> 1. >>> From the diff it seems that 'loop_count' is a dynamic way of >>> checking that loops are not infinite, but max_loop_count = 10 >>> if loop body has many instructions, such large count may trigger >>> hung_task panic. >>> >> Actually I'm planing use time-based time to avoid this, minor issue. >> >>> 2. >>> jumps are not counted, so if userspace makes an error and loads >>> ktap bytecode with wrong jumps, the interpreter will hang. >>> >> There leave a todo in validation code, as kernel developers don't like >> many todo in there, so I will also address it. >> >>> 3. >>> recursive functions and f1()->f2()->f1() are not detected either. >>> Another possible hang? >>> >> No, it will exit by ktap stack overflow check. >> >>> 4. >>> bc_[ft]new instruction are allocating memory and garbage collector >>> suppose to free things when ktap module is unloaded, right? >>> since max_loop_cnt is 100k, a script can allocate quite a bit of memory >>> and kernel will be waiting for userspace trigger to free it? >>> Sounds dangerous. >>> >> There will have table/function number limitation, so this is not a problem. >> >>> These concerns are just from quick code review. > > ok. spotted few more things: > > 5. > bc_callt (ktap tailcall) doesn't have loop_count check. > so tailcall can loop forever. > Of course you can fix it with elapsed time check at every branch > or call instruction. > > 6. > do_bc_kstr doesn't check that 'd' value is valid and goes kbase[~d] > Can be fixed of course. > > 7. > uget/uset and others seem to have similar problems. > > It seems that your definition of 'safe ktap' is that user cannot break > kernel if he uses ktap scripting syntax. > In that sense ktap is not much different from stap. > Definitely not. Safety don't means bound check which you listed above, ktap have safety on bytecode level in design, not syntax, it means whatever you do on the bytecode, it will never crash kernel, please see more in ktapvm. > Overall it seems you view ktap bytecode as a continuation > of ktap syntax. > ktap language allows to read pid,uid,tid, so they were added as > separate instructions to ktap bytecode... > ktap allows dump of a table, so kernel has to do tab_histdump() > including sorting of fields and printf formatting. > What if ktap user wants a different table dump? > or new features from the language? > keep extending bytecode for every printf tweak is not a great solution. > > I think design approach to ktap needs to change. >
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Mon, Mar 31, 2014 at 9:47 PM, Jovi Zhangwei wrote: > Hi Alexei, > > On Tue, Apr 1, 2014 at 5:29 AM, Alexei Starovoitov wrote: >> On Mon, Mar 31, 2014 at 3:01 AM, Jovi Zhangwei >> wrote: >>> Hi Ingo, >>> >>> On Mon, Mar 31, 2014 at 3:17 PM, Ingo Molnar wrote: * Jovi Zhangwei wrote: > Hi All, > > The following set of patches add ktap tracing tool. > > ktap is a new script-based dynamic tracing tool for Linux. > It uses a scripting language and lets the user trace system dynamically. > > Highlights features: > * a simple but powerful scripting language > * register-based interpreter (heavily optimized) in Linux kernel > * small and lightweight > * not depend on the GCC toolchain for each script run > * easy to use in embedded environments without debugging info > * support for tracepoint, kprobe, uprobe, function trace, timer, and more > * supported in x86, ARM, PowerPC, MIPS > * safety in sandbox I've asked this fundamental design question before but got no full answer: how does ktap compare to the ongoing effort of improving the BPF scripting engine? >>> >>> From long experiences of ktap development, what make me really >>> love ktap is: >>> >>> 1) Availability >>> ktap is only available tool to use in small embedded platform, stap >>> and BPF both need GCC now, stap have its own language, so it's much >>> better than BPF. >>> (IMO there may need several years to complete a skeleton of dynamic >>> tracing script language, see stap and dtrace) >>> >>> 2) Simplicity >>> ktap is simplest dynamic scripting trace solution now in Linux world, >>> compare with stap/dtrace/BPF. >>> a). It have simple syntax which make many people like it, it have >>> b). It have simple associate array, make dynamic tracing powerful. >>> c). It have a simple compiler which only have 87K in x86_64. >>> d). It have a simple tracing syntax which constant with perf events. >>> >>> 3) Safety >>> ktap already delivered its safety to end user, many people use ktap >>> in their dev lab to investigate problem. >>> But BPF need time to prove its safety, especially proved by end user, >>> and IMO BPF safety check would be more complex if the runtime >>> support more features as time goes. >> >> safety of ktap is arguable. >> >> 1. >> From the diff it seems that 'loop_count' is a dynamic way of >> checking that loops are not infinite, but max_loop_count = 10 >> if loop body has many instructions, such large count may trigger >> hung_task panic. >> > Actually I'm planing use time-based time to avoid this, minor issue. > >> 2. >> jumps are not counted, so if userspace makes an error and loads >> ktap bytecode with wrong jumps, the interpreter will hang. >> > There leave a todo in validation code, as kernel developers don't like > many todo in there, so I will also address it. > >> 3. >> recursive functions and f1()->f2()->f1() are not detected either. >> Another possible hang? >> > No, it will exit by ktap stack overflow check. > >> 4. >> bc_[ft]new instruction are allocating memory and garbage collector >> suppose to free things when ktap module is unloaded, right? >> since max_loop_cnt is 100k, a script can allocate quite a bit of memory >> and kernel will be waiting for userspace trigger to free it? >> Sounds dangerous. >> > There will have table/function number limitation, so this is not a problem. > >> These concerns are just from quick code review. ok. spotted few more things: 5. bc_callt (ktap tailcall) doesn't have loop_count check. so tailcall can loop forever. Of course you can fix it with elapsed time check at every branch or call instruction. 6. do_bc_kstr doesn't check that 'd' value is valid and goes kbase[~d] Can be fixed of course. 7. uget/uset and others seem to have similar problems. It seems that your definition of 'safe ktap' is that user cannot break kernel if he uses ktap scripting syntax. In that sense ktap is not much different from stap. Overall it seems you view ktap bytecode as a continuation of ktap syntax. ktap language allows to read pid,uid,tid, so they were added as separate instructions to ktap bytecode... ktap allows dump of a table, so kernel has to do tab_histdump() including sorting of fields and printf formatting. What if ktap user wants a different table dump? or new features from the language? keep extending bytecode for every printf tweak is not a great solution. I think design approach to ktap needs to change. What I'm proposing is the following: - keep ktap syntax as-is, but remove loops - ktap style of accessing tables is definitely less verbose then C, so keep it, but don't let compiled program to own the memory - keep table dump as-is, but do it in userspace instead In other words compiler for ktap scripts can generate kernel program and userspace program at the same time. the end users won't notice the difference vs what you have now.
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
Hi Alexei, On Tue, Apr 1, 2014 at 5:29 AM, Alexei Starovoitov wrote: > On Mon, Mar 31, 2014 at 3:01 AM, Jovi Zhangwei > wrote: >> Hi Ingo, >> >> On Mon, Mar 31, 2014 at 3:17 PM, Ingo Molnar wrote: >>> >>> * Jovi Zhangwei wrote: >>> Hi All, The following set of patches add ktap tracing tool. ktap is a new script-based dynamic tracing tool for Linux. It uses a scripting language and lets the user trace system dynamically. Highlights features: * a simple but powerful scripting language * register-based interpreter (heavily optimized) in Linux kernel * small and lightweight * not depend on the GCC toolchain for each script run * easy to use in embedded environments without debugging info * support for tracepoint, kprobe, uprobe, function trace, timer, and more * supported in x86, ARM, PowerPC, MIPS * safety in sandbox >>> >>> I've asked this fundamental design question before but got no full >>> answer: how does ktap compare to the ongoing effort of improving the >>> BPF scripting engine? >>> >> >> From long experiences of ktap development, what make me really >> love ktap is: >> >> 1) Availability >> ktap is only available tool to use in small embedded platform, stap >> and BPF both need GCC now, stap have its own language, so it's much >> better than BPF. >> (IMO there may need several years to complete a skeleton of dynamic >> tracing script language, see stap and dtrace) >> >> 2) Simplicity >> ktap is simplest dynamic scripting trace solution now in Linux world, >> compare with stap/dtrace/BPF. >> a). It have simple syntax which make many people like it, it have >> b). It have simple associate array, make dynamic tracing powerful. >> c). It have a simple compiler which only have 87K in x86_64. >> d). It have a simple tracing syntax which constant with perf events. >> >> 3) Safety >> ktap already delivered its safety to end user, many people use ktap >> in their dev lab to investigate problem. >> But BPF need time to prove its safety, especially proved by end user, >> and IMO BPF safety check would be more complex if the runtime >> support more features as time goes. > > safety of ktap is arguable. > > 1. > From the diff it seems that 'loop_count' is a dynamic way of > checking that loops are not infinite, but max_loop_count = 10 > if loop body has many instructions, such large count may trigger > hung_task panic. > Actually I'm planing use time-based time to avoid this, minor issue. > 2. > jumps are not counted, so if userspace makes an error and loads > ktap bytecode with wrong jumps, the interpreter will hang. > There leave a todo in validation code, as kernel developers don't like many todo in there, so I will also address it. > 3. > recursive functions and f1()->f2()->f1() are not detected either. > Another possible hang? > No, it will exit by ktap stack overflow check. > 4. > bc_[ft]new instruction are allocating memory and garbage collector > suppose to free things when ktap module is unloaded, right? > since max_loop_cnt is 100k, a script can allocate quite a bit of memory > and kernel will be waiting for userspace trigger to free it? > Sounds dangerous. > There will have table/function number limitation, so this is not a problem. > These concerns are just from quick code review. > >> 4). Samples >> Many people like those ktap samples, ktap shows the attractive by >> samples. >> >> Even I so love ktap and would like share ktap values to everyone, but in >> technical point of view, I still agree with you that there should have >> unified scripting engines in kernel if that engine can service for many >> domains(like networking), but that solution should show its availability/ >> simplicity/safety firstly to user, not just proved by end user. >> >> Dynamic tracing scripting environment should contains: >> simple compiler, clean language syntax, fast script engine, >> associative array, aggregation, kstack, ustack, event management, >> ring buffer, samples, tapset/library, CTF, etc. >> >> ktap already fixed most of these issues by its simple design, but >> BPF only have "script engine" part(its associative array still cannot >> vmalloc), which have long road before could use by end user. > > 'internal bpf' instruction set is an assembler instruction set. > Low level just like x86 instruction set. > It doesn't have vmalloc instruction and shouldn't have. > 'internal bpf' program can theoretically make a call to allocate > memory, but I don't think it's safe to let loadable programs to > arbitrarily allocate memory. > It's a matter of ownership of the memory. > If script can allocate and receive a pointer to memory, > the script owns that memory and kernel cannot touch it until > script does 'free' or terminates and GC kicks in. Wrong, ktap don't have vmalloc instruction, ktap only use vmalloc for table and memory pool pre-allocation. > ktap can be invoked through
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Mon, Mar 31, 2014 at 3:01 AM, Jovi Zhangwei wrote: > Hi Ingo, > > On Mon, Mar 31, 2014 at 3:17 PM, Ingo Molnar wrote: >> >> * Jovi Zhangwei wrote: >> >>> Hi All, >>> >>> The following set of patches add ktap tracing tool. >>> >>> ktap is a new script-based dynamic tracing tool for Linux. >>> It uses a scripting language and lets the user trace system dynamically. >>> >>> Highlights features: >>> * a simple but powerful scripting language >>> * register-based interpreter (heavily optimized) in Linux kernel >>> * small and lightweight >>> * not depend on the GCC toolchain for each script run >>> * easy to use in embedded environments without debugging info >>> * support for tracepoint, kprobe, uprobe, function trace, timer, and more >>> * supported in x86, ARM, PowerPC, MIPS >>> * safety in sandbox >> >> I've asked this fundamental design question before but got no full >> answer: how does ktap compare to the ongoing effort of improving the >> BPF scripting engine? >> > > From long experiences of ktap development, what make me really > love ktap is: > > 1) Availability > ktap is only available tool to use in small embedded platform, stap > and BPF both need GCC now, stap have its own language, so it's much > better than BPF. > (IMO there may need several years to complete a skeleton of dynamic > tracing script language, see stap and dtrace) > > 2) Simplicity > ktap is simplest dynamic scripting trace solution now in Linux world, > compare with stap/dtrace/BPF. > a). It have simple syntax which make many people like it, it have > b). It have simple associate array, make dynamic tracing powerful. > c). It have a simple compiler which only have 87K in x86_64. > d). It have a simple tracing syntax which constant with perf events. > > 3) Safety > ktap already delivered its safety to end user, many people use ktap > in their dev lab to investigate problem. > But BPF need time to prove its safety, especially proved by end user, > and IMO BPF safety check would be more complex if the runtime > support more features as time goes. safety of ktap is arguable. 1. >From the diff it seems that 'loop_count' is a dynamic way of checking that loops are not infinite, but max_loop_count = 10 if loop body has many instructions, such large count may trigger hung_task panic. 2. jumps are not counted, so if userspace makes an error and loads ktap bytecode with wrong jumps, the interpreter will hang. 3. recursive functions and f1()->f2()->f1() are not detected either. Another possible hang? 4. bc_[ft]new instruction are allocating memory and garbage collector suppose to free things when ktap module is unloaded, right? since max_loop_cnt is 100k, a script can allocate quite a bit of memory and kernel will be waiting for userspace trigger to free it? Sounds dangerous. These concerns are just from quick code review. > 4). Samples > Many people like those ktap samples, ktap shows the attractive by > samples. > > Even I so love ktap and would like share ktap values to everyone, but in > technical point of view, I still agree with you that there should have > unified scripting engines in kernel if that engine can service for many > domains(like networking), but that solution should show its availability/ > simplicity/safety firstly to user, not just proved by end user. > > Dynamic tracing scripting environment should contains: > simple compiler, clean language syntax, fast script engine, > associative array, aggregation, kstack, ustack, event management, > ring buffer, samples, tapset/library, CTF, etc. > > ktap already fixed most of these issues by its simple design, but > BPF only have "script engine" part(its associative array still cannot > vmalloc), which have long road before could use by end user. 'internal bpf' instruction set is an assembler instruction set. Low level just like x86 instruction set. It doesn't have vmalloc instruction and shouldn't have. 'internal bpf' program can theoretically make a call to allocate memory, but I don't think it's safe to let loadable programs to arbitrarily allocate memory. It's a matter of ownership of the memory. If script can allocate and receive a pointer to memory, the script owns that memory and kernel cannot touch it until script does 'free' or terminates and GC kicks in. ktap can be invoked through timers, so this dynamically allocated tables may be living for long time affecting the whole system. The tracing tool should be safer than that. > ktap is not just bring a bytecode engine, it bring a complete simple > dynamic tracing environment to end user, it bring clean language syntax, > samples, flexible table, perf like event management, etc, those is the > key part to end user, not bytecode engine, so if we can develop simple > BPF compiler with similar ktap syntax in some day, then we can replace > kp_lex.c/kp_parse.c/kp_vm.c, and there have zero reason why other > parts cannot be shared(associative array, aggregation, k
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On 03/31/2014 09:17 AM, Ingo Molnar wrote: ... 2) better BPF integration with networking: [PATCH net-next v3 8/9] net: filter: rework/optimize internal BPF interpreter's instruction set [ ... v4 is now queued in net-next tree btw ] -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
Hi Ingo, On Mon, Mar 31, 2014 at 3:17 PM, Ingo Molnar wrote: > > * Jovi Zhangwei wrote: > >> Hi All, >> >> The following set of patches add ktap tracing tool. >> >> ktap is a new script-based dynamic tracing tool for Linux. >> It uses a scripting language and lets the user trace system dynamically. >> >> Highlights features: >> * a simple but powerful scripting language >> * register-based interpreter (heavily optimized) in Linux kernel >> * small and lightweight >> * not depend on the GCC toolchain for each script run >> * easy to use in embedded environments without debugging info >> * support for tracepoint, kprobe, uprobe, function trace, timer, and more >> * supported in x86, ARM, PowerPC, MIPS >> * safety in sandbox > > I've asked this fundamental design question before but got no full > answer: how does ktap compare to the ongoing effort of improving the > BPF scripting engine? > >From long experiences of ktap development, what make me really love ktap is: 1) Availability ktap is only available tool to use in small embedded platform, stap and BPF both need GCC now, stap have its own language, so it's much better than BPF. (IMO there may need several years to complete a skeleton of dynamic tracing script language, see stap and dtrace) 2) Simplicity ktap is simplest dynamic scripting trace solution now in Linux world, compare with stap/dtrace/BPF. a). It have simple syntax which make many people like it, it have b). It have simple associate array, make dynamic tracing powerful. c). It have a simple compiler which only have 87K in x86_64. d). It have a simple tracing syntax which constant with perf events. 3) Safety ktap already delivered its safety to end user, many people use ktap in their dev lab to investigate problem. But BPF need time to prove its safety, especially proved by end user, and IMO BPF safety check would be more complex if the runtime support more features as time goes. 4). Samples Many people like those ktap samples, ktap shows the attractive by samples. Even I so love ktap and would like share ktap values to everyone, but in technical point of view, I still agree with you that there should have unified scripting engines in kernel if that engine can service for many domains(like networking), but that solution should show its availability/ simplicity/safety firstly to user, not just proved by end user. Dynamic tracing scripting environment should contains: simple compiler, clean language syntax, fast script engine, associative array, aggregation, kstack, ustack, event management, ring buffer, samples, tapset/library, CTF, etc. ktap already fixed most of these issues by its simple design, but BPF only have "script engine" part(its associative array still cannot vmalloc), which have long road before could use by end user. ktap is not just bring a bytecode engine, it bring a complete simple dynamic tracing environment to end user, it bring clean language syntax, samples, flexible table, perf like event management, etc, those is the key part to end user, not bytecode engine, so if we can develop simple BPF compiler with similar ktap syntax in some day, then we can replace kp_lex.c/kp_parse.c/kp_vm.c, and there have zero reason why other parts cannot be shared(associative array, aggregation, kstack, ustack, event management, ring buffer, samples, tapset/library, CTF, etc). So ktap is deviling a simple designed tracing script environment, but BPF just provide script engine which still not proved by end user in many facts(Availability/Simplicity/Safety). Sorry for the length of this reply. Thanks. Jovi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
Hi Jovi, (2014/03/28 22:47), Jovi Zhangwei wrote: > Hi All, > > The following set of patches add ktap tracing tool. > > ktap is a new script-based dynamic tracing tool for Linux. > It uses a scripting language and lets the user trace system dynamically. > > Highlights features: > * a simple but powerful scripting language > * register-based interpreter (heavily optimized) in Linux kernel > * small and lightweight > * not depend on the GCC toolchain for each script run > * easy to use in embedded environments without debugging info > * support for tracepoint, kprobe, uprobe, function trace, timer, and more > * supported in x86, ARM, PowerPC, MIPS > * safety in sandbox Is this the minimal code base of ktap for upstream? It seems that this series includes many extras. Would you slim this down to the minimum function set (minimum runtime and lua compiler code)? If we succeed to merge the basic functionality, we can add other extra features one by one. Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
* Jovi Zhangwei wrote: > Hi All, > > The following set of patches add ktap tracing tool. > > ktap is a new script-based dynamic tracing tool for Linux. > It uses a scripting language and lets the user trace system dynamically. > > Highlights features: > * a simple but powerful scripting language > * register-based interpreter (heavily optimized) in Linux kernel > * small and lightweight > * not depend on the GCC toolchain for each script run > * easy to use in embedded environments without debugging info > * support for tracepoint, kprobe, uprobe, function trace, timer, and more > * supported in x86, ARM, PowerPC, MIPS > * safety in sandbox I've asked this fundamental design question before but got no full answer: how does ktap compare to the ongoing effort of improving the BPF scripting engine? There's several efforts here that I'm aware of: 1) 64-bit BPF, integration with ftrace scripting, see this lkml thread: [RFC PATCH v2 tip 0/7] 64-bit BPF insn set and tracing filters 2) better BPF integration with networking: [PATCH net-next v3 8/9] net: filter: rework/optimize internal BPF interpreter's instruction set Your patches introduce a separate bytecode interpreter in kernel/trace/ktap/ and that's overlapping with BPF. >From a long term instrumentation code maintenance point of view the last thing we want is several overlapping scripting engines. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Sat, Mar 29, 2014 at 12:08 AM, Greg Kroah-Hartman wrote: > On Fri, Mar 28, 2014 at 09:47:21AM -0400, Jovi Zhangwei wrote: >> Hi All, >> >> The following set of patches add ktap tracing tool. > > Yeah! > > Thanks for putting this patch set together, it looks good. > > But, why hide the kernel portions of ktap in the tools/ directory? Is > that ok with everyone else? I don't expect to see kernel code burried > in that directory, so any kernel API changes that might happen, might > not get propagated into that location. > Reasonable, I will try to put kernel module side into kernel/trace/ktap directory in next version. There have some duplicated code with kernel/trace/*.c, then maybe we can integrate it if ktap location more closer with ftrace. Thanks for this suggestion. Jovi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/28] ktap: A lightweight dynamic tracing tool for Linux
On Fri, Mar 28, 2014 at 09:47:21AM -0400, Jovi Zhangwei wrote: > Hi All, > > The following set of patches add ktap tracing tool. Yeah! Thanks for putting this patch set together, it looks good. But, why hide the kernel portions of ktap in the tools/ directory? Is that ok with everyone else? I don't expect to see kernel code burried in that directory, so any kernel API changes that might happen, might not get propagated into that location. thanks, greg k-h -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
