Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
On Mon, Mar 31, 2014 at 1:19 AM, Andi Kleen wrote:
>> See test/benchmark/cmp_table.sh, that script compare
>
> Is that a realistic tracing scenario?
>
Yes, it's quite common to use string key in dynamic tracing tool,
for example, See samples/userspace/glibc_func_hist.kp
var s = {}
trace probe:/lib64/libc.so.6:* {
s[probename] += 1
}
trace_end {
print_hist(s)
}
Result:
Tracing... Hit Ctrl-C to end.
^C
value - Distribution - count
_IO_sputbackc |@@108344
__GI__IO_sputbackc |@@107768
_IO_default_xsputn | 46639
__GI__IO_default_xsputn | 46624
free | 36871
__libc_free | 36841
cfree | 36841
__free | 36811
__cfree | 36811
__GI___libc_free | 36804
strtoull_l_internal | 28670
__GI_strtoul_l_internal | 28670
__GI_strtoull_l_internal | 28518
strtoul_l_internal | 28518
strchrnul | 27763
__strchrnul | 27741
_IO_putc | 27589
__GI__IO_putc | 27589
putc | 27589
... |
Above script output histogram of glibc function call, you will know
which function will be called frequently, a very useful script.
'probename' return probe name string, then insert table as key.
The magic of above script is there have no string copy and string hash
in probe context, because probename string is interned.
>> table operation between ktap with stap, the result is very
>> inspiring, ktap table operation overhead is quite lower than
>> stap, especially when use constant string key.
>
> Ok fair enough.
>
>>
>> But I agree with you partly, because in some cases we don't
>> want/need to interning all string, for example:
>> trace xxx:yyy {
>> var str = cast("char *", arg1)
>> print(str)
>> }
>>
>> In above case, arg1 is a long kernel string, and no table insert,
>> so definitely no need to interned, so we need to add
>> KTAP_TRAWSTR to represent these values.
>
> Please don't make it more complicated. If there's a good rationale
> for interning it' ok to use always.
>
> It would be better to find ways to simplify things.
>
Definitely, the reason I implement ktap based on lua is the simplicity
and efficiency of lua.
The whole bytecode design and value type is very simple, it could
build a complete safe sandbox for kernel scripting, and easy to
extend to fulfill our need(like multi-key table, aggregation)
Thanks.
Jovi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
> See test/benchmark/cmp_table.sh, that script compare
Is that a realistic tracing scenario?
> table operation between ktap with stap, the result is very
> inspiring, ktap table operation overhead is quite lower than
> stap, especially when use constant string key.
Ok fair enough.
>
> But I agree with you partly, because in some cases we don't
> want/need to interning all string, for example:
> trace xxx:yyy {
> var str = cast("char *", arg1)
> print(str)
> }
>
> In above case, arg1 is a long kernel string, and no table insert,
> so definitely no need to interned, so we need to add
> KTAP_TRAWSTR to represent these values.
Please don't make it more complicated. If there's a good rationale
for interning it' ok to use always.
It would be better to find ways to simplify things.
-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
On Sun, Mar 30, 2014 at 11:50 AM, Andi Kleen wrote:
>
> It's not clear to me why a kernel script language needs
> all that complicated string interning code.
>
> What kind of scripts would create as many strings that
> it would be worth it?
>
> I think it would be better to replace it with a really
> simple non interning dynamic string type.
>
Basically I think string interning is very useful in ktap, and
the implementation is not complicated(kp_str_new function
is very simple).
String interning will make string comparison and table index
extremely fast, just pointer equality, no strcmp. table index
is heavily used in these dynamic tracing tool ktap/stap/dtrace.
String interning make there don't need to copy whole string
each time when use string key in associative array(table)
(stap/dtrace need copy it), and don't need to compute
string hash every time when use string table key.
(Things became more easily if need to support multi-key
table, ktap don't need to pre-allocate string in table)
See test/benchmark/cmp_table.sh, that script compare
table operation between ktap with stap, the result is very
inspiring, ktap table operation overhead is quite lower than
stap, especially when use constant string key.
But I agree with you partly, because in some cases we don't
want/need to interning all string, for example:
trace xxx:yyy {
var str = cast("char *", arg1)
print(str)
}
In above case, arg1 is a long kernel string, and no table insert,
so definitely no need to interned, so we need to add
KTAP_TRAWSTR to represent these values.
The simplicity design of ktap make it very flexible to support
different kind of value type. :)
Thanks.
Jovi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
On Sun, Mar 30, 2014 at 11:50 AM, Andi Kleen a...@firstfloor.org wrote:
It's not clear to me why a kernel script language needs
all that complicated string interning code.
What kind of scripts would create as many strings that
it would be worth it?
I think it would be better to replace it with a really
simple non interning dynamic string type.
Basically I think string interning is very useful in ktap, and
the implementation is not complicated(kp_str_new function
is very simple).
String interning will make string comparison and table index
extremely fast, just pointer equality, no strcmp. table index
is heavily used in these dynamic tracing tool ktap/stap/dtrace.
String interning make there don't need to copy whole string
each time when use string key in associative array(table)
(stap/dtrace need copy it), and don't need to compute
string hash every time when use string table key.
(Things became more easily if need to support multi-key
table, ktap don't need to pre-allocate string in table)
See test/benchmark/cmp_table.sh, that script compare
table operation between ktap with stap, the result is very
inspiring, ktap table operation overhead is quite lower than
stap, especially when use constant string key.
But I agree with you partly, because in some cases we don't
want/need to interning all string, for example:
trace xxx:yyy {
var str = cast(char *, arg1)
print(str)
}
In above case, arg1 is a long kernel string, and no table insert,
so definitely no need to interned, so we need to add
KTAP_TRAWSTR to represent these values.
The simplicity design of ktap make it very flexible to support
different kind of value type. :)
Thanks.
Jovi
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
See test/benchmark/cmp_table.sh, that script compare
Is that a realistic tracing scenario?
table operation between ktap with stap, the result is very
inspiring, ktap table operation overhead is quite lower than
stap, especially when use constant string key.
Ok fair enough.
But I agree with you partly, because in some cases we don't
want/need to interning all string, for example:
trace xxx:yyy {
var str = cast(char *, arg1)
print(str)
}
In above case, arg1 is a long kernel string, and no table insert,
so definitely no need to interned, so we need to add
KTAP_TRAWSTR to represent these values.
Please don't make it more complicated. If there's a good rationale
for interning it' ok to use always.
It would be better to find ways to simplify things.
-Andi
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
On Mon, Mar 31, 2014 at 1:19 AM, Andi Kleen a...@firstfloor.org wrote:
See test/benchmark/cmp_table.sh, that script compare
Is that a realistic tracing scenario?
Yes, it's quite common to use string key in dynamic tracing tool,
for example, See samples/userspace/glibc_func_hist.kp
var s = {}
trace probe:/lib64/libc.so.6:* {
s[probename] += 1
}
trace_end {
print_hist(s)
}
Result:
Tracing... Hit Ctrl-C to end.
^C
value - Distribution - count
_IO_sputbackc |@@108344
__GI__IO_sputbackc |@@107768
_IO_default_xsputn | 46639
__GI__IO_default_xsputn | 46624
free | 36871
__libc_free | 36841
cfree | 36841
__free | 36811
__cfree | 36811
__GI___libc_free | 36804
strtoull_l_internal | 28670
__GI_strtoul_l_internal | 28670
__GI_strtoull_l_internal | 28518
strtoul_l_internal | 28518
strchrnul | 27763
__strchrnul | 27741
_IO_putc | 27589
__GI__IO_putc | 27589
putc | 27589
... |
Above script output histogram of glibc function call, you will know
which function will be called frequently, a very useful script.
'probename' return probe name string, then insert table as key.
The magic of above script is there have no string copy and string hash
in probe context, because probename string is interned.
table operation between ktap with stap, the result is very
inspiring, ktap table operation overhead is quite lower than
stap, especially when use constant string key.
Ok fair enough.
But I agree with you partly, because in some cases we don't
want/need to interning all string, for example:
trace xxx:yyy {
var str = cast(char *, arg1)
print(str)
}
In above case, arg1 is a long kernel string, and no table insert,
so definitely no need to interned, so we need to add
KTAP_TRAWSTR to represent these values.
Please don't make it more complicated. If there's a good rationale
for interning it' ok to use always.
It would be better to find ways to simplify things.
Definitely, the reason I implement ktap based on lua is the simplicity
and efficiency of lua.
The whole bytecode design and value type is very simple, it could
build a complete safe sandbox for kernel scripting, and easy to
extend to fulfill our need(like multi-key table, aggregation)
Thanks.
Jovi
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
It's not clear to me why a kernel script language needs all that complicated string interning code. What kind of scripts would create as many strings that it would be worth it? I think it would be better to replace it with a really simple non interning dynamic string type. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
Exposed functions:
1). kp_str_new:
Return a interned string, failure if out of memory
or exceed max string number(default ).
It allocate memory from mempool, not kmalloc, because we
don't want dynamic allocate string in probe context.
2). kp_str_resize:
Initizate interned hash table g->strhash.
3). kp_str_fmt:
Return a format string, called from 'printf' built-in function.
4). kp_mempool_init/kp_mempool_destroy/kp_mempool_alloc
mempool is only service for string allocation currently.
All string in ktap is interned, it means ktap keeps
a single copy for any string. Whenever a new string appears, ktap
checks whether it already has a copy of that string and, if so,
reuses that copy. Internalization makes operations like string
comparison and table indexing very fast, but it slows down string creation.
Signed-off-by: Jovi Zhangwei
---
kernel/trace/ktap/kp_mempool.c | 94 +++
kernel/trace/ktap/kp_mempool.h | 8 +
kernel/trace/ktap/kp_str.c | 360 +
kernel/trace/ktap/kp_str.h | 13 ++
4 files changed, 475 insertions(+)
create mode 100644 kernel/trace/ktap/kp_mempool.c
create mode 100644 kernel/trace/ktap/kp_mempool.h
create mode 100644 kernel/trace/ktap/kp_str.c
create mode 100644 kernel/trace/ktap/kp_str.h
diff --git a/kernel/trace/ktap/kp_mempool.c b/kernel/trace/ktap/kp_mempool.c
new file mode 100644
index 000..37fb8e4
--- /dev/null
+++ b/kernel/trace/ktap/kp_mempool.c
@@ -0,0 +1,94 @@
+/*
+ * kp_mempool.c - ktap memory pool, service for string allocation
+ *
+ * This file is part of ktap by Jovi Zhangwei.
+ *
+ * Copyright (C) 2012-2014 Jovi Zhangwei .
+ *
+ * ktap is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * ktap is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#include
+#include "kp_obj.h"
+#include "kp_str.h"
+
+#include
+#include
+#include "ktap.h"
+
+
+/*
+ * allocate memory from mempool, the allocated memory will be free
+ * util ktap exit.
+ * TODO: lock-free allocation
+ */
+void *kp_mempool_alloc(ktap_state_t *ks, int size)
+{
+ ktap_global_state_t *g = G(ks);
+ void *mempool = g->mempool;
+ void *freepos = g->mp_freepos;
+ void *addr;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ arch_spin_lock(>mp_lock);
+
+ if (unlikely((unsigned long)((char *)freepos + size)) >
+(unsigned long)((char *)mempool + g->mp_size)) {
+ addr = NULL;
+ goto out;
+ }
+
+ addr = freepos;
+ g->mp_freepos = (char *)freepos + size;
+ out:
+
+ arch_spin_unlock(>mp_lock);
+ local_irq_restore(flags);
+ return addr;
+}
+
+/*
+ * destroy mempool.
+ */
+void kp_mempool_destroy(ktap_state_t *ks)
+{
+ ktap_global_state_t *g = G(ks);
+
+ if (!g->mempool)
+ return;
+
+ vfree(g->mempool);
+ g->mempool = NULL;
+ g->mp_freepos = NULL;
+ g->mp_size = 0;
+}
+
+/*
+ * pre-allocate size Kbytes memory pool.
+ */
+int kp_mempool_init(ktap_state_t *ks, int size)
+{
+ ktap_global_state_t *g = G(ks);
+
+ g->mempool = vmalloc(size * 1024);
+ if (!g->mempool)
+ return -ENOMEM;
+
+ g->mp_freepos = g->mempool;
+ g->mp_size = size * 1024;
+ g->mp_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
+ return 0;
+}
+
diff --git a/kernel/trace/ktap/kp_mempool.h b/kernel/trace/ktap/kp_mempool.h
new file mode 100644
index 000..3eabf5e
--- /dev/null
+++ b/kernel/trace/ktap/kp_mempool.h
@@ -0,0 +1,8 @@
+#ifndef __KTAP_MEMPOOL_H__
+#define __KTAP_MEMPOOL_H__
+
+void *kp_mempool_alloc(ktap_state_t *ks, int size);
+void kp_mempool_destroy(ktap_state_t *ks);
+int kp_mempool_init(ktap_state_t *ks, int size);
+
+#endif /* __KTAP_MEMPOOL_H__ */
diff --git a/kernel/trace/ktap/kp_str.c b/kernel/trace/ktap/kp_str.c
new file mode 100644
index 000..9d2e741
--- /dev/null
+++ b/kernel/trace/ktap/kp_str.c
@@ -0,0 +1,360 @@
+/*
+ * kp_str.c - ktap string data struction manipulation
+ *
+ * This file is part of ktap by Jovi Zhangwei.
+ *
+ * Copyright (C) 2012-2014 Jovi Zhangwei .
+ *
+ * Adapted from luajit and lua interpreter.
+ * Copyright (C) 2005-2014 Mike Pall.
+ * Copyright (C) 1994-2008 Lua.org, PUC-Rio.
+ *
+ * ktap is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public
[PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
Exposed functions:
1). kp_str_new:
Return a interned string, failure if out of memory
or exceed max string number(default ).
It allocate memory from mempool, not kmalloc, because we
don't want dynamic allocate string in probe context.
2). kp_str_resize:
Initizate interned hash table g-strhash.
3). kp_str_fmt:
Return a format string, called from 'printf' built-in function.
4). kp_mempool_init/kp_mempool_destroy/kp_mempool_alloc
mempool is only service for string allocation currently.
All string in ktap is interned, it means ktap keeps
a single copy for any string. Whenever a new string appears, ktap
checks whether it already has a copy of that string and, if so,
reuses that copy. Internalization makes operations like string
comparison and table indexing very fast, but it slows down string creation.
Signed-off-by: Jovi Zhangwei jovi.zhang...@gmail.com
---
kernel/trace/ktap/kp_mempool.c | 94 +++
kernel/trace/ktap/kp_mempool.h | 8 +
kernel/trace/ktap/kp_str.c | 360 +
kernel/trace/ktap/kp_str.h | 13 ++
4 files changed, 475 insertions(+)
create mode 100644 kernel/trace/ktap/kp_mempool.c
create mode 100644 kernel/trace/ktap/kp_mempool.h
create mode 100644 kernel/trace/ktap/kp_str.c
create mode 100644 kernel/trace/ktap/kp_str.h
diff --git a/kernel/trace/ktap/kp_mempool.c b/kernel/trace/ktap/kp_mempool.c
new file mode 100644
index 000..37fb8e4
--- /dev/null
+++ b/kernel/trace/ktap/kp_mempool.c
@@ -0,0 +1,94 @@
+/*
+ * kp_mempool.c - ktap memory pool, service for string allocation
+ *
+ * This file is part of ktap by Jovi Zhangwei.
+ *
+ * Copyright (C) 2012-2014 Jovi Zhangwei jovi.zhang...@gmail.com.
+ *
+ * ktap is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * ktap is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#include uapi/ktap/ktap_types.h
+#include kp_obj.h
+#include kp_str.h
+
+#include linux/ctype.h
+#include linux/module.h
+#include ktap.h
+
+
+/*
+ * allocate memory from mempool, the allocated memory will be free
+ * util ktap exit.
+ * TODO: lock-free allocation
+ */
+void *kp_mempool_alloc(ktap_state_t *ks, int size)
+{
+ ktap_global_state_t *g = G(ks);
+ void *mempool = g-mempool;
+ void *freepos = g-mp_freepos;
+ void *addr;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ arch_spin_lock(g-mp_lock);
+
+ if (unlikely((unsigned long)((char *)freepos + size))
+(unsigned long)((char *)mempool + g-mp_size)) {
+ addr = NULL;
+ goto out;
+ }
+
+ addr = freepos;
+ g-mp_freepos = (char *)freepos + size;
+ out:
+
+ arch_spin_unlock(g-mp_lock);
+ local_irq_restore(flags);
+ return addr;
+}
+
+/*
+ * destroy mempool.
+ */
+void kp_mempool_destroy(ktap_state_t *ks)
+{
+ ktap_global_state_t *g = G(ks);
+
+ if (!g-mempool)
+ return;
+
+ vfree(g-mempool);
+ g-mempool = NULL;
+ g-mp_freepos = NULL;
+ g-mp_size = 0;
+}
+
+/*
+ * pre-allocate size Kbytes memory pool.
+ */
+int kp_mempool_init(ktap_state_t *ks, int size)
+{
+ ktap_global_state_t *g = G(ks);
+
+ g-mempool = vmalloc(size * 1024);
+ if (!g-mempool)
+ return -ENOMEM;
+
+ g-mp_freepos = g-mempool;
+ g-mp_size = size * 1024;
+ g-mp_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
+ return 0;
+}
+
diff --git a/kernel/trace/ktap/kp_mempool.h b/kernel/trace/ktap/kp_mempool.h
new file mode 100644
index 000..3eabf5e
--- /dev/null
+++ b/kernel/trace/ktap/kp_mempool.h
@@ -0,0 +1,8 @@
+#ifndef __KTAP_MEMPOOL_H__
+#define __KTAP_MEMPOOL_H__
+
+void *kp_mempool_alloc(ktap_state_t *ks, int size);
+void kp_mempool_destroy(ktap_state_t *ks);
+int kp_mempool_init(ktap_state_t *ks, int size);
+
+#endif /* __KTAP_MEMPOOL_H__ */
diff --git a/kernel/trace/ktap/kp_str.c b/kernel/trace/ktap/kp_str.c
new file mode 100644
index 000..9d2e741
--- /dev/null
+++ b/kernel/trace/ktap/kp_str.c
@@ -0,0 +1,360 @@
+/*
+ * kp_str.c - ktap string data struction manipulation
+ *
+ * This file is part of ktap by Jovi Zhangwei.
+ *
+ * Copyright (C) 2012-2014 Jovi Zhangwei jovi.zhang...@gmail.com.
+ *
+ * Adapted from luajit and lua interpreter.
+ * Copyright (C) 2005-2014 Mike Pall.
+ * Copyright (C) 1994-2008 Lua.org, PUC-Rio.
+ *
+ * ktap is free software; you can
Re: [PATCH v2 10/29] ktap: add string handling code(kernel/trace/ktap/kp_[str|mempool].[c|h])
It's not clear to me why a kernel script language needs all that complicated string interning code. What kind of scripts would create as many strings that it would be worth it? I think it would be better to replace it with a really simple non interning dynamic string type. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
