Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 18:47 UTC-0700 ~ Alexei Starovoitov> On Mon, Apr 09, 2018 at 02:25:26PM +0100, Quentin Monnet wrote: >> >> Anyway, I am fine with keeping just signatures, descriptions and return >> values for now. I will submit a new version with only those items. > > Thank you. > > Could you also split it into few patches? > include/uapi/linux/bpf.h | 2237 > > scripts/bpf_helpers_doc.py | 568 +++ > 2 files changed, 2429 insertions(+), 376 deletions(-) > > replying back and forth on a single patch of such size will be tedious > for others to follow. > May be document ~10 helpers at a time ? Total of ~7 patches and extra > patch for .py ? > Sure, I'll do that. And I'll try to group helpers in a patch by author, it should also help for reviewing the descriptions. Quentin
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On Mon, Apr 09, 2018 at 02:25:26PM +0100, Quentin Monnet wrote: > > Anyway, I am fine with keeping just signatures, descriptions and return > values for now. I will submit a new version with only those items. Thank you. Could you also split it into few patches? include/uapi/linux/bpf.h | 2237 scripts/bpf_helpers_doc.py | 568 +++ 2 files changed, 2429 insertions(+), 376 deletions(-) replying back and forth on a single patch of such size will be tedious for others to follow. May be document ~10 helpers at a time ? Total of ~7 patches and extra patch for .py ?
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 12:52 UTC+0200 ~ Markus Heiser> >> Am 09.04.2018 um 12:08 schrieb Daniel Borkmann : > [...] > >>> May I completely misunderstood you, so correct my if I'am wrong: >>> >>> - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments >>> - ./scripts/kerne-doc : produces reST markup from C-comments >>> >>> IMO: both are doing the same job, so why not using kernel-doc? >> >> They are not really doing the same job, in bpf_helpers_doc.py case you don't >> want the whole header rendered, but just a fraction of it, that is, the >> single big comment which describes all BPF helper functions that a BPF >> program developer has available to use in user space - aka the entries in >> the __BPF_FUNC_MAPPER() macro; > > >> I also doubt the latter would actually qualify >> in kdoc context as some sort of a function description. > > latter .. ah, OK .. thanks for clarifying. > > -- Markus -- As Daniel explained, kernel-doc does not apply in this case, we do not have the full function prototype for eBPF helpers in the header file. But to be honest, I didn't even realise kernel-doc was available and could do something close to what I was looking for, so thanks for your feedback! :) Quentin
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 11:01 UTC+0200 ~ Daniel Borkmann> On 04/06/2018 01:11 PM, Quentin Monnet wrote: >> eBPF helper functions can be called from within eBPF programs to perform >> a variety of tasks that would be otherwise hard or impossible to do with >> eBPF itself. There is a growing number of such helper functions in the >> kernel, but documentation is scarce. The main user space header file >> does contain a short commented description of most helpers, but it is >> somewhat outdated and not complete. It is more a "cheat sheet" than a >> real documentation accessible to new eBPF developers. >> >> This commit attempts to improve the situation by replacing the existing >> overview for the helpers with a more developed description. Furthermore, >> a Python script is added to generate a manual page for eBPF helpers. The >> workflow is the following, and requires the rst2man utility: >> >> $ ./scripts/bpf_helpers_doc.py \ >> --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst >> $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 >> $ man /tmp/bpf-helpers.7 >> >> The objective is to keep all documentation related to the helpers in a >> single place, and to be able to generate from here a manual page that >> could be packaged in the man-pages repository and shipped with most >> distributions [1]. >> >> Additionally, parsing the prototypes of the helper functions could >> hopefully be reused, with a different Printer object, to generate >> header files needed in some eBPF-related projects. >> >> Regarding the description of each helper, it comprises several items: >> >> - The function prototype. >> - A description of the function and of its arguments (except for a >> couple of cases, when there are no arguments and the return value >> makes the function usage really obvious). >> - A description of return values (if not void). >> - A listing of eBPF program types (if relevant, map types) compatible >> with the helper. >> - Information about the helper being restricted to GPL programs, or not. >> - The kernel version in which the helper was introduced. >> - The commit that introduced the helper (this is mostly to have it in >> the source of the man page, as it can be used to track changes and >> update the page). >> >> For several helpers, descriptions are inspired (at times, nearly copied) >> from the commit logs introducing them in the kernel--Many thanks to >> their respective authors! They were completed as much as possible, the >> objective being to have something easily accessible even for people just >> starting with eBPF. There is probably a bit more work to do in this >> direction for some helpers. >> >> Some RST formatting is used in the descriptions (not in function >> prototypes, to keep them readable, but the Python script provided in >> order to generate the RST for the manual page does add formatting to >> prototypes, to produce something pretty) to get "bold" and "italics" in >> manual pages. Hopefully, the descriptions in bpf.h file remains >> perfectly readable. Note that the few trailing white spaces are >> intentional, removing them would break paragraphs for rst2man. >> >> The descriptions should ideally be updated each time someone adds a new >> helper, or updates the behaviour (compatibility extended to new program >> types, new socket option supported...) or the interface (new flags >> available, ...) of existing ones. >> >> [1] I have not contacted people from the man-pages project prior to >> sending this RFC, so I can offer no guaranty at this time that they >> would accept to take the generated man page. >> >> Cc: linux-...@vger.kernel.org >> Cc: linux-...@vger.kernel.org >> Signed-off-by: Quentin Monnet > > Great work, thanks a lot for doing this! > > [...] >> + * int bpf_probe_read(void *dst, u32 size, const void *src) >> + * Description >> + * For tracing programs, safely attempt to read *size* bytes from >> + * address *src* and store the data in *dst*. >> + * Return >> + * 0 on success, or a negative error in case of failure. >> + * For >> + * *Tracing programs*. >> + * GPL only >> + * Yes >> + * Since >> + * Linux 4.1 >> + * >> + * .. commit 2541517c32be >> * >> * u64 bpf_ktime_get_ns(void) >> - * Return: current ktime >> - * >> - * int bpf_trace_printk(const char *fmt, int fmt_size, ...) >> - * Return: length of buffer written or negative error >> + * Description >> + * Return the time elapsed since system boot, in nanoseconds. >> + * Return >> + * Current *ktime*. >> + * For >> + * All program types, except >> + * **BPF_PROG_TYPE_CGROUP_DEVICE**. > > I think we should probably always just list the actual program types instead, > since when new types get added to the kernel not supporting bpf_ktime_get_ns() > helper in this example, the above statement would be misleading
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
> Am 09.04.2018 um 12:08 schrieb Daniel Borkmann: [...] >> May I completely misunderstood you, so correct my if I'am wrong: >> >> - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments >> - ./scripts/kerne-doc : produces reST markup from C-comments >> >> IMO: both are doing the same job, so why not using kernel-doc? > > They are not really doing the same job, in bpf_helpers_doc.py case you don't > want the whole header rendered, but just a fraction of it, that is, the > single big comment which describes all BPF helper functions that a BPF > program developer has available to use in user space - aka the entries in > the __BPF_FUNC_MAPPER() macro; > I also doubt the latter would actually qualify > in kdoc context as some sort of a function description. latter .. ah, OK .. thanks for clarifying. -- Markus --
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/09/2018 11:35 AM, Markus Heiser wrote: > >> Am 09.04.2018 um 11:25 schrieb Daniel Borkmann: >> >> On 04/09/2018 11:21 AM, Markus Heiser wrote: >> [...] >>> Do we really need another kernel-doc parser? >>> >>> ./scripts/kernel-doc include/uapi/linux/bpf.h >>> >>> should already do the job (producing .rst). For more infos, take a look at >> >> This has absolutely zero to do with kernel-doc, but rather producing >> a description of BPF helper function that are later assembled into an >> actual man-page that BPF program developers (user space) can use. > > May I completely misunderstood you, so correct my if I'am wrong: > > - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments > - ./scripts/kerne-doc : produces reST markup from C-comments > > IMO: both are doing the same job, so why not using kernel-doc? They are not really doing the same job, in bpf_helpers_doc.py case you don't want the whole header rendered, but just a fraction of it, that is, the single big comment which describes all BPF helper functions that a BPF program developer has available to use in user space - aka the entries in the __BPF_FUNC_MAPPER() macro; I also doubt the latter would actually qualify in kdoc context as some sort of a function description.
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
> Am 09.04.2018 um 11:25 schrieb Daniel Borkmann: > > On 04/09/2018 11:21 AM, Markus Heiser wrote: > [...] >> Do we really need another kernel-doc parser? >> >> ./scripts/kernel-doc include/uapi/linux/bpf.h >> >> should already do the job (producing .rst). For more infos, take a look at > > This has absolutely zero to do with kernel-doc, but rather producing > a description of BPF helper function that are later assembled into an > actual man-page that BPF program developers (user space) can use. May I completely misunderstood you, so correct my if I'am wrong: - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments - ./scripts/kerne-doc : produces reST markup from C-comments IMO: both are doing the same job, so why not using kernel-doc? -- Markus --
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/06/2018 01:11 PM, Quentin Monnet wrote: >> eBPF helper functions can be called from within eBPF programs to perform >> a variety of tasks that would be otherwise hard or impossible to do with >> eBPF itself. There is a growing number of such helper functions in the >> kernel, but documentation is scarce. The main user space header file >> does contain a short commented description of most helpers, but it is >> somewhat outdated and not complete. It is more a "cheat sheet" than a >> real documentation accessible to new eBPF developers. >> >> This commit attempts to improve the situation by replacing the existing >> overview for the helpers with a more developed description. Furthermore, >> a Python script is added to generate a manual page for eBPF helpers. The >> workflow is the following, and requires the rst2man utility: >> >>$ ./scripts/bpf_helpers_doc.py \ >>--filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst >>$ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 >>$ man /tmp/bpf-helpers.7 >> >> The objective is to keep all documentation related to the helpers in a >> single place, Do we really need another kernel-doc parser? ./scripts/kernel-doc include/uapi/linux/bpf.h should already do the job (producing .rst). For more infos, take a look at https://www.kernel.org/doc/html/latest/doc-guide/index.html -- Markus -- PS: sorry for re-post, first post was HTML which is not accepted by ML :o
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/09/2018 11:21 AM, Markus Heiser wrote: [...] > Do we really need another kernel-doc parser? > > ./scripts/kernel-doc include/uapi/linux/bpf.h > > should already do the job (producing .rst). For more infos, take a look at This has absolutely zero to do with kernel-doc, but rather producing a description of BPF helper function that are later assembled into an actual man-page that BPF program developers (user space) can use.
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/06/2018 01:11 PM, Quentin Monnet wrote: > eBPF helper functions can be called from within eBPF programs to perform > a variety of tasks that would be otherwise hard or impossible to do with > eBPF itself. There is a growing number of such helper functions in the > kernel, but documentation is scarce. The main user space header file > does contain a short commented description of most helpers, but it is > somewhat outdated and not complete. It is more a "cheat sheet" than a > real documentation accessible to new eBPF developers. > > This commit attempts to improve the situation by replacing the existing > overview for the helpers with a more developed description. Furthermore, > a Python script is added to generate a manual page for eBPF helpers. The > workflow is the following, and requires the rst2man utility: > > $ ./scripts/bpf_helpers_doc.py \ > --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst > $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 > $ man /tmp/bpf-helpers.7 > > The objective is to keep all documentation related to the helpers in a > single place, and to be able to generate from here a manual page that > could be packaged in the man-pages repository and shipped with most > distributions [1]. > > Additionally, parsing the prototypes of the helper functions could > hopefully be reused, with a different Printer object, to generate > header files needed in some eBPF-related projects. > > Regarding the description of each helper, it comprises several items: > > - The function prototype. > - A description of the function and of its arguments (except for a > couple of cases, when there are no arguments and the return value > makes the function usage really obvious). > - A description of return values (if not void). > - A listing of eBPF program types (if relevant, map types) compatible > with the helper. > - Information about the helper being restricted to GPL programs, or not. > - The kernel version in which the helper was introduced. > - The commit that introduced the helper (this is mostly to have it in > the source of the man page, as it can be used to track changes and > update the page). > > For several helpers, descriptions are inspired (at times, nearly copied) > from the commit logs introducing them in the kernel--Many thanks to > their respective authors! They were completed as much as possible, the > objective being to have something easily accessible even for people just > starting with eBPF. There is probably a bit more work to do in this > direction for some helpers. > > Some RST formatting is used in the descriptions (not in function > prototypes, to keep them readable, but the Python script provided in > order to generate the RST for the manual page does add formatting to > prototypes, to produce something pretty) to get "bold" and "italics" in > manual pages. Hopefully, the descriptions in bpf.h file remains > perfectly readable. Note that the few trailing white spaces are > intentional, removing them would break paragraphs for rst2man. > > The descriptions should ideally be updated each time someone adds a new > helper, or updates the behaviour (compatibility extended to new program > types, new socket option supported...) or the interface (new flags > available, ...) of existing ones. > > [1] I have not contacted people from the man-pages project prior to > sending this RFC, so I can offer no guaranty at this time that they > would accept to take the generated man page. > > Cc: linux-...@vger.kernel.org > Cc: linux-...@vger.kernel.org > Signed-off-by: Quentin MonnetGreat work, thanks a lot for doing this! [...] > + * int bpf_probe_read(void *dst, u32 size, const void *src) > + * Description > + * For tracing programs, safely attempt to read *size* bytes from > + * address *src* and store the data in *dst*. > + * Return > + * 0 on success, or a negative error in case of failure. > + * For > + * *Tracing programs*. > + * GPL only > + * Yes > + * Since > + * Linux 4.1 > + * > + * .. commit 2541517c32be > * > * u64 bpf_ktime_get_ns(void) > - * Return: current ktime > - * > - * int bpf_trace_printk(const char *fmt, int fmt_size, ...) > - * Return: length of buffer written or negative error > + * Description > + * Return the time elapsed since system boot, in nanoseconds. > + * Return > + * Current *ktime*. > + * For > + * All program types, except > + * **BPF_PROG_TYPE_CGROUP_DEVICE**. I think we should probably always just list the actual program types instead, since when new types get added to the kernel not supporting bpf_ktime_get_ns() helper in this example, the above statement would be misleading (potentially more misleading than the other way around when it's not yet mentioned to be supported). > + * GPL only > + * Yes > + * Since
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On Fri, Apr 06, 2018 at 12:11:22PM +0100, Quentin Monnet wrote: > eBPF helper functions can be called from within eBPF programs to perform > a variety of tasks that would be otherwise hard or impossible to do with > eBPF itself. There is a growing number of such helper functions in the > kernel, but documentation is scarce. The main user space header file > does contain a short commented description of most helpers, but it is > somewhat outdated and not complete. It is more a "cheat sheet" than a > real documentation accessible to new eBPF developers. > > This commit attempts to improve the situation by replacing the existing > overview for the helpers with a more developed description. Furthermore, > a Python script is added to generate a manual page for eBPF helpers. The > workflow is the following, and requires the rst2man utility: > > $ ./scripts/bpf_helpers_doc.py \ > --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst > $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 > $ man /tmp/bpf-helpers.7 > > The objective is to keep all documentation related to the helpers in a > single place, and to be able to generate from here a manual page that > could be packaged in the man-pages repository and shipped with most > distributions [1]. > > Additionally, parsing the prototypes of the helper functions could > hopefully be reused, with a different Printer object, to generate > header files needed in some eBPF-related projects. > > Regarding the description of each helper, it comprises several items: > > - The function prototype. > - A description of the function and of its arguments (except for a > couple of cases, when there are no arguments and the return value > makes the function usage really obvious). > - A description of return values (if not void). > - A listing of eBPF program types (if relevant, map types) compatible > with the helper. > - Information about the helper being restricted to GPL programs, or not. > - The kernel version in which the helper was introduced. > - The commit that introduced the helper (this is mostly to have it in > the source of the man page, as it can be used to track changes and > update the page). > > For several helpers, descriptions are inspired (at times, nearly copied) > from the commit logs introducing them in the kernel--Many thanks to > their respective authors! They were completed as much as possible, the > objective being to have something easily accessible even for people just > starting with eBPF. There is probably a bit more work to do in this > direction for some helpers. > > Some RST formatting is used in the descriptions (not in function > prototypes, to keep them readable, but the Python script provided in > order to generate the RST for the manual page does add formatting to > prototypes, to produce something pretty) to get "bold" and "italics" in > manual pages. Hopefully, the descriptions in bpf.h file remains > perfectly readable. Note that the few trailing white spaces are > intentional, removing them would break paragraphs for rst2man. > > The descriptions should ideally be updated each time someone adds a new > helper, or updates the behaviour (compatibility extended to new program > types, new socket option supported...) or the interface (new flags > available, ...) of existing ones. > > [1] I have not contacted people from the man-pages project prior to > sending this RFC, so I can offer no guaranty at this time that they > would accept to take the generated man page. > > Cc: linux-...@vger.kernel.org > Cc: linux-...@vger.kernel.org > Signed-off-by: Quentin Monnet> --- > include/uapi/linux/bpf.h | 2237 > > scripts/bpf_helpers_doc.py | 568 +++ > 2 files changed, 2429 insertions(+), 376 deletions(-) > create mode 100755 scripts/bpf_helpers_doc.py > > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index c5ec89732a8d..f47aeddbbe0a 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -367,394 +367,1879 @@ union bpf_attr { > > /* BPF helper function descriptions: > * > - * void *bpf_map_lookup_elem(, ) > - * Return: Map value or NULL > - * > - * int bpf_map_update_elem(, , , flags) > - * Return: 0 on success or negative error > - * > - * int bpf_map_delete_elem(, ) > - * Return: 0 on success or negative error > - * > - * int bpf_probe_read(void *dst, int size, void *src) > - * Return: 0 on success or negative error > + * void *bpf_map_lookup_elem(struct bpf_map *map, void *key) > + * Description > + * Perform a lookup in *map* for an entry associated to *key*. > + * Return > + * Map value associated to *key*, or **NULL** if no entry was > + * found. > + * For > + * All types of programs. Limited to maps of types > + * **BPF_MAP_TYPE_HASH**, > + * **BPF_MAP_TYPE_ARRAY**, >
[RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
eBPF helper functions can be called from within eBPF programs to perform a variety of tasks that would be otherwise hard or impossible to do with eBPF itself. There is a growing number of such helper functions in the kernel, but documentation is scarce. The main user space header file does contain a short commented description of most helpers, but it is somewhat outdated and not complete. It is more a "cheat sheet" than a real documentation accessible to new eBPF developers. This commit attempts to improve the situation by replacing the existing overview for the helpers with a more developed description. Furthermore, a Python script is added to generate a manual page for eBPF helpers. The workflow is the following, and requires the rst2man utility: $ ./scripts/bpf_helpers_doc.py \ --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 $ man /tmp/bpf-helpers.7 The objective is to keep all documentation related to the helpers in a single place, and to be able to generate from here a manual page that could be packaged in the man-pages repository and shipped with most distributions [1]. Additionally, parsing the prototypes of the helper functions could hopefully be reused, with a different Printer object, to generate header files needed in some eBPF-related projects. Regarding the description of each helper, it comprises several items: - The function prototype. - A description of the function and of its arguments (except for a couple of cases, when there are no arguments and the return value makes the function usage really obvious). - A description of return values (if not void). - A listing of eBPF program types (if relevant, map types) compatible with the helper. - Information about the helper being restricted to GPL programs, or not. - The kernel version in which the helper was introduced. - The commit that introduced the helper (this is mostly to have it in the source of the man page, as it can be used to track changes and update the page). For several helpers, descriptions are inspired (at times, nearly copied) from the commit logs introducing them in the kernel--Many thanks to their respective authors! They were completed as much as possible, the objective being to have something easily accessible even for people just starting with eBPF. There is probably a bit more work to do in this direction for some helpers. Some RST formatting is used in the descriptions (not in function prototypes, to keep them readable, but the Python script provided in order to generate the RST for the manual page does add formatting to prototypes, to produce something pretty) to get "bold" and "italics" in manual pages. Hopefully, the descriptions in bpf.h file remains perfectly readable. Note that the few trailing white spaces are intentional, removing them would break paragraphs for rst2man. The descriptions should ideally be updated each time someone adds a new helper, or updates the behaviour (compatibility extended to new program types, new socket option supported...) or the interface (new flags available, ...) of existing ones. [1] I have not contacted people from the man-pages project prior to sending this RFC, so I can offer no guaranty at this time that they would accept to take the generated man page. Cc: linux-...@vger.kernel.org Cc: linux-...@vger.kernel.org Signed-off-by: Quentin Monnet--- include/uapi/linux/bpf.h | 2237 scripts/bpf_helpers_doc.py | 568 +++ 2 files changed, 2429 insertions(+), 376 deletions(-) create mode 100755 scripts/bpf_helpers_doc.py diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c5ec89732a8d..f47aeddbbe0a 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -367,394 +367,1879 @@ union bpf_attr { /* BPF helper function descriptions: * - * void *bpf_map_lookup_elem(, ) - * Return: Map value or NULL - * - * int bpf_map_update_elem(, , , flags) - * Return: 0 on success or negative error - * - * int bpf_map_delete_elem(, ) - * Return: 0 on success or negative error - * - * int bpf_probe_read(void *dst, int size, void *src) - * Return: 0 on success or negative error + * void *bpf_map_lookup_elem(struct bpf_map *map, void *key) + * Description + * Perform a lookup in *map* for an entry associated to *key*. + * Return + * Map value associated to *key*, or **NULL** if no entry was + * found. + * For + * All types of programs. Limited to maps of types + * **BPF_MAP_TYPE_HASH**, + * **BPF_MAP_TYPE_ARRAY**, + * **BPF_MAP_TYPE_PERCPU_HASH**, + * **BPF_MAP_TYPE_PERCPU_ARRAY**, + * **BPF_MAP_TYPE_LRU_HASH**, + * **BPF_MAP_TYPE_LRU_PERCPU_HASH**, + * **BPF_MAP_TYPE_LPM_TRIE**, + *