Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On Fri, Apr 06, 2018 at 12:11:22PM +0100, Quentin Monnet wrote:
> eBPF helper functions can be called from within eBPF programs to perform
> a variety of tasks that would be otherwise hard or impossible to do with
> eBPF itself. There is a growing number of such helper functions in the
> kernel, but documentation is scarce. The main user space header file
> does contain a short commented description of most helpers, but it is
> somewhat outdated and not complete. It is more a "cheat sheet" than a
> real documentation accessible to new eBPF developers.
>
> This commit attempts to improve the situation by replacing the existing
> overview for the helpers with a more developed description. Furthermore,
> a Python script is added to generate a manual page for eBPF helpers. The
> workflow is the following, and requires the rst2man utility:
>
> $ ./scripts/bpf_helpers_doc.py \
> --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst
> $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7
> $ man /tmp/bpf-helpers.7
>
> The objective is to keep all documentation related to the helpers in a
> single place, and to be able to generate from here a manual page that
> could be packaged in the man-pages repository and shipped with most
> distributions [1].
>
> Additionally, parsing the prototypes of the helper functions could
> hopefully be reused, with a different Printer object, to generate
> header files needed in some eBPF-related projects.
>
> Regarding the description of each helper, it comprises several items:
>
> - The function prototype.
> - A description of the function and of its arguments (except for a
> couple of cases, when there are no arguments and the return value
> makes the function usage really obvious).
> - A description of return values (if not void).
> - A listing of eBPF program types (if relevant, map types) compatible
> with the helper.
> - Information about the helper being restricted to GPL programs, or not.
> - The kernel version in which the helper was introduced.
> - The commit that introduced the helper (this is mostly to have it in
> the source of the man page, as it can be used to track changes and
> update the page).
>
> For several helpers, descriptions are inspired (at times, nearly copied)
> from the commit logs introducing them in the kernel--Many thanks to
> their respective authors! They were completed as much as possible, the
> objective being to have something easily accessible even for people just
> starting with eBPF. There is probably a bit more work to do in this
> direction for some helpers.
>
> Some RST formatting is used in the descriptions (not in function
> prototypes, to keep them readable, but the Python script provided in
> order to generate the RST for the manual page does add formatting to
> prototypes, to produce something pretty) to get "bold" and "italics" in
> manual pages. Hopefully, the descriptions in bpf.h file remains
> perfectly readable. Note that the few trailing white spaces are
> intentional, removing them would break paragraphs for rst2man.
>
> The descriptions should ideally be updated each time someone adds a new
> helper, or updates the behaviour (compatibility extended to new program
> types, new socket option supported...) or the interface (new flags
> available, ...) of existing ones.
>
> [1] I have not contacted people from the man-pages project prior to
> sending this RFC, so I can offer no guaranty at this time that they
> would accept to take the generated man page.
>
> Cc: linux-...@vger.kernel.org
> Cc: linux-...@vger.kernel.org
> Signed-off-by: Quentin Monnet
> ---
> include/uapi/linux/bpf.h | 2237
>
> scripts/bpf_helpers_doc.py | 568 +++
> 2 files changed, 2429 insertions(+), 376 deletions(-)
> create mode 100755 scripts/bpf_helpers_doc.py
>
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index c5ec89732a8d..f47aeddbbe0a 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -367,394 +367,1879 @@ union bpf_attr {
>
> /* BPF helper function descriptions:
> *
> - * void *bpf_map_lookup_elem(&map, &key)
> - * Return: Map value or NULL
> - *
> - * int bpf_map_update_elem(&map, &key, &value, flags)
> - * Return: 0 on success or negative error
> - *
> - * int bpf_map_delete_elem(&map, &key)
> - * Return: 0 on success or negative error
> - *
> - * int bpf_probe_read(void *dst, int size, void *src)
> - * Return: 0 on success or negative error
> + * void *bpf_map_lookup_elem(struct bpf_map *map, void *key)
> + * Description
> + * Perform a lookup in *map* for an entry associated to *key*.
> + * Return
> + * Map value associated to *key*, or **NULL** if no entry was
> + * found.
> + * For
> + * All types of programs. Limited to maps of types
> + * **BPF_MAP_TYPE_HASH**,
> + * **BPF_MAP_TYPE_ARRAY**,
>
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/06/2018 01:11 PM, Quentin Monnet wrote: > eBPF helper functions can be called from within eBPF programs to perform > a variety of tasks that would be otherwise hard or impossible to do with > eBPF itself. There is a growing number of such helper functions in the > kernel, but documentation is scarce. The main user space header file > does contain a short commented description of most helpers, but it is > somewhat outdated and not complete. It is more a "cheat sheet" than a > real documentation accessible to new eBPF developers. > > This commit attempts to improve the situation by replacing the existing > overview for the helpers with a more developed description. Furthermore, > a Python script is added to generate a manual page for eBPF helpers. The > workflow is the following, and requires the rst2man utility: > > $ ./scripts/bpf_helpers_doc.py \ > --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst > $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 > $ man /tmp/bpf-helpers.7 > > The objective is to keep all documentation related to the helpers in a > single place, and to be able to generate from here a manual page that > could be packaged in the man-pages repository and shipped with most > distributions [1]. > > Additionally, parsing the prototypes of the helper functions could > hopefully be reused, with a different Printer object, to generate > header files needed in some eBPF-related projects. > > Regarding the description of each helper, it comprises several items: > > - The function prototype. > - A description of the function and of its arguments (except for a > couple of cases, when there are no arguments and the return value > makes the function usage really obvious). > - A description of return values (if not void). > - A listing of eBPF program types (if relevant, map types) compatible > with the helper. > - Information about the helper being restricted to GPL programs, or not. > - The kernel version in which the helper was introduced. > - The commit that introduced the helper (this is mostly to have it in > the source of the man page, as it can be used to track changes and > update the page). > > For several helpers, descriptions are inspired (at times, nearly copied) > from the commit logs introducing them in the kernel--Many thanks to > their respective authors! They were completed as much as possible, the > objective being to have something easily accessible even for people just > starting with eBPF. There is probably a bit more work to do in this > direction for some helpers. > > Some RST formatting is used in the descriptions (not in function > prototypes, to keep them readable, but the Python script provided in > order to generate the RST for the manual page does add formatting to > prototypes, to produce something pretty) to get "bold" and "italics" in > manual pages. Hopefully, the descriptions in bpf.h file remains > perfectly readable. Note that the few trailing white spaces are > intentional, removing them would break paragraphs for rst2man. > > The descriptions should ideally be updated each time someone adds a new > helper, or updates the behaviour (compatibility extended to new program > types, new socket option supported...) or the interface (new flags > available, ...) of existing ones. > > [1] I have not contacted people from the man-pages project prior to > sending this RFC, so I can offer no guaranty at this time that they > would accept to take the generated man page. > > Cc: linux-...@vger.kernel.org > Cc: linux-...@vger.kernel.org > Signed-off-by: Quentin Monnet Great work, thanks a lot for doing this! [...] > + * int bpf_probe_read(void *dst, u32 size, const void *src) > + * Description > + * For tracing programs, safely attempt to read *size* bytes from > + * address *src* and store the data in *dst*. > + * Return > + * 0 on success, or a negative error in case of failure. > + * For > + * *Tracing programs*. > + * GPL only > + * Yes > + * Since > + * Linux 4.1 > + * > + * .. commit 2541517c32be > * > * u64 bpf_ktime_get_ns(void) > - * Return: current ktime > - * > - * int bpf_trace_printk(const char *fmt, int fmt_size, ...) > - * Return: length of buffer written or negative error > + * Description > + * Return the time elapsed since system boot, in nanoseconds. > + * Return > + * Current *ktime*. > + * For > + * All program types, except > + * **BPF_PROG_TYPE_CGROUP_DEVICE**. I think we should probably always just list the actual program types instead, since when new types get added to the kernel not supporting bpf_ktime_get_ns() helper in this example, the above statement would be misleading (potentially more misleading than the other way around when it's not yet mentioned to be supported). > + * GPL only > + * Yes > + * Since > + * Linux 4.1 > +
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/09/2018 11:21 AM, Markus Heiser wrote: [...] > Do we really need another kernel-doc parser? > > ./scripts/kernel-doc include/uapi/linux/bpf.h > > should already do the job (producing .rst). For more infos, take a look at This has absolutely zero to do with kernel-doc, but rather producing a description of BPF helper function that are later assembled into an actual man-page that BPF program developers (user space) can use.
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/06/2018 01:11 PM, Quentin Monnet wrote: >> eBPF helper functions can be called from within eBPF programs to perform >> a variety of tasks that would be otherwise hard or impossible to do with >> eBPF itself. There is a growing number of such helper functions in the >> kernel, but documentation is scarce. The main user space header file >> does contain a short commented description of most helpers, but it is >> somewhat outdated and not complete. It is more a "cheat sheet" than a >> real documentation accessible to new eBPF developers. >> >> This commit attempts to improve the situation by replacing the existing >> overview for the helpers with a more developed description. Furthermore, >> a Python script is added to generate a manual page for eBPF helpers. The >> workflow is the following, and requires the rst2man utility: >> >>$ ./scripts/bpf_helpers_doc.py \ >>--filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst >>$ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 >>$ man /tmp/bpf-helpers.7 >> >> The objective is to keep all documentation related to the helpers in a >> single place, Do we really need another kernel-doc parser? ./scripts/kernel-doc include/uapi/linux/bpf.h should already do the job (producing .rst). For more infos, take a look at https://www.kernel.org/doc/html/latest/doc-guide/index.html -- Markus -- PS: sorry for re-post, first post was HTML which is not accepted by ML :o
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
> Am 09.04.2018 um 11:25 schrieb Daniel Borkmann : > > On 04/09/2018 11:21 AM, Markus Heiser wrote: > [...] >> Do we really need another kernel-doc parser? >> >> ./scripts/kernel-doc include/uapi/linux/bpf.h >> >> should already do the job (producing .rst). For more infos, take a look at > > This has absolutely zero to do with kernel-doc, but rather producing > a description of BPF helper function that are later assembled into an > actual man-page that BPF program developers (user space) can use. May I completely misunderstood you, so correct my if I'am wrong: - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments - ./scripts/kerne-doc : produces reST markup from C-comments IMO: both are doing the same job, so why not using kernel-doc? -- Markus --
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On 04/09/2018 11:35 AM, Markus Heiser wrote: > >> Am 09.04.2018 um 11:25 schrieb Daniel Borkmann : >> >> On 04/09/2018 11:21 AM, Markus Heiser wrote: >> [...] >>> Do we really need another kernel-doc parser? >>> >>> ./scripts/kernel-doc include/uapi/linux/bpf.h >>> >>> should already do the job (producing .rst). For more infos, take a look at >> >> This has absolutely zero to do with kernel-doc, but rather producing >> a description of BPF helper function that are later assembled into an >> actual man-page that BPF program developers (user space) can use. > > May I completely misunderstood you, so correct my if I'am wrong: > > - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments > - ./scripts/kerne-doc : produces reST markup from C-comments > > IMO: both are doing the same job, so why not using kernel-doc? They are not really doing the same job, in bpf_helpers_doc.py case you don't want the whole header rendered, but just a fraction of it, that is, the single big comment which describes all BPF helper functions that a BPF program developer has available to use in user space - aka the entries in the __BPF_FUNC_MAPPER() macro; I also doubt the latter would actually qualify in kdoc context as some sort of a function description.
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
> Am 09.04.2018 um 12:08 schrieb Daniel Borkmann : [...] >> May I completely misunderstood you, so correct my if I'am wrong: >> >> - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments >> - ./scripts/kerne-doc : produces reST markup from C-comments >> >> IMO: both are doing the same job, so why not using kernel-doc? > > They are not really doing the same job, in bpf_helpers_doc.py case you don't > want the whole header rendered, but just a fraction of it, that is, the > single big comment which describes all BPF helper functions that a BPF > program developer has available to use in user space - aka the entries in > the __BPF_FUNC_MAPPER() macro; > I also doubt the latter would actually qualify > in kdoc context as some sort of a function description. latter .. ah, OK .. thanks for clarifying. -- Markus --
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 11:01 UTC+0200 ~ Daniel Borkmann > On 04/06/2018 01:11 PM, Quentin Monnet wrote: >> eBPF helper functions can be called from within eBPF programs to perform >> a variety of tasks that would be otherwise hard or impossible to do with >> eBPF itself. There is a growing number of such helper functions in the >> kernel, but documentation is scarce. The main user space header file >> does contain a short commented description of most helpers, but it is >> somewhat outdated and not complete. It is more a "cheat sheet" than a >> real documentation accessible to new eBPF developers. >> >> This commit attempts to improve the situation by replacing the existing >> overview for the helpers with a more developed description. Furthermore, >> a Python script is added to generate a manual page for eBPF helpers. The >> workflow is the following, and requires the rst2man utility: >> >> $ ./scripts/bpf_helpers_doc.py \ >> --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst >> $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7 >> $ man /tmp/bpf-helpers.7 >> >> The objective is to keep all documentation related to the helpers in a >> single place, and to be able to generate from here a manual page that >> could be packaged in the man-pages repository and shipped with most >> distributions [1]. >> >> Additionally, parsing the prototypes of the helper functions could >> hopefully be reused, with a different Printer object, to generate >> header files needed in some eBPF-related projects. >> >> Regarding the description of each helper, it comprises several items: >> >> - The function prototype. >> - A description of the function and of its arguments (except for a >> couple of cases, when there are no arguments and the return value >> makes the function usage really obvious). >> - A description of return values (if not void). >> - A listing of eBPF program types (if relevant, map types) compatible >> with the helper. >> - Information about the helper being restricted to GPL programs, or not. >> - The kernel version in which the helper was introduced. >> - The commit that introduced the helper (this is mostly to have it in >> the source of the man page, as it can be used to track changes and >> update the page). >> >> For several helpers, descriptions are inspired (at times, nearly copied) >> from the commit logs introducing them in the kernel--Many thanks to >> their respective authors! They were completed as much as possible, the >> objective being to have something easily accessible even for people just >> starting with eBPF. There is probably a bit more work to do in this >> direction for some helpers. >> >> Some RST formatting is used in the descriptions (not in function >> prototypes, to keep them readable, but the Python script provided in >> order to generate the RST for the manual page does add formatting to >> prototypes, to produce something pretty) to get "bold" and "italics" in >> manual pages. Hopefully, the descriptions in bpf.h file remains >> perfectly readable. Note that the few trailing white spaces are >> intentional, removing them would break paragraphs for rst2man. >> >> The descriptions should ideally be updated each time someone adds a new >> helper, or updates the behaviour (compatibility extended to new program >> types, new socket option supported...) or the interface (new flags >> available, ...) of existing ones. >> >> [1] I have not contacted people from the man-pages project prior to >> sending this RFC, so I can offer no guaranty at this time that they >> would accept to take the generated man page. >> >> Cc: linux-...@vger.kernel.org >> Cc: linux-...@vger.kernel.org >> Signed-off-by: Quentin Monnet > > Great work, thanks a lot for doing this! > > [...] >> + * int bpf_probe_read(void *dst, u32 size, const void *src) >> + * Description >> + * For tracing programs, safely attempt to read *size* bytes from >> + * address *src* and store the data in *dst*. >> + * Return >> + * 0 on success, or a negative error in case of failure. >> + * For >> + * *Tracing programs*. >> + * GPL only >> + * Yes >> + * Since >> + * Linux 4.1 >> + * >> + * .. commit 2541517c32be >> * >> * u64 bpf_ktime_get_ns(void) >> - * Return: current ktime >> - * >> - * int bpf_trace_printk(const char *fmt, int fmt_size, ...) >> - * Return: length of buffer written or negative error >> + * Description >> + * Return the time elapsed since system boot, in nanoseconds. >> + * Return >> + * Current *ktime*. >> + * For >> + * All program types, except >> + * **BPF_PROG_TYPE_CGROUP_DEVICE**. > > I think we should probably always just list the actual program types instead, > since when new types get added to the kernel not supporting bpf_ktime_get_ns() > helper in this example, the above statement would be misleading (potentially > more misleading than the other way aro
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 12:52 UTC+0200 ~ Markus Heiser > >> Am 09.04.2018 um 12:08 schrieb Daniel Borkmann : > [...] > >>> May I completely misunderstood you, so correct my if I'am wrong: >>> >>> - ./scripts/bpf_helpers_doc.py : produces reST markup from C-comments >>> - ./scripts/kerne-doc : produces reST markup from C-comments >>> >>> IMO: both are doing the same job, so why not using kernel-doc? >> >> They are not really doing the same job, in bpf_helpers_doc.py case you don't >> want the whole header rendered, but just a fraction of it, that is, the >> single big comment which describes all BPF helper functions that a BPF >> program developer has available to use in user space - aka the entries in >> the __BPF_FUNC_MAPPER() macro; > > >> I also doubt the latter would actually qualify >> in kdoc context as some sort of a function description. > > latter .. ah, OK .. thanks for clarifying. > > -- Markus -- As Daniel explained, kernel-doc does not apply in this case, we do not have the full function prototype for eBPF helpers in the header file. But to be honest, I didn't even realise kernel-doc was available and could do something close to what I was looking for, so thanks for your feedback! :) Quentin
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
On Mon, Apr 09, 2018 at 02:25:26PM +0100, Quentin Monnet wrote: > > Anyway, I am fine with keeping just signatures, descriptions and return > values for now. I will submit a new version with only those items. Thank you. Could you also split it into few patches? include/uapi/linux/bpf.h | 2237 scripts/bpf_helpers_doc.py | 568 +++ 2 files changed, 2429 insertions(+), 376 deletions(-) replying back and forth on a single patch of such size will be tedious for others to follow. May be document ~10 helpers at a time ? Total of ~7 patches and extra patch for .py ?
Re: [RFC bpf-next] bpf: document eBPF helpers and add a script to generate man page
2018-04-09 18:47 UTC-0700 ~ Alexei Starovoitov > On Mon, Apr 09, 2018 at 02:25:26PM +0100, Quentin Monnet wrote: >> >> Anyway, I am fine with keeping just signatures, descriptions and return >> values for now. I will submit a new version with only those items. > > Thank you. > > Could you also split it into few patches? > include/uapi/linux/bpf.h | 2237 > > scripts/bpf_helpers_doc.py | 568 +++ > 2 files changed, 2429 insertions(+), 376 deletions(-) > > replying back and forth on a single patch of such size will be tedious > for others to follow. > May be document ~10 helpers at a time ? Total of ~7 patches and extra > patch for .py ? > Sure, I'll do that. And I'll try to group helpers in a patch by author, it should also help for reviewing the descriptions. Quentin
