Re: [PATCH v4 03/10] livepatch: Add klp-convert tool

2019-08-12 Thread Masahiro Yamada
On Sat, Aug 10, 2019 at 3:42 AM Joe Lawrence  wrote:

> > > > diff --git a/scripts/livepatch/Makefile b/scripts/livepatch/Makefile
> > > > new file mode 100644
> > > > index ..2842ecdba3fd
> > > > --- /dev/null
> > > > +++ b/scripts/livepatch/Makefile
> > > > @@ -0,0 +1,7 @@
> > > > +hostprogs-y:= klp-convert
> > > > +always := $(hostprogs-y)
> > > > +
> > > > +klp-convert-objs   := klp-convert.o elf.o
> > > > +
> > > > +HOST_EXTRACFLAGS   := -g -I$(INSTALL_HDR_PATH)/include 
> > > > -Wall
> > >
> > > This looks strange.
> > >
> > > Theoretically, you cannot include headers in $(INSTALL_HDR_PATH)/include
> > > from host programs.
> > >
> > > headers_install works for the target architecture, not host architecture.
> > > This may cause a strange result when you are cross-compiling the kernel.
> > >
> > > BTW, which header in $(INSTALL_HDR_PATH)/include do you need to include ?
> > >
> > >
> > > Also, -Wall is redundant because it is set by the top-level Makefile.
> >
> >
> > I deleted HOST_EXTRACFLAGS entirely,
> > and I was still able to build klp-convert.
> >
> >
> > What is the purpose of '-g' ?
> > If it is only needed for local debugging,
> > it should be removed from the upstream code, in my opinion.
> >
>
> HOST_EXTRACFLAGS looks like it was present in the patchset from the
> early RFC days and inherited through each revision.
>
> These are the files that the klp-convert code includes, mostly typical C
> usercode headers like stdio.h and a few local headers like elf.h:
>
>   % grep -h '^#include' scripts/livepatch/*.{c,h} | sort -u
>   #include "elf.h"
>   #include 
>   #include 
>   #include "klp-convert.h"
>   #include "list.h"
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>
> If HOST_EXTRACFLAGS is really unneeded, we can easily drop it in the
> next patchset version.

Yes, please do so.

Thanks.



-- 
Best Regards
Masahiro Yamada


Re: [PATCH v4 03/10] livepatch: Add klp-convert tool

2019-08-09 Thread Joe Lawrence
On Wed, Jul 31, 2019 at 12:36:05PM +0900, Masahiro Yamada wrote:
> On Wed, Jul 31, 2019 at 11:50 AM Masahiro Yamada
>  wrote:
> >
> > On Thu, May 9, 2019 at 11:39 PM Joe Lawrence  
> > wrote:
> > >
> > > From: Josh Poimboeuf 
> > >
> > > Livepatches may use symbols which are not contained in its own scope,
> > > and, because of that, may end up compiled with relocations that will
> > > only be resolved during module load. Yet, when the referenced symbols
> > > are not exported, solving this relocation requires information on the
> > > object that holds the symbol (either vmlinux or modules) and its
> > > position inside the object, as an object may contain multiple symbols
> > > with the same name. Providing such information must be done
> > > accordingly to what is specified in
> > > Documentation/livepatch/module-elf-format.txt.
> > >
> > > Currently, there is no trivial way to embed the required information
> > > as requested in the final livepatch elf object. klp-convert solves
> > > this problem in two different forms: (i) by relying on Symbols.list,
> > > which is built during kernel compilation, to automatically infer the
> > > relocation targeted symbol, and, when such inference is not possible
> > > (ii) by using annotations in the elf object to convert the relocation
> > > accordingly to the specification, enabling it to be handled by the
> > > livepatch loader.
> > >
> > > Given the above, create scripts/livepatch to hold tools developed for
> > > livepatches and add source files for klp-convert there.
> > >
> > > The core file of klp-convert is scripts/livepatch/klp-convert.c, which
> > > implements the heuristics used to solve the relocations and the
> > > conversion of unresolved symbols into the expected format, as defined
> > > in [1].
> > >
> > > klp-convert receives as arguments the Symbols.list file, an input
> > > livepatch module to be converted and the output name for the converted
> > > livepatch. When it starts running, klp-convert parses Symbols.list and
> > > builds two internal lists of symbols, one containing the exported and
> > > another containing the non-exported symbols. Then, by parsing the rela
> > > sections in the elf object, klp-convert identifies which symbols must
> > > be converted, which are those unresolved and that do not have a
> > > corresponding exported symbol, and attempts to convert them
> > > accordingly to the specification.
> > >
> > > By using Symbols.list, klp-convert identifies which symbols have names
> > > that only appear in a single kernel object, thus being capable of
> > > resolving these cases without the intervention of the developer. When
> > > various homonymous symbols exist through kernel objects, it is not
> > > possible to infer the right one, thus klp-convert falls back into
> > > using developer annotations. If these were not provided, then the tool
> > > will print a list with all acceptable targets for the symbol being
> > > processed.
> > >
> > > Annotations in the context of klp-convert are accessible as struct
> > > klp_module_reloc entries in sections named
> > > .klp.module_relocs.. These entries are pairs of symbol
> > > references and positions which are to be resolved against definitions
> > > in .
> > >
> > > Define the structure klp_module_reloc in
> > > include/linux/uapi/livepatch.h allowing developers to annotate the
> > > livepatch source code with it.
> > >
> > > klp-convert relies on libelf and on a list implementation. Add files
> > > scripts/livepatch/elf.c and scripts/livepatch/elf.h, which are a
> > > libelf interfacing layer and scripts/livepatch/list.h, which is a
> > > list implementation.
> > >
> > > Update Makefiles to correctly support the compilation of the new tool,
> > > update MAINTAINERS file and add a .gitignore file.
> > >
> > > [1] - Documentation/livepatch/module-elf-format.txt
> > >
> > > Signed-off-by: Josh Poimboeuf 
> > > Signed-off-by: Konstantin Khlebnikov 
> > > Signed-off-by: Joao Moreira 
> > > Signed-off-by: Joe Lawrence 
> > > ---
> > >  MAINTAINERS |   1 +
> > >  include/uapi/linux/livepatch.h  |   5 +
> > >  scripts/Makefile|   1 +
> > >  scripts/livepatch/.gitignore|   1 +
> > >  scripts/livepatch/Makefile  |   7 +
> > >  scripts/livepatch/elf.c | 753 
> > >  scripts/livepatch/elf.h |  73 
> > >  scripts/livepatch/klp-convert.c | 713 ++
> > >  scripts/livepatch/klp-convert.h |  39 ++
> > >  scripts/livepatch/list.h| 391 +
> > >  10 files changed, 1984 insertions(+)
> > >  create mode 100644 scripts/livepatch/.gitignore
> > >  create mode 100644 scripts/livepatch/Makefile
> > >  create mode 100644 scripts/livepatch/elf.c
> > >  create mode 100644 scripts/livepatch/elf.h
> > >  create mode 100644 scripts/livepatch/klp-convert.c
> > >  create mode 100644 scripts/livepatch/klp-convert.h
> > >  create mode 100644 scripts/livepatch/list.h
> > >
>

Re: [PATCH v4 03/10] livepatch: Add klp-convert tool

2019-07-30 Thread Masahiro Yamada
On Wed, Jul 31, 2019 at 11:50 AM Masahiro Yamada
 wrote:
>
> On Thu, May 9, 2019 at 11:39 PM Joe Lawrence  wrote:
> >
> > From: Josh Poimboeuf 
> >
> > Livepatches may use symbols which are not contained in its own scope,
> > and, because of that, may end up compiled with relocations that will
> > only be resolved during module load. Yet, when the referenced symbols
> > are not exported, solving this relocation requires information on the
> > object that holds the symbol (either vmlinux or modules) and its
> > position inside the object, as an object may contain multiple symbols
> > with the same name. Providing such information must be done
> > accordingly to what is specified in
> > Documentation/livepatch/module-elf-format.txt.
> >
> > Currently, there is no trivial way to embed the required information
> > as requested in the final livepatch elf object. klp-convert solves
> > this problem in two different forms: (i) by relying on Symbols.list,
> > which is built during kernel compilation, to automatically infer the
> > relocation targeted symbol, and, when such inference is not possible
> > (ii) by using annotations in the elf object to convert the relocation
> > accordingly to the specification, enabling it to be handled by the
> > livepatch loader.
> >
> > Given the above, create scripts/livepatch to hold tools developed for
> > livepatches and add source files for klp-convert there.
> >
> > The core file of klp-convert is scripts/livepatch/klp-convert.c, which
> > implements the heuristics used to solve the relocations and the
> > conversion of unresolved symbols into the expected format, as defined
> > in [1].
> >
> > klp-convert receives as arguments the Symbols.list file, an input
> > livepatch module to be converted and the output name for the converted
> > livepatch. When it starts running, klp-convert parses Symbols.list and
> > builds two internal lists of symbols, one containing the exported and
> > another containing the non-exported symbols. Then, by parsing the rela
> > sections in the elf object, klp-convert identifies which symbols must
> > be converted, which are those unresolved and that do not have a
> > corresponding exported symbol, and attempts to convert them
> > accordingly to the specification.
> >
> > By using Symbols.list, klp-convert identifies which symbols have names
> > that only appear in a single kernel object, thus being capable of
> > resolving these cases without the intervention of the developer. When
> > various homonymous symbols exist through kernel objects, it is not
> > possible to infer the right one, thus klp-convert falls back into
> > using developer annotations. If these were not provided, then the tool
> > will print a list with all acceptable targets for the symbol being
> > processed.
> >
> > Annotations in the context of klp-convert are accessible as struct
> > klp_module_reloc entries in sections named
> > .klp.module_relocs.. These entries are pairs of symbol
> > references and positions which are to be resolved against definitions
> > in .
> >
> > Define the structure klp_module_reloc in
> > include/linux/uapi/livepatch.h allowing developers to annotate the
> > livepatch source code with it.
> >
> > klp-convert relies on libelf and on a list implementation. Add files
> > scripts/livepatch/elf.c and scripts/livepatch/elf.h, which are a
> > libelf interfacing layer and scripts/livepatch/list.h, which is a
> > list implementation.
> >
> > Update Makefiles to correctly support the compilation of the new tool,
> > update MAINTAINERS file and add a .gitignore file.
> >
> > [1] - Documentation/livepatch/module-elf-format.txt
> >
> > Signed-off-by: Josh Poimboeuf 
> > Signed-off-by: Konstantin Khlebnikov 
> > Signed-off-by: Joao Moreira 
> > Signed-off-by: Joe Lawrence 
> > ---
> >  MAINTAINERS |   1 +
> >  include/uapi/linux/livepatch.h  |   5 +
> >  scripts/Makefile|   1 +
> >  scripts/livepatch/.gitignore|   1 +
> >  scripts/livepatch/Makefile  |   7 +
> >  scripts/livepatch/elf.c | 753 
> >  scripts/livepatch/elf.h |  73 
> >  scripts/livepatch/klp-convert.c | 713 ++
> >  scripts/livepatch/klp-convert.h |  39 ++
> >  scripts/livepatch/list.h| 391 +
> >  10 files changed, 1984 insertions(+)
> >  create mode 100644 scripts/livepatch/.gitignore
> >  create mode 100644 scripts/livepatch/Makefile
> >  create mode 100644 scripts/livepatch/elf.c
> >  create mode 100644 scripts/livepatch/elf.h
> >  create mode 100644 scripts/livepatch/klp-convert.c
> >  create mode 100644 scripts/livepatch/klp-convert.h
> >  create mode 100644 scripts/livepatch/list.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 52842fa37261..c1587e1cc385 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -9022,6 +9022,7 @@ F:arch/x86/kernel/livepatch.c
> >  F: Documentation/livepatch/
> >  F: Documentation/ABI/tes

Re: [PATCH v4 03/10] livepatch: Add klp-convert tool

2019-07-30 Thread Masahiro Yamada
On Thu, May 9, 2019 at 11:39 PM Joe Lawrence  wrote:
>
> From: Josh Poimboeuf 
>
> Livepatches may use symbols which are not contained in its own scope,
> and, because of that, may end up compiled with relocations that will
> only be resolved during module load. Yet, when the referenced symbols
> are not exported, solving this relocation requires information on the
> object that holds the symbol (either vmlinux or modules) and its
> position inside the object, as an object may contain multiple symbols
> with the same name. Providing such information must be done
> accordingly to what is specified in
> Documentation/livepatch/module-elf-format.txt.
>
> Currently, there is no trivial way to embed the required information
> as requested in the final livepatch elf object. klp-convert solves
> this problem in two different forms: (i) by relying on Symbols.list,
> which is built during kernel compilation, to automatically infer the
> relocation targeted symbol, and, when such inference is not possible
> (ii) by using annotations in the elf object to convert the relocation
> accordingly to the specification, enabling it to be handled by the
> livepatch loader.
>
> Given the above, create scripts/livepatch to hold tools developed for
> livepatches and add source files for klp-convert there.
>
> The core file of klp-convert is scripts/livepatch/klp-convert.c, which
> implements the heuristics used to solve the relocations and the
> conversion of unresolved symbols into the expected format, as defined
> in [1].
>
> klp-convert receives as arguments the Symbols.list file, an input
> livepatch module to be converted and the output name for the converted
> livepatch. When it starts running, klp-convert parses Symbols.list and
> builds two internal lists of symbols, one containing the exported and
> another containing the non-exported symbols. Then, by parsing the rela
> sections in the elf object, klp-convert identifies which symbols must
> be converted, which are those unresolved and that do not have a
> corresponding exported symbol, and attempts to convert them
> accordingly to the specification.
>
> By using Symbols.list, klp-convert identifies which symbols have names
> that only appear in a single kernel object, thus being capable of
> resolving these cases without the intervention of the developer. When
> various homonymous symbols exist through kernel objects, it is not
> possible to infer the right one, thus klp-convert falls back into
> using developer annotations. If these were not provided, then the tool
> will print a list with all acceptable targets for the symbol being
> processed.
>
> Annotations in the context of klp-convert are accessible as struct
> klp_module_reloc entries in sections named
> .klp.module_relocs.. These entries are pairs of symbol
> references and positions which are to be resolved against definitions
> in .
>
> Define the structure klp_module_reloc in
> include/linux/uapi/livepatch.h allowing developers to annotate the
> livepatch source code with it.
>
> klp-convert relies on libelf and on a list implementation. Add files
> scripts/livepatch/elf.c and scripts/livepatch/elf.h, which are a
> libelf interfacing layer and scripts/livepatch/list.h, which is a
> list implementation.
>
> Update Makefiles to correctly support the compilation of the new tool,
> update MAINTAINERS file and add a .gitignore file.
>
> [1] - Documentation/livepatch/module-elf-format.txt
>
> Signed-off-by: Josh Poimboeuf 
> Signed-off-by: Konstantin Khlebnikov 
> Signed-off-by: Joao Moreira 
> Signed-off-by: Joe Lawrence 
> ---
>  MAINTAINERS |   1 +
>  include/uapi/linux/livepatch.h  |   5 +
>  scripts/Makefile|   1 +
>  scripts/livepatch/.gitignore|   1 +
>  scripts/livepatch/Makefile  |   7 +
>  scripts/livepatch/elf.c | 753 
>  scripts/livepatch/elf.h |  73 
>  scripts/livepatch/klp-convert.c | 713 ++
>  scripts/livepatch/klp-convert.h |  39 ++
>  scripts/livepatch/list.h| 391 +
>  10 files changed, 1984 insertions(+)
>  create mode 100644 scripts/livepatch/.gitignore
>  create mode 100644 scripts/livepatch/Makefile
>  create mode 100644 scripts/livepatch/elf.c
>  create mode 100644 scripts/livepatch/elf.h
>  create mode 100644 scripts/livepatch/klp-convert.c
>  create mode 100644 scripts/livepatch/klp-convert.h
>  create mode 100644 scripts/livepatch/list.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 52842fa37261..c1587e1cc385 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9022,6 +9022,7 @@ F:arch/x86/kernel/livepatch.c
>  F: Documentation/livepatch/
>  F: Documentation/ABI/testing/sysfs-kernel-livepatch
>  F: samples/livepatch/
> +F: scripts/livepatch/
>  F: tools/testing/selftests/livepatch/
>  L: live-patch...@vger.kernel.org
>  T: git 
> git://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.g

[PATCH v4 03/10] livepatch: Add klp-convert tool

2019-05-09 Thread Joe Lawrence
From: Josh Poimboeuf 

Livepatches may use symbols which are not contained in its own scope,
and, because of that, may end up compiled with relocations that will
only be resolved during module load. Yet, when the referenced symbols
are not exported, solving this relocation requires information on the
object that holds the symbol (either vmlinux or modules) and its
position inside the object, as an object may contain multiple symbols
with the same name. Providing such information must be done
accordingly to what is specified in
Documentation/livepatch/module-elf-format.txt.

Currently, there is no trivial way to embed the required information
as requested in the final livepatch elf object. klp-convert solves
this problem in two different forms: (i) by relying on Symbols.list,
which is built during kernel compilation, to automatically infer the
relocation targeted symbol, and, when such inference is not possible
(ii) by using annotations in the elf object to convert the relocation
accordingly to the specification, enabling it to be handled by the
livepatch loader.

Given the above, create scripts/livepatch to hold tools developed for
livepatches and add source files for klp-convert there.

The core file of klp-convert is scripts/livepatch/klp-convert.c, which
implements the heuristics used to solve the relocations and the
conversion of unresolved symbols into the expected format, as defined
in [1].

klp-convert receives as arguments the Symbols.list file, an input
livepatch module to be converted and the output name for the converted
livepatch. When it starts running, klp-convert parses Symbols.list and
builds two internal lists of symbols, one containing the exported and
another containing the non-exported symbols. Then, by parsing the rela
sections in the elf object, klp-convert identifies which symbols must
be converted, which are those unresolved and that do not have a
corresponding exported symbol, and attempts to convert them
accordingly to the specification.

By using Symbols.list, klp-convert identifies which symbols have names
that only appear in a single kernel object, thus being capable of
resolving these cases without the intervention of the developer. When
various homonymous symbols exist through kernel objects, it is not
possible to infer the right one, thus klp-convert falls back into
using developer annotations. If these were not provided, then the tool
will print a list with all acceptable targets for the symbol being
processed.

Annotations in the context of klp-convert are accessible as struct
klp_module_reloc entries in sections named
.klp.module_relocs.. These entries are pairs of symbol
references and positions which are to be resolved against definitions
in .

Define the structure klp_module_reloc in
include/linux/uapi/livepatch.h allowing developers to annotate the
livepatch source code with it.

klp-convert relies on libelf and on a list implementation. Add files
scripts/livepatch/elf.c and scripts/livepatch/elf.h, which are a
libelf interfacing layer and scripts/livepatch/list.h, which is a
list implementation.

Update Makefiles to correctly support the compilation of the new tool,
update MAINTAINERS file and add a .gitignore file.

[1] - Documentation/livepatch/module-elf-format.txt

Signed-off-by: Josh Poimboeuf 
Signed-off-by: Konstantin Khlebnikov 
Signed-off-by: Joao Moreira 
Signed-off-by: Joe Lawrence 
---
 MAINTAINERS |   1 +
 include/uapi/linux/livepatch.h  |   5 +
 scripts/Makefile|   1 +
 scripts/livepatch/.gitignore|   1 +
 scripts/livepatch/Makefile  |   7 +
 scripts/livepatch/elf.c | 753 
 scripts/livepatch/elf.h |  73 
 scripts/livepatch/klp-convert.c | 713 ++
 scripts/livepatch/klp-convert.h |  39 ++
 scripts/livepatch/list.h| 391 +
 10 files changed, 1984 insertions(+)
 create mode 100644 scripts/livepatch/.gitignore
 create mode 100644 scripts/livepatch/Makefile
 create mode 100644 scripts/livepatch/elf.c
 create mode 100644 scripts/livepatch/elf.h
 create mode 100644 scripts/livepatch/klp-convert.c
 create mode 100644 scripts/livepatch/klp-convert.h
 create mode 100644 scripts/livepatch/list.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 52842fa37261..c1587e1cc385 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9022,6 +9022,7 @@ F:arch/x86/kernel/livepatch.c
 F: Documentation/livepatch/
 F: Documentation/ABI/testing/sysfs-kernel-livepatch
 F: samples/livepatch/
+F: scripts/livepatch/
 F: tools/testing/selftests/livepatch/
 L: live-patch...@vger.kernel.org
 T: git 
git://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git
diff --git a/include/uapi/linux/livepatch.h b/include/uapi/linux/livepatch.h
index e19430918a07..1c364d42d38e 100644
--- a/include/uapi/linux/livepatch.h
+++ b/include/uapi/linux/livepatch.h
@@ -12,4 +12,9 @@
 #define KLP_RELA_PREFIX