Re: [PATCH] Documentation: kbuild: Add document about reproducible builds
On Fri, 2019-09-13 at 08:28 +0200, Nicolas Schier wrote: > Hi Ben, > > thanks for that document, I really enjoyed reading it! > > On Wed, Sep 11, 2019 at 12:53:53PM +0100, Ben Hutchings wrote: > [...] > > diff --git a/Documentation/kbuild/reproducible-builds.rst > > b/Documentation/kbuild/reproducible-builds.rst > > new file mode 100644 > > index ..4d988faf93b8 > > --- /dev/null > > +++ b/Documentation/kbuild/reproducible-builds.rst > > @@ -0,0 +1,115 @@ > > +=== > > +Reproducible builds > > +=== > > + > > +It is generally desirable that the building the same source code with > > In this sentence, I think there is either one word to much (the first > 'the') or some word is missing (e.h. 'of'). Yes, thanks. Ben. -- Ben Hutchings If you seem to know what you are doing, you'll be given more to do. signature.asc Description: This is a digitally signed message part
[PATCH v2] Documentation: kbuild: Add document about reproducible builds
In the Distribution Kernels track at Linux Plumbers Conference there was some discussion around the difficulty of making kernel builds reproducible. This is a solved problem, but the solutions don't appear to be documented in one place. This document lists the issues I know about and the settings needed to ensure reproducibility. Signed-off-by: Ben Hutchings --- v2: * Delete extra "the" in the first paragraph. * Rewrite paragraph about filenames to reflect that -fmacro-prefix-map is used by default. * Rewrite paragraph about source packages to be clearer. Documentation/kbuild/index.rst | 1 + Documentation/kbuild/reproducible-builds.rst | 122 +++ 2 files changed, 123 insertions(+) create mode 100644 Documentation/kbuild/reproducible-builds.rst diff --git a/Documentation/kbuild/index.rst b/Documentation/kbuild/index.rst index e323a3f2cc81..0f144fad99a6 100644 --- a/Documentation/kbuild/index.rst +++ b/Documentation/kbuild/index.rst @@ -18,6 +18,7 @@ Kernel Build System headers_install issues +reproducible-builds .. only:: subproject and html diff --git a/Documentation/kbuild/reproducible-builds.rst b/Documentation/kbuild/reproducible-builds.rst new file mode 100644 index ..ab92e98c89c8 --- /dev/null +++ b/Documentation/kbuild/reproducible-builds.rst @@ -0,0 +1,122 @@ +=== +Reproducible builds +=== + +It is generally desirable that building the same source code with +the same set of tools is reproducible, i.e. the output is always +exactly the same. This makes it possible to verify that the build +infrastructure for a binary distribution or embedded system has not +been subverted. This can also make it easier to verify that a source +or tool change does not make any difference to the resulting binaries. + +The `Reproducible Builds project`_ has more information about this +general topic. This document covers the various reasons why building +the kernel may be unreproducible, and how to avoid them. + +Timestamps +-- + +The kernel embeds a timestamp in two places: + +* The version string exposed by ``uname()`` and included in + ``/proc/version`` + +* File timestamps in the embedded initramfs + +By default the timestamp is the current time. This must be overridden +using the `KBUILD_BUILD_TIMESTAMP`_ variable. If you are building +from a git commit, you could use its commit date. + +The kernel does *not* use the ``__DATE__`` and ``__TIME__`` macros, +and enables warnings if they are used. If you incorporate external +code that does use these, you must override the timestamp they +correspond to by setting the `SOURCE_DATE_EPOCH`_ environment +variable. + +User, host +-- + +The kernel embeds the building user and host names in +``/proc/version``. These must be overridden using the +`KBUILD_BUILD_USER and KBUILD_BUILD_HOST`_ variables. If you are +building from a git commit, you could use its committer address. + +Absolute filenames +-- + +When the kernel is built out-of-tree, debug information may include +absolute filenames for the source files. This must be overridden by +including the ``-fdebug-prefix-map`` option in the `KCFLAGS`_ variable. + +Depending on the compiler used, the ``__FILE__`` macro may also expand +to an absolute filename in an out-of-tree build. Kbuild automatically +uses the ``-fmacro-prefix-map`` option to prevent this, if it is +supported. + +The Reproducible Builds web site has more information about these +`prefix-map options`_. + +Generated files in source packages +-- + +The build processes for some programs under the ``tools/`` +subdirectory do not completely support out-of-tree builds. This may +cause a later source package build using e.g. ``make rpm-pkg`` to +include generated files. You should ensure the source tree is +pristine by running ``make mrproper`` or ``git clean -d -f -x`` before +building a source package. + +Module signing +-- + +If you enable ``CONFIG_MODULE_SIG_ALL``, the default behaviour is to +generate a different temporary key for each build, resulting in the +modules being unreproducible. However, including a signing key with +your source would presumably defeat the purpose of signing modules. + +One approach to this is to divide up the build process so that the +unreproducible parts can be treated as sources: + +1. Generate a persistent signing key. Add the certificate for the key + to the kernel source. + +2. Set the ``CONFIG_SYSTEM_TRUSTED_KEYS`` symbol to include the + signing key's certificate, set ``CONFIG_MODULE_SIG_KEY`` to an + empty string, and disable ``CONFIG_MODULE_SIG_ALL``. + Build the kernel and modules. + +3. Create detached signatures for the modules, and publish them as + sources. + +4. Perform a second build that attaches the module signatures. It + can either rebuild the modules or use the output of
Re: [PATCH] Documentation: kbuild: Add document about reproducible builds
On Wed, 2019-09-11 at 14:04 +0100, Ben Hutchings wrote: > On Wed, 2019-09-11 at 21:17 +0900, Masahiro Yamada wrote: > > Hi Ben, > > > > > > Thanks for this. > > Please let me add some comments. > > > > > > On Wed, Sep 11, 2019 at 8:54 PM Ben Hutchings wrote: > [...] > > > +Absolute filenames > > > +-- > > > + > > > +When the kernel is built out-of-tree, debug information may include > > > +absolute filenames for the source files. The ``__FILE__`` macro may > > > +also expand to an absolute filename. This must be overridden by > > > +including `prefix-map options`_ in the `KCFLAGS`_ variable. > > > > Do you mean -fmacro-prefix-map ? > > No, I mean -ffile-prefix-map or the older -fdebug-prefix-map. > > > If so, it is already taken care of by the top Makefile. > > If you use GCC 8 or newer, it is automatically added to > > KBUILD_CFLAGS. > > Ah, that's helpful. So, I suppose I should just mention > -fdebug-prefix-map here and warn that __FILE__ will still be a proble > if using older compiler versions. My revised text for this section is: --- When the kernel is built out-of-tree, debug information may include absolute filenames for the source files. This must be overridden by including the ``-fdebug-prefix-map`` option in the `KCFLAGS`_ variable. Depending on the compiler used, the ``__FILE__`` macro may also expand to an absolute filename in an out-of-tree build. Kbuild automatically uses the ``-fmacro-prefix-map`` option to prevent this, if it is supported. The Reproducible Builds web site has more information about these `prefix-map options`_. --- Does that look OK to you? > > > +Generated files in source packages > > > +-- > > > + > > > +The build processes for some programs under the ``tools/`` > > > +subdirectory do not completely support out-of-tree builds. This may > > > +cause source packages built using e.g. ``make rpm-pkg`` to include > > > +generated files and so be unreproducible. It may be necessary to > > > +clean the source tree completely (``make mrproper`` or > > > +``git clean -d -f -x``) before building a source package. > > > > Currently, the source package building does not support > > out-of-tree build anyway. > > Yes, I realise that. > > > 'make O=foo rpm-pkg' fails with an error message. > > > > Building in a pristine source will solve the issue. > [...] > > The issue I'm thinking about is that an out-of-tree build, prior to the > package build, *should* leave the source pristine and sometimes does > not. > > For Debian's official kernel packages, we build a binary package of the > upstream source, and at some times this has unexpectedly included some > generated files. I believe a similar issue would affect the upstream > package scripts. My revised text for this section is: --- The build processes for some programs under the ``tools/`` subdirectory do not completely support out-of-tree builds. This may cause a later source package build using e.g. ``make rpm-pkg`` to include generated files. You should ensure the source tree is pristine by running ``make mrproper`` or ``git clean -d -f -x`` before building a source package. --- Ben. -- Ben Hutchings The obvious mathematical breakthrough [to break modern encryption] would be development of an easy way to factor large prime numbers. - Bill Gates signature.asc Description: This is a digitally signed message part
Re: [PATCH] Documentation: kbuild: Add document about reproducible builds
On Wed, 2019-09-11 at 21:17 +0900, Masahiro Yamada wrote: > Hi Ben, > > > Thanks for this. > Please let me add some comments. > > > On Wed, Sep 11, 2019 at 8:54 PM Ben Hutchings wrote: [...] > > +Absolute filenames > > +-- > > + > > +When the kernel is built out-of-tree, debug information may include > > +absolute filenames for the source files. The ``__FILE__`` macro may > > +also expand to an absolute filename. This must be overridden by > > +including `prefix-map options`_ in the `KCFLAGS`_ variable. > > Do you mean -fmacro-prefix-map ? No, I mean -ffile-prefix-map or the older -fdebug-prefix-map. > If so, it is already taken care of by the top Makefile. > If you use GCC 8 or newer, it is automatically added to > KBUILD_CFLAGS. Ah, that's helpful. So, I suppose I should just mention -fdebug-prefix-map here and warn that __FILE__ will still be a proble if using older compiler versions. > > +Generated files in source packages > > +-- > > + > > +The build processes for some programs under the ``tools/`` > > +subdirectory do not completely support out-of-tree builds. This may > > +cause source packages built using e.g. ``make rpm-pkg`` to include > > +generated files and so be unreproducible. It may be necessary to > > +clean the source tree completely (``make mrproper`` or > > +``git clean -d -f -x``) before building a source package. > > Currently, the source package building does not support > out-of-tree build anyway. Yes, I realise that. > 'make O=foo rpm-pkg' fails with an error message. > > Building in a pristine source will solve the issue. [...] The issue I'm thinking about is that an out-of-tree build, prior to the package build, *should* leave the source pristine and sometimes does not. For Debian's official kernel packages, we build a binary package of the upstream source, and at some times this has unexpectedly included some generated files. I believe a similar issue would affect the upstream package scripts. Ben. -- Ben Hutchings The obvious mathematical breakthrough [to break modern encryption] would be development of an easy way to factor large prime numbers. - Bill Gates signature.asc Description: This is a digitally signed message part
[PATCH] Documentation: kbuild: Add document about reproducible builds
In the Distribution Kernels track at Linux Plumbers Conference there was some discussion around the difficulty of making kernel builds reproducible. This is a solved problem, but the solutions don't appear to be documented in one place. This document lists the issues I know about and the settings needed to ensure reproducibility. Signed-off-by: Ben Hutchings --- Documentation/kbuild/index.rst | 1 + Documentation/kbuild/reproducible-builds.rst | 115 +++ 2 files changed, 116 insertions(+) create mode 100644 Documentation/kbuild/reproducible-builds.rst diff --git a/Documentation/kbuild/index.rst b/Documentation/kbuild/index.rst index e323a3f2cc81..0f144fad99a6 100644 --- a/Documentation/kbuild/index.rst +++ b/Documentation/kbuild/index.rst @@ -18,6 +18,7 @@ Kernel Build System headers_install issues +reproducible-builds .. only:: subproject and html diff --git a/Documentation/kbuild/reproducible-builds.rst b/Documentation/kbuild/reproducible-builds.rst new file mode 100644 index ..4d988faf93b8 --- /dev/null +++ b/Documentation/kbuild/reproducible-builds.rst @@ -0,0 +1,115 @@ +=== +Reproducible builds +=== + +It is generally desirable that the building the same source code with +the same set of tools is reproducible, i.e. the output is always +exactly the same. This makes it possible to verify that the build +infrastructure for a binary distribution or embedded system has not +been subverted. This can also make it easier to verify that a source +or tool change does not make any difference to the resulting binaries. + +The `Reproducible Builds project`_ has more information about this +general topic. This document covers the various reasons why building +the kernel may be unreproducible, and how to avoid them. + +Timestamps +-- + +The kernel embeds a timestamp in two places: + +* The version string exposed by ``uname()`` and included in + ``/proc/version`` + +* File timestamps in the embedded initramfs + +By default the timestamp is the current time. This must be overridden +using the `KBUILD_BUILD_TIMESTAMP`_ variable. If you are building +from a git commit, you could use its commit date. + +The kernel does *not* use the ``__DATE__`` and ``__TIME__`` macros, +and enables warnings if they are used. If you incorporate external +code that does use these, you must override the timestamp they +correspond to by setting the `SOURCE_DATE_EPOCH`_ environment +variable. + +User, host +-- + +The kernel embeds the building user and host names in +``/proc/version``. These must be overridden using the +`KBUILD_BUILD_USER and KBUILD_BUILD_HOST`_ variables. If you are +building from a git commit, you could use its committer address. + +Absolute filenames +-- + +When the kernel is built out-of-tree, debug information may include +absolute filenames for the source files. The ``__FILE__`` macro may +also expand to an absolute filename. This must be overridden by +including `prefix-map options`_ in the `KCFLAGS`_ variable. + +Generated files in source packages +-- + +The build processes for some programs under the ``tools/`` +subdirectory do not completely support out-of-tree builds. This may +cause source packages built using e.g. ``make rpm-pkg`` to include +generated files and so be unreproducible. It may be necessary to +clean the source tree completely (``make mrproper`` or +``git clean -d -f -x``) before building a source package. + +Module signing +-- + +If you enable ``CONFIG_MODULE_SIG_ALL``, the default behaviour is to +generate a different temporary key for each build, resulting in the +modules being unreproducible. However, including a signing key with +your source would presumably defeat the purpose of signing modules. + +One approach to this is to divide up the build process so that the +unreproducible parts can be treated as sources: + +1. Generate a persistent signing key. Add the certificate for the key + to the kernel source. + +2. Set the ``CONFIG_SYSTEM_TRUSTED_KEYS`` symbol to include the + signing key's certificate, set ``CONFIG_MODULE_SIG_KEY`` to an + empty string, and disable ``CONFIG_MODULE_SIG_ALL``. + Build the kernel and modules. + +3. Create detached signatures for the modules, and publish them as + sources. + +4. Perform a second build that attaches the module signatures. It + can either rebuild the modules or use the output of step 2. + +Structure randomisation +--- + +If you enable ``CONFIG_GCC_PLUGIN_RANDSTRUCT``, you will need to +pre-generate the random seed in +``scripts/gcc-plgins/randomize_layout_seed.h`` so the same value +is used in rebuilds. + +Debug info conflicts + + +This is not a problem of unreproducibility, but of generated files +being *too* reproducible. + +Once you set all the necessary variables for a reproducible build,
[PATCH 3.16 004/305] x86/mm: Simplify p[g4um]d_page() macros
3.16.63-rc1 review patch. If anyone has any objections, please let me know.
--
From: Tom Lendacky
commit fd7e315988b784509ba3f1b42f539bd0b1fca9bb upstream.
Create a pgd_pfn() macro similar to the p[4um]d_pfn() macros and then
use the p[g4um]d_pfn() macros in the p[g4um]d_page() macros instead of
duplicating the code.
Signed-off-by: Tom Lendacky
Reviewed-by: Thomas Gleixner
Reviewed-by: Borislav Petkov
Cc: Alexander Potapenko
Cc: Andrey Ryabinin
Cc: Andy Lutomirski
Cc: Arnd Bergmann
Cc: Borislav Petkov
Cc: Brijesh Singh
Cc: Dave Young
Cc: Dmitry Vyukov
Cc: Jonathan Corbet
Cc: Konrad Rzeszutek Wilk
Cc: Larry Woodman
Cc: Linus Torvalds
Cc: Matt Fleming
Cc: Michael S. Tsirkin
Cc: Paolo Bonzini
Cc: Peter Zijlstra
Cc: Radim Krčmář
Cc: Rik van Riel
Cc: Toshimitsu Kani
Cc: kasan-...@googlegroups.com
Cc: k...@vger.kernel.org
Cc: linux-a...@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-...@vger.kernel.org
Cc: linux...@kvack.org
Link:
http://lkml.kernel.org/r/e61eb533a6d0aac941db2723d8aa63ef6b882dee.1500319216.git.thomas.lenda...@amd.com
Signed-off-by: Ingo Molnar
[Backported to 4.9 stable by AK, suggested by Michael Hocko]
Signed-off-by: Andi Kleen
Signed-off-by: Greg Kroah-Hartman
Signed-off-by: Wenkuan Wang
Signed-off-by: Ben Hutchings
---
arch/x86/include/asm/pgtable.h | 13 -
1 file changed, 8 insertions(+), 5 deletions(-)
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -166,6 +166,11 @@ static inline unsigned long pud_pfn(pud_
return (pfn & pud_pfn_mask(pud)) >> PAGE_SHIFT;
}
+static inline unsigned long pgd_pfn(pgd_t pgd)
+{
+ return (pgd_val(pgd) & PTE_PFN_MASK) >> PAGE_SHIFT;
+}
+
#define pte_page(pte) pfn_to_page(pte_pfn(pte))
static inline int pmd_large(pmd_t pte)
@@ -591,8 +596,7 @@ static inline unsigned long pmd_page_vad
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pmd_page(pmd) \
- pfn_to_page((pmd_val(pmd) & pmd_pfn_mask(pmd)) >> PAGE_SHIFT)
+#define pmd_page(pmd) pfn_to_page(pmd_pfn(pmd))
/*
* the pmd page can be thought of an array like this: pmd_t[PTRS_PER_PMD]
@@ -665,8 +669,7 @@ static inline unsigned long pud_page_vad
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pud_page(pud) \
- pfn_to_page((pud_val(pud) & pud_pfn_mask(pud)) >> PAGE_SHIFT)
+#define pud_page(pud) pfn_to_page(pud_pfn(pud))
/* Find an entry in the second-level page table.. */
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address)
@@ -706,7 +709,7 @@ static inline unsigned long pgd_page_vad
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pgd_page(pgd) pfn_to_page(pgd_val(pgd) >> PAGE_SHIFT)
+#define pgd_page(pgd) pfn_to_page(pgd_pfn(pgd))
/* to find an entry in a page-table-directory. */
static inline unsigned long pud_index(unsigned long address)
Re: Bug#905116: [PATCH v2] scripts/kernel-doc: Escape all literal braces in regexes
On Mon, 2018-08-06 at 07:14 -0600, Jonathan Corbet wrote:
> On Sun, 5 Aug 2018 17:41:09 +0100
> Ben Hutchings wrote:
>
> > Commit 720ac2ef479d ("PATCH scripts/kernel-doc") fixed the two
> > instances of literal braces that Perl 5.28 warns about, but there are
> > still more than it doesn't warn about.
>
> So where can I find this commit of which you speak? I can't find it in
> mainline, -next, or -stable...
>
> The patch looks good, I'd like to get it in this coming merge window if
> possible.
Sorry, that was the wrong commit hash. I mean commit 701b3a3c0ac4.
Ben.
--
Ben Hutchings
Larkinson's Law: All laws are basically false.
signature.asc
Description: This is a digitally signed message part
[PATCH v2] scripts/kernel-doc: Escape all literal braces in regexes
Commit 720ac2ef479d ("PATCH scripts/kernel-doc") fixed the two
instances of literal braces that Perl 5.28 warns about, but there are
still more than it doesn't warn about.
Escape all left braces that are treated as literal characters. Also
escape literal right braces, for consistency and to avoid confusing
bracket-matching in text editors.
Signed-off-by: Ben Hutchings
---
v2: Rebase on top of commit 720ac2ef479d; reword accordingly
scripts/kernel-doc | 20 ++--
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 31a34ced55a3..8f0f508a78e9 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1062,7 +1062,7 @@ sub dump_struct($$) {
my $x = shift;
my $file = shift;
-if ($x =~ /(struct|union)\s+(\w+)\s*{(.*)}/) {
+if ($x =~ /(struct|union)\s+(\w+)\s*\{(.*)\}/) {
my $decl_type = $1;
$declaration_name = $2;
my $members = $3;
@@ -1148,20 +1148,20 @@ sub dump_struct($$) {
}
}
}
- $members =~
s/(struct|union)([^\{\};]+)\{([^\{\}]*)}([^\{\}\;]*)\;/$newmember/;
+ $members =~
s/(struct|union)([^\{\};]+)\{([^\{\}]*)\}([^\{\}\;]*)\;/$newmember/;
}
# Ignore other nested elements, like enums
- $members =~ s/(\{[^\{\}]*})//g;
+ $members =~ s/(\{[^\{\}]*\})//g;
create_parameterlist($members, ';', $file, $declaration_name);
check_sections($file, $declaration_name, $decl_type, $sectcheck,
$struct_actual);
# Adjust declaration for better display
- $declaration =~ s/([{;])/$1\n/g;
- $declaration =~ s/}\s+;/};/g;
+ $declaration =~ s/([\{;])/$1\n/g;
+ $declaration =~ s/\}\s+;/};/g;
# Better handle inlined enums
- do {} while ($declaration =~ s/(enum\s+{[^}]+),([^\n])/$1,\n$2/);
+ do {} while ($declaration =~ s/(enum\s+\{[^\}]+),([^\n])/$1,\n$2/);
my @def_args = split /\n/, $declaration;
my $level = 1;
@@ -1171,12 +1171,12 @@ sub dump_struct($$) {
$clause =~ s/\s+$//;
$clause =~ s/\s+/ /;
next if (!$clause);
- $level-- if ($clause =~ m/(})/ && $level > 1);
+ $level-- if ($clause =~ m/(\})/ && $level > 1);
if (!($clause =~ m/^\s*#/)) {
$declaration .= "\t" x $level;
}
$declaration .= "\t" . $clause . "\n";
- $level++ if ($clause =~ m/(\{)/ && !($clause =~m/}/));
+ $level++ if ($clause =~ m/(\{)/ && !($clause =~m/\}/));
}
output_declaration($declaration_name,
'struct',
@@ -1244,7 +1244,7 @@ sub dump_enum($$) {
# strip #define macros inside enums
$x =~ s@#\s*((define|ifdef)\s+|endif)[^;]*;@@gos;
-if ($x =~ /enum\s+(\w+)\s*{(.*)}/) {
+if ($x =~ /enum\s+(\w+)\s*\{(.*)\}/) {
$declaration_name = $1;
my $members = $2;
my %_members;
@@ -1785,7 +1785,7 @@ sub process_proto_type($$) {
}
while (1) {
- if ( $x =~ /([^{};]*)([{};])(.*)/ ) {
+ if ( $x =~ /([^\{\};]*)([\{\};])(.*)/ ) {
if( length $prototype ) {
$prototype .= " "
}
signature.asc
Description: Digital signature
[PATCH] scripts/kernel-doc: Escape all literal braces in regexes
Braces are usually metacharacters in regexes, used to specify a
number of repetitions or as part of an escape sequence. If this
interpretation is not possible then Perl currently treats them
as literal characters.
Perl 5.28 has deprecated the literal interpretation of left braces,
and Perl 5.32 will remove it (resulting in a fatal error).
Escape all left braces that are treated as literal characters. Also
escape literal right braces, for consistency and to avoid confusing
bracket-matching in text editors.
References: https://bugs.debian.org/905116
Signed-off-by: Ben Hutchings
---
I tested that the output of "make xmldocs" is identical before and
after this change, whether using Perl 5.26 or 5.28.
Ben.
scripts/kernel-doc | 20 ++--
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 0057d8eafcc1..8f0f508a78e9 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1062,7 +1062,7 @@ sub dump_struct($$) {
my $x = shift;
my $file = shift;
-if ($x =~ /(struct|union)\s+(\w+)\s*{(.*)}/) {
+if ($x =~ /(struct|union)\s+(\w+)\s*\{(.*)\}/) {
my $decl_type = $1;
$declaration_name = $2;
my $members = $3;
@@ -1148,20 +1148,20 @@ sub dump_struct($$) {
}
}
}
- $members =~
s/(struct|union)([^\{\};]+)\{([^\{\}]*)}([^\{\}\;]*)\;/$newmember/;
+ $members =~
s/(struct|union)([^\{\};]+)\{([^\{\}]*)\}([^\{\}\;]*)\;/$newmember/;
}
# Ignore other nested elements, like enums
- $members =~ s/({[^\{\}]*})//g;
+ $members =~ s/(\{[^\{\}]*\})//g;
create_parameterlist($members, ';', $file, $declaration_name);
check_sections($file, $declaration_name, $decl_type, $sectcheck,
$struct_actual);
# Adjust declaration for better display
- $declaration =~ s/([{;])/$1\n/g;
- $declaration =~ s/}\s+;/};/g;
+ $declaration =~ s/([\{;])/$1\n/g;
+ $declaration =~ s/\}\s+;/};/g;
# Better handle inlined enums
- do {} while ($declaration =~ s/(enum\s+{[^}]+),([^\n])/$1,\n$2/);
+ do {} while ($declaration =~ s/(enum\s+\{[^\}]+),([^\n])/$1,\n$2/);
my @def_args = split /\n/, $declaration;
my $level = 1;
@@ -1171,12 +1171,12 @@ sub dump_struct($$) {
$clause =~ s/\s+$//;
$clause =~ s/\s+/ /;
next if (!$clause);
- $level-- if ($clause =~ m/(})/ && $level > 1);
+ $level-- if ($clause =~ m/(\})/ && $level > 1);
if (!($clause =~ m/^\s*#/)) {
$declaration .= "\t" x $level;
}
$declaration .= "\t" . $clause . "\n";
- $level++ if ($clause =~ m/({)/ && !($clause =~m/}/));
+ $level++ if ($clause =~ m/(\{)/ && !($clause =~m/\}/));
}
output_declaration($declaration_name,
'struct',
@@ -1244,7 +1244,7 @@ sub dump_enum($$) {
# strip #define macros inside enums
$x =~ s@#\s*((define|ifdef)\s+|endif)[^;]*;@@gos;
-if ($x =~ /enum\s+(\w+)\s*{(.*)}/) {
+if ($x =~ /enum\s+(\w+)\s*\{(.*)\}/) {
$declaration_name = $1;
my $members = $2;
my %_members;
@@ -1785,7 +1785,7 @@ sub process_proto_type($$) {
}
while (1) {
- if ( $x =~ /([^{};]*)([{};])(.*)/ ) {
+ if ( $x =~ /([^\{\};]*)([\{\};])(.*)/ ) {
if( length $prototype ) {
$prototype .= " "
}
signature.asc
Description: Digital signature
[PATCH] Documentation: Update references to drivers/base/firmware_class.c
This source file has been renamed in 4.17. Signed-off-by: Ben Hutchings --- Documentation/driver-api/firmware/request_firmware.rst | 10 +- Documentation/driver-api/infrastructure.rst| 2 +- Documentation/power/suspend-and-cpuhotplug.txt | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/driver-api/firmware/request_firmware.rst b/Documentation/driver-api/firmware/request_firmware.rst index cf4516dfbf96..20f21ed427a5 100644 --- a/Documentation/driver-api/firmware/request_firmware.rst +++ b/Documentation/driver-api/firmware/request_firmware.rst @@ -17,17 +17,17 @@ an error is returned. request_firmware -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :functions: request_firmware request_firmware_direct --- -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :functions: request_firmware_direct request_firmware_into_buf - -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :functions: request_firmware_into_buf Asynchronous firmware requests @@ -41,7 +41,7 @@ in atomic contexts. request_firmware_nowait --- -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :functions: request_firmware_nowait Special optimizations on reboot @@ -55,7 +55,7 @@ firmare to be loaded. firmware_request_cache() --- -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :functions: firmware_request_cache request firmware API expected driver use diff --git a/Documentation/driver-api/infrastructure.rst b/Documentation/driver-api/infrastructure.rst index 6d9ff316b608..bee1b9a1702f 100644 --- a/Documentation/driver-api/infrastructure.rst +++ b/Documentation/driver-api/infrastructure.rst @@ -28,7 +28,7 @@ Device Drivers Base .. kernel-doc:: drivers/base/node.c :internal: -.. kernel-doc:: drivers/base/firmware_class.c +.. kernel-doc:: drivers/base/firmware_loader/main.c :export: .. kernel-doc:: drivers/base/transport_class.c diff --git a/Documentation/power/suspend-and-cpuhotplug.txt b/Documentation/power/suspend-and-cpuhotplug.txt index 31abd04b9572..6f55eb960a6d 100644 --- a/Documentation/power/suspend-and-cpuhotplug.txt +++ b/Documentation/power/suspend-and-cpuhotplug.txt @@ -168,7 +168,7 @@ There are some interesting situations involving CPU hotplug and microcode [Please bear in mind that the kernel requests the microcode images from userspace, using the request_firmware() function defined in -drivers/base/firmware_class.c] +drivers/base/firmware_loader/main.c] a. When all the CPUs are identical: signature.asc Description: Digital signature
Re: [PATCH 2/3] doc-rst: Delete output of failed dot-SVG conversion
On Wed, 2017-02-01 at 13:20 -0700, Jonathan Corbet wrote: > On Tue, 31 Jan 2017 15:37:50 + > Ben Hutchings wrote: > > > > I'd just use dot -o. > > > > That does make more sense. I looked for such an option before > > writing > > this, but the manual page doesn't mention it! > > Can I get an updated patch from you? Then I'll apply the whole > set... I've now tested this, and found that dot still touches the output file even if it fails. So changing to -o doesn't fix anything. Please apply the original patch. Ben. -- Ben Hutchings A free society is one where it is safe to be unpopular. - Adlai Stevenson -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/3] doc-rst: Delete output of failed dot-SVG conversion
On Tue, 2017-01-31 at 09:36 +0200, Jani Nikula wrote:
> On Tue, 31 Jan 2017, Ben Hutchings wrote:
> > As we use redirection to create the SVG file, even a failed
> > conversion
> > will create the file and 'make' will consider it up-to-date if the
> > build is retried. We should delete it in case of failure.
> >
> > Fixes: ec868e4ee2bc ("docs-rst: media: build SVG from graphviz
> > files")
> > Signed-off-by: Ben Hutchings
> > ---
> > Documentation/media/Makefile | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/Documentation/media/Makefile
> > b/Documentation/media/Makefile
> > index 32663602ff25..730d73db7c7a 100644
> > --- a/Documentation/media/Makefile
> > +++ b/Documentation/media/Makefile
> > @@ -36,7 +36,7 @@ quiet_cmd_genpdf = GENPDF $2
> > cmd_genpdf = convert $2 $3
> >
> > quiet_cmd_gendot = DOT $2
> > - cmd_gendot = dot -Tsvg $2 > $3
> > + cmd_gendot = dot -Tsvg $2 > $3 || { rm -f $3; exit 1; }
>
> I'd just use dot -o.
That does make more sense. I looked for such an option before writing
this, but the manual page doesn't mention it!
Ben.
> >
> > %.pdf: %.svg
> > @$(call cmd,genpdf,$<,$@)
> >
>
>
--
Ben Hutchings
It is easier to write an incorrect program than to understand a correct
one.
signature.asc
Description: This is a digitally signed message part
[PATCH 1/3] doc-rst: Break shell command sequences on failure
As $(SHELL) doesn't include the -e option, any loop or other sequence
needs to include explicit checks for failing commands.
Fixes: 609afe6b49ef ("Documentation/sphinx: build the media intermediate ...")
Fixes: 606b9ac81a63 ("doc-rst: generic way to build only sphinx sub-folders")
Fixes: cd21379b1698 ("doc-rst: generic way to build PDF of sub-folders")
Signed-off-by: Ben Hutchings
---
Documentation/Makefile.sphinx | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 707c65337ebf..02d84e6dc7fc 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -43,7 +43,7 @@ ALLSPHINXOPTS = $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER))
$(SPHINXOPTS)
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
-loop_cmd = $(echo-cmd) $(cmd_$(1))
+loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
# $2 sphinx builder e.g. "html"
# $3 name of the build subfolder / e.g. "media", used as:
@@ -54,7 +54,7 @@ loop_cmd = $(echo-cmd) $(cmd_$(1))
#e.g. "media" for the linux-tv book-set at ./Documentation/media
quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
- cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR))
$(build)=Documentation/media $2;\
+ cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR))
$(build)=Documentation/media $2 && \
BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath
$(srctree)/$(src)/$5/$(SPHINX_CONF)) \
$(SPHINXBUILD) \
-b $2 \
@@ -63,7 +63,7 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath
$(BUILDDIR)/$3/$4)
-D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
$(ALLSPHINXOPTS) \
$(abspath $(srctree)/$(src)/$5) \
- $(abspath $(BUILDDIR)/$3/$4);
+ $(abspath $(BUILDDIR)/$3/$4)
htmldocs:
@$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,html,$(var),,$(var)))
@@ -80,7 +80,7 @@ pdfdocs:
else # HAVE_PDFLATEX
pdfdocs: latexdocs
- $(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX=$(PDFLATEX)
LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex;)
+ $(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX=$(PDFLATEX)
LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;)
endif # HAVE_PDFLATEX
signature.asc
Description: Digital signature
[PATCH 3/3] doc-rst: Fix recursive make invocation from macros
In any case where we recurse but don't mention $(MAKE) literally in
the recipe, we need to add a '+' at the start of the command to ensure
that parallel makes and various other options work properly.
Fixes: 609afe6b49ef ("Documentation/sphinx: build the media intermediate ...")
Signed-off-by: Ben Hutchings
---
Documentation/Makefile.sphinx | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 02d84e6dc7fc..e14d82a98062 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -66,10 +66,10 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath
$(BUILDDIR)/$3/$4)
$(abspath $(BUILDDIR)/$3/$4)
htmldocs:
- @$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,html,$(var),,$(var)))
+ @+$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,html,$(var),,$(var)))
latexdocs:
- @$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,latex,$(var),latex,$(var)))
+ @+$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,latex,$(var),latex,$(var)))
ifeq ($(HAVE_PDFLATEX),0)
@@ -85,10 +85,10 @@ pdfdocs: latexdocs
endif # HAVE_PDFLATEX
epubdocs:
- @$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,epub,$(var),epub,$(var)))
+ @+$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,epub,$(var),epub,$(var)))
xmldocs:
- @$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,xml,$(var),xml,$(var)))
+ @+$(foreach var,$(SPHINXDIRS),$(call
loop_cmd,sphinx,xml,$(var),xml,$(var)))
# no-ops for the Sphinx toolchain
sgmldocs:
signature.asc
Description: Digital signature
[PATCH 2/3] doc-rst: Delete output of failed dot-SVG conversion
As we use redirection to create the SVG file, even a failed conversion
will create the file and 'make' will consider it up-to-date if the
build is retried. We should delete it in case of failure.
Fixes: ec868e4ee2bc ("docs-rst: media: build SVG from graphviz files")
Signed-off-by: Ben Hutchings
---
Documentation/media/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index 32663602ff25..730d73db7c7a 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -36,7 +36,7 @@ quiet_cmd_genpdf = GENPDF $2
cmd_genpdf = convert $2 $3
quiet_cmd_gendot = DOT $2
- cmd_gendot = dot -Tsvg $2 > $3
+ cmd_gendot = dot -Tsvg $2 > $3 || { rm -f $3; exit 1; }
%.pdf: %.svg
@$(call cmd,genpdf,$<,$@)
signature.asc
Description: Digital signature
[PATCH 0/3] doc-rst: Build fixes
This series fixes some bugs I found in the new doc build system. Ben. Ben Hutchings (3): doc-rst: Break shell command sequences on failure doc-rst: Delete output of failed dot-SVG conversion doc-rst: Fix recursive make invocation from macros Documentation/Makefile.sphinx | 16 Documentation/media/Makefile | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) signature.asc Description: Digital signature
Sphinx builds always write to the source directory
I'm having trouble with the Debian packaging of kernel documentation. We try to build everything in a separate output directory (underneath debian/build), but the new Sphinx-based build system writes to the source directory. Firstly, Python creates bytecode files alongside the module sources. This could be avoided by setting environment variable PYTHONDONTWRITEBYTECODE=1, at the expense of requiring recompilation more often. Secondly, starting with 4.10, Documentation/media/Makefile converts from dot to SVG and from SVG to PDF in the source directory. It seems like it should be possible to put the converted files in the output directory, but I don't know how to tell Sphinx to find them there. Are these issues likely to be fixed? Ben. -- Ben Hutchings Q. Which is the greater problem in the world today, ignorance or apathy? A. I don't know and I couldn't care less. signature.asc Description: This is a digitally signed message part
Re: [PATCH 2/2] security,perf: Allow further restriction of perf_event_open
On Fri, 2016-06-17 at 08:56 +0300, Alexander Shishkin wrote: > Ben Hutchings writes: > > > When kernel.perf_event_open is set to 3 (or greater), disallow all > > access to performance events by users without CAP_SYS_ADMIN. > > Add a Kconfig symbol CONFIG_SECURITY_PERF_EVENTS_RESTRICT that > > makes this value the default. > > So this patch does two things, can it then be made into two patches? It could. > > > > This is based on a similar feature in grsecurity > > (CONFIG_GRKERNSEC_PERF_HARDEN). This version doesn't include making > > the variable read-only. It also allows enabling further restriction > > at run-time regardless of whether the default is changed. > > This paragraph doesn't seem to belong in the commit message. I'm giving credit where credit is due. > What this commit message is missing entirely is the rationale behind > this change other than "grsecurity does the same". Can you please > elaborate? It allows disabling a facility which in many systems is not needed and is only a security risk. > > Signed-off-by: Ben Hutchings > > --- > > I made a similar change to Debian's kernel packages in August, > > including the more restrictive default, and no-one has complained yet. > > As a debian user, is this a good place to complain? Because it does get > it the way. OK, then you're the first one. And you know how to change this, don't you? Ben. -- Ben Hutchings We get into the habit of living before acquiring the habit of thinking. - Albert Camus signature.asc Description: This is a digitally signed message part
Re: [kernel-hardening] Re: [PATCH 2/2] sysctl: allow CLONE_NEWUSER to be disabled
On Fri, 2016-01-22 at 15:00 -0800, Kees Cook wrote: > On Fri, Jan 22, 2016 at 2:55 PM, Robert Święcki wrote: > > 2016-01-22 23:50 GMT+01:00 Kees Cook : > > > > > > Seems that Debian and some older Ubuntu versions are already using > > > > > > > > $ sysctl -a | grep usern > > > > kernel.unprivileged_userns_clone = 0 > > > > > > > > Shall we be consistent wit it? > > > > > > Oh! I didn't see that on systems I checked. On which version did you find > > > that? > > > > $ uname -a > > Linux bc1 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.3-5~bpo8+1 > > (2016-01-07) x86_64 GNU/Linux > > $ cat /etc/debian_version > > 8.2 > > Ah-ha, Debian only, though it looks like this was just committed to > the Ubuntu kernel tree too: > > > > IIRC some older kernels delivered with Ubuntu Precise were also using > > it (but maybe I'm mistaken) > > I don't see it there. > > I think my patch is more complete, but I'm happy to change the name if > this sysctl has already started to enter the global consciousness. ;) > > Serge, Ben, what do you think? I agree that using the '_restrict' suffix for new restrictions makes sense. I also don't think that a third possible value for kernel.unprivileged_userns_clone would would be understandable. I would probably make kernel.unprivileged_userns_clone a wrapper for kernel.userns_restrict in Debian, then deprecate and eventually remove it. Ben. -- Ben Hutchings Life is what happens to you while you're busy making other plans. - John Lennon signature.asc Description: This is a digitally signed message part
