Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
On 3/19/20 8:04 PM, Eli Schwartz wrote: > On 3/19/20 10:02 PM, Dimitri John Ledkov wrote: >> I am sorry which email are the "quoted" phrases from? not from an email. directly from the bug I referenced, prior to coming here: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868138/comments/4 , subsequent to chats in #irc >> Because that's nothing that I have said. sure. that's clear, and certainly not assumed! >> I hope you do understand that it >> is intended for all of the Ubuntu installer & upgrade integrations to keep >> valid configs compatible. That's the assumption -- or at least the hope -- for all the mainstream distros. And Grub's certainly about as consistent/portable/reliable/cross-everything as it gets. And appears to include aforementioned Posix script compliance for the config script. Ubu dev's proclaimation as otherwise is what caused the ... surprise ... & the questions here. Bottom line ... Grub continues to work exactly reliably/predictably, as it should. Ubu's install should be fixed, not hand-waved away for specious reasons. That can be dealt with best in the bug itself. thx. ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
On 3/19/20 10:02 PM, Dimitri John Ledkov wrote: > I am sorry which email are the "quoted" phrases from? > > Because that's nothing that I have said. I hope you do understand that it > is intended for all of the Ubuntu installer & upgrade integrations to keep > valid configs compatible. And in general, distros do not intentially break > their users for no reason. I simply stated what are the most typical and > the most tested customisation and integration point for grub configs on > Ubuntu, without making any statements about supportability of the configs > that you have observed getting malformed. It is the position of the Ubuntu developer who responded to the OP's launchpad.net bug which motivated this cross-post to the grub-devel list. Direct link to the quoted phrases: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868138/comments/4 -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
I am sorry which email are the "quoted" phrases from? Because that's nothing that I have said. I hope you do understand that it is intended for all of the Ubuntu installer & upgrade integrations to keep valid configs compatible. And in general, distros do not intentially break their users for no reason. I simply stated what are the most typical and the most tested customisation and integration point for grub configs on Ubuntu, without making any statements about supportability of the configs that you have observed getting malformed. On Fri, 20 Mar 2020, 01:08 PGNet Dev, wrote: > hi > > On 3/19/20 5:57 PM, Dimitri John Ledkov wrote: > > > > What is the bug number there? > > > > > https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868138 > > > > > In general, we advise to customize via grub.d drop-in files instead of > modifying etc/default/grub file itself. > > > > Sure, that's fine as advice. Fwiw, it's different for different distros. > > > > Declarations that using posix-compliant continuations is "highly unusual > abuse of POSIX shell semantics" is a different matter. > > > > Changing/breaking behavior in managing valid/working grub configs, and > rationalizing it by "I do not see a need to support it" > > is ... unhelpful. > > > > Particularly if Grub project/devs say that its usage *is* OK & compliant. > Which is why I asked _here_; and, iiuc, seems that it *is*. > > > > Grub's stability, reproducibility and portability are critical. It all > works great, particularly when the excellent docs are followed. > > Happy to take ongoing conversation to the bug. > > > > ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
hi, On 3/19/20 6:41 PM, Eli Schwartz wrote: > In general, other distros avoid parsing and rewriting configs because > this should be the user's responsibility. :p > > The OP should consider that maybe such distros are more stable than Ubuntu? I do. Where I directly control them, they're in use. It includes my own spins ... All use Grub. "Happily". ;-) Ubu's a not-infrequent reality nonetheless. > The OP's question is > > "Does the grub project consider /etc/default/grub as a shell-compatible > file which is sourced, to be the PUBLIC INTERFACE of this configuration > file?" yep. well restated. > The conclusion is simple enough: if you are going to automatically > rewrite this grub file, i'd thought that non-mangling of it would be a reasonable position; that, apparently, is not a mandate, iiuc. tbh, i'm not clear _why_ it's being written at all. i agree with the "this should be the user's responsibility", and would add "only the user's prerogative". but that's me. well, maybe not JUST me ... > you must do so using a parser engine that > implements POSIX shell and knows how to properly tokenize and rewrite > the full range of language features permitted by POSIX shell. > > Doing otherwise is YOLO, rash, buggy, and prone to breakage as > demonstrated by the OP's bug report. > > If a downstream project making use of grub, then goes ahead and calls > the use of backslash escapes for the very simple purpose of line > continuation across multiple lines (of all the things one might do in a > POSIX sh formatted configuration file, this is pretty tame), an issue of > "highly unusual abuse of POSIX shell semantics", this is pretty bad I > have to say. +1 > Just my $0.02 as a distro person (not a grub dev). heh. cheap at twice the price! ;-) thx. ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
On 3/19/20 8:57 PM, Dimitri John Ledkov wrote: > Have you opened a launchpad bug report against the grub2 package with both > configs before and after? What is the bug number there? > > In general, we do parse and rewrite configs using debconf which is Perl / C > / Shell processing using tools external to grub. In general, we advise to > customize via grub.d drop-in files instead of modifying etc/default/grub > file itself. > > This is a bit out of scope for the upstream mailing list, as debconf > integration is downstream specific on Ubuntu/Debian systems. In general, other distros avoid parsing and rewriting configs because this should be the user's responsibility. :p The OP should consider that maybe such distros are more stable than Ubuntu? ... Anyway, it seems clear-cut to me, as per the question which is NOT about what should be done with debconf. The OP's question is very simple: "Does the grub project consider /etc/default/grub as a shell-compatible file which is sourced, to be the PUBLIC INTERFACE of this configuration file?" It seems clear-cut to me. The quoted manual section explicitly documents that it is POSIX shell format, then calls out the idea of KEY=VALUE assignments and declares this to be *merely* "normally", thus highlighting the possibility that it isn't exclusively so. Compare this to, for example, the os-release(5) man page which states the following format for /etc/os-release: """ The basic file format of os-release is a newline-separated list of environment-like shell-compatible variable assignments. It is possible to source the configuration from shell scripts, however, beyond mere variable assignments, no shell features are supported (this means variable expansion is explicitly not supported), allowing applications to read the file without implementing a shell compatible execution engine. """ Note how os-release(5) explicitly goes to some lengths to declare the need for parsing this file in C, that it is *not* valid to simply use shell code, and then continues on to enumerate which subset of shell code it supports. The conclusion is simple enough: if you are going to automatically rewrite this grub file, you must do so using a parser engine that implements POSIX shell and knows how to properly tokenize and rewrite the full range of language features permitted by POSIX shell. Doing otherwise is YOLO, rash, buggy, and prone to breakage as demonstrated by the OP's bug report. If a downstream project making use of grub, then goes ahead and calls the use of backslash escapes for the very simple purpose of line continuation across multiple lines (of all the things one might do in a POSIX sh formatted configuration file, this is pretty tame), an issue of "highly unusual abuse of POSIX shell semantics", this is pretty bad I have to say. Also, it's victim blaming. Just my $0.02 as a distro person (not a grub dev). -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
hi On 3/19/20 5:57 PM, Dimitri John Ledkov wrote: > What is the bug number there? https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868138 > In general, we advise to customize via grub.d drop-in files instead of > modifying etc/default/grub file itself. Sure, that's fine as advice. Fwiw, it's different for different distros. Declarations that using posix-compliant continuations is "highly unusual abuse of POSIX shell semantics" is a different matter. Changing/breaking behavior in managing valid/working grub configs, and rationalizing it by "I do not see a need to support it" is ... unhelpful. Particularly if Grub project/devs say that its usage *is* OK & compliant. Which is why I asked _here_; and, iiuc, seems that it *is*. Grub's stability, reproducibility and portability are critical. It all works great, particularly when the excellent docs are followed. Happy to take ongoing conversation to the bug. ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
Have you opened a launchpad bug report against the grub2 package with both configs before and after? What is the bug number there? In general, we do parse and rewrite configs using debconf which is Perl / C / Shell processing using tools external to grub. In general, we advise to customize via grub.d drop-in files instead of modifying etc/default/grub file itself. This is a bit out of scope for the upstream mailing list, as debconf integration is downstream specific on Ubuntu/Debian systems. On Thu, 19 Mar 2020, 21:20 PGNet Dev, wrote: > a recent grub package update, in ubuntu 18LTS, is breaking > /etc/default/grub by mangling/overwriting users' entries, in the specific > case of using continuation lines in the file/config. and, subsequently, the > upgrade process on these servers. > > as stated clearly at > > > https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html#Simple-configuration-handling > > "... > The file /etc/default/grub controls the operation of > grub-mkconfig. It is sourced by a shell script, and so must be valid POSIX > shell input; normally, it will just be a sequence of ‘KEY=value’ lines, but > if the value contains spaces or other special characters then it must be > quoted > ..." > > per POSIX > > > https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_01 > > "... > Escape Character (Backslash) > > A that is not quoted shall preserve the literal value > of the following character, with the exception of a . If a > follows the , the shell shall interpret this as line > continuation. The and shall be removed before > splitting the input into tokens. Since the escaped is removed > entirely from the input and is not replaced by any white space, it cannot > serve as a token separator. > > ..." > > all that's to say that continuation lines appear to be supported in > posix-shell-compliant /etc/default/grub > > they've been in use here, on 100s of servers, in grub configs for ages. > > with grub 2.04 on other, non Ubu OS, it continues to work fine. > > re: the use of continuation lines, ubu dev states that > > "Regardless of what the POSIX spec says, this is highly unusual > abuse of POSIX shell semantics, and I do not see a need to support it" > > it'd be useful/helpful to get clear on what the 'official' grub project > support for use of posix-shell-compliant continuation lines is ... > > are they 'supported' as valid use in /etc/default/grub, or not? > > also useful to know/understand whether any grub update can/should mangle a > user's /etc/default/grub. allowed? expected? > > thx! > > > > ___ > Grub-devel mailing list > Grub-devel@gnu.org > https://lists.gnu.org/mailman/listinfo/grub-devel > ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
On Thu, Mar 19, 2020 at 5:19 PM PGNet Dev wrote: > > a recent grub package update, in ubuntu 18LTS, is breaking /etc/default/grub > by mangling/overwriting users' entries, in the specific case of using > continuation lines in the file/config. and, subsequently, the upgrade process > on these servers. This sounds like an issue you should hash out with your distro maintainer. How Ubuntu manages package upgrades is not within the scope of this mailing list. > it'd be useful/helpful to get clear on what the 'official' grub project > support for use of posix-shell-compliant continuation lines is ... > are they 'supported' as valid use in /etc/default/grub, or not? The manual section you quoted seems pretty clear to me: it must be valid POSIX shell input. Backslash-escaped newlines are valid POSIX shell input. Technically speaking, /etc/default/grub is sourced by grub-mkconfig, which is executed by /bin/sh. So long as /bin/sh is able to correctly process both grub-mkconfig and /etc/default/grub, you probably wont have a problem. If /bin/sh happens to be bash, you could probably get away with putting bash-specific code in /etc/default/grub if you really wanted to. What your distro chooses to support is really up to them, however. > also useful to know/understand whether any grub update can/should mangle a > user's /etc/default/grub. allowed? expected? I think this is not something the grub developers care about; work it out with your distro maintainer. ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
are posix-shell-compliant continuation lines valid/supported, or not, in /etc/default/grub ?
a recent grub package update, in ubuntu 18LTS, is breaking /etc/default/grub by mangling/overwriting users' entries, in the specific case of using continuation lines in the file/config. and, subsequently, the upgrade process on these servers. as stated clearly at https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html#Simple-configuration-handling "... The file /etc/default/grub controls the operation of grub-mkconfig. It is sourced by a shell script, and so must be valid POSIX shell input; normally, it will just be a sequence of ‘KEY=value’ lines, but if the value contains spaces or other special characters then it must be quoted ..." per POSIX https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_01 "... Escape Character (Backslash) A that is not quoted shall preserve the literal value of the following character, with the exception of a . If a follows the , the shell shall interpret this as line continuation. The and shall be removed before splitting the input into tokens. Since the escaped is removed entirely from the input and is not replaced by any white space, it cannot serve as a token separator. ..." all that's to say that continuation lines appear to be supported in posix-shell-compliant /etc/default/grub they've been in use here, on 100s of servers, in grub configs for ages. with grub 2.04 on other, non Ubu OS, it continues to work fine. re: the use of continuation lines, ubu dev states that "Regardless of what the POSIX spec says, this is highly unusual abuse of POSIX shell semantics, and I do not see a need to support it" it'd be useful/helpful to get clear on what the 'official' grub project support for use of posix-shell-compliant continuation lines is ... are they 'supported' as valid use in /etc/default/grub, or not? also useful to know/understand whether any grub update can/should mangle a user's /etc/default/grub. allowed? expected? thx! ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: Verifier running out of memory on ieee1275/powerpc64
On 3/18/20 6:17 PM, Simon Hardy wrote: I was wondering whether it would not be possible to load the raw file into memory, pass it to the firmware for hashing (and logging) via the verifier, and if we do not trust that the firmware treated the file data as a read-only array, load the file again into the same array right after. This way we wouldn't need more memory. [*] However, I am not sure how it fits into the architecture with the verifiers or whether the TPM verifier would have to take on a special role (possibly with a flag) then. You didn't pick up on the idea of a bigger heap. Is there a problem with the heap size somehow? My machine has GBs of memory, so it really shouldn't be a problem to get memory. I think that's the problem to solve, at least for this platform, since none of the verifiers will work due to the memory exhaustion issue. Increasing the heap size may be the easiest way for you to solve this problem. I am not so familiar with the issues around doing this, but I have seen recent discussion on this list. I found the lsmmap command and it shows me some impressive memory ranges to be known: base_addr = 0x4000, length = 0x17c000, available RAM base_addr = 0x1f, length = 0x1, available RAM base_addr = 0x21, length = 0x1, available RAM base_addr = 0x200, length = 0x7bbf, available RAM base_addr = 0x7feff, length = 0x1, available RAM base_addr = 0x800903dc, length = 0x7ff6fc24, available RAM There's more than needed memory here. It doesn't seem to be available for grub_malloc, though. The reason there is extra memory usage is that the verifier framework is built on top of the file filter framework. The process is: 1. The file is opened using grub_file_open. 2. The verifier framework opens the underlying file and reads all of it into a new heap-allocated memory buffer. 3. The verifier framework provides the buffer to the verifiers for verification. 4. The verifier framework returns a file handle. 5. Read operations on this file handle are served by copying from the verification buffer. Yes, saw this, verify_fs. 6. The file is closed. The verification framework frees its buffer. There are good reasons for doing this. It is more secure to ensure that verification is applied to the same data that is actually used. (An attacker could give you different data on a second read from the disk.) Also true. Some verifiers, like the TPM, prefer to receive the data in a single chunk. Existing code depends on the open/read/seek/close file access model. The problem you have run into is that where you have a use case that involves loading a large file into memory then you need enough free memory to hold two copies of that file. Ideally there would be an alternative grub_file_read_all function that loads the contents of a named file to a user-provided memory buffer. This doesn't look like it would be simple or easy, for example I don't know how this would work for compressed files. Stefan ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: [PATCH v1] fix kernel cmdline corruption
Am Thu, 19 Mar 2020 15:48:28 +0800 schrieb Michael Chang : > compatibilty issue. There can not possibly be any compat issue. Please go ahead and fix this bug. Olaf pgp_HXWiYCbGr.pgp Description: Digitale Signatur von OpenPGP ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel
Re: [PATCH v1] fix kernel cmdline corruption
Hi Olaf, The patch rang a bell to me and eventually I figured out that I had similar patch posted. https://lists.gnu.org/archive/html/grub-devel/2018-04/msg00038.html In short, the issue is not only in the fix itself, but also to take care and provide a measure to avoid incompatible grub.cfg before and after the patch. To recap, there were two options provided to deal with the compatibilty issue. 1. Introduce a shell variable to disable the change, whenever anything goes wrong. 2. Use readonly feature variable similar to $feature_menuentry_id that can be used to detect capability and apply settings accordingly, thus still fulfill the requirement of one grub.cfg can work for all versions IMHO. I am in favour of #2, but still they didn't seem to have upstream's consent yet. Thanks, Michael On Wed, Mar 18, 2020 at 12:13:12PM +0100, Olaf Hering wrote: > The functions grub_create_loader_cmdline and its sibling > grub_loader_cmdline_size add extra, unexpected characters to the kernel > cmdline. Both functions are supposed to copy each argv[] element > verbatim. It is up to the caller of the kernel command ('linux' for > example) to add desired quoting or escaping characters for the consumers > of the kernel command line. The built-in grub shell gives enough > opportunities to construct such argv[] elements. > > After this change, these arguments result in the following kernel cmdline: > > var='val' var="val" var=\'v1 v2\' var=\"v1 v2\" var=\\\"v1 v2\\\" x > var=val var=val var='v1 v2' var="v1 v2" var=\"v1 v2\" x > > Fixes commit 25953e10553dad2e378541a68686fc094603ec54 > > Signed-off-by: Olaf Hering > --- > grub-core/lib/cmdline.c | 58 > - > 1 file changed, 9 insertions(+), 49 deletions(-) > > diff --git a/grub-core/lib/cmdline.c b/grub-core/lib/cmdline.c > index ed0b149dc..b572a367f 100644 > --- a/grub-core/lib/cmdline.c > +++ b/grub-core/lib/cmdline.c > @@ -20,31 +20,6 @@ > #include > #include > > -static unsigned int check_arg (char *c, int *has_space) > -{ > - int space = 0; > - unsigned int size = 0; > - > - while (*c) > -{ > - if (*c == '\\' || *c == '\'' || *c == '"') > - size++; > - else if (*c == ' ') > - space = 1; > - > - size++; > - c++; > -} > - > - if (space) > -size += 2; > - > - if (has_space) > -*has_space = space; > - > - return size; > -} > - > unsigned int grub_loader_cmdline_size (int argc, char *argv[]) > { >int i; > @@ -52,7 +27,7 @@ unsigned int grub_loader_cmdline_size (int argc, char > *argv[]) > >for (i = 0; i < argc; i++) > { > - size += check_arg (argv[i], 0); > + size += grub_strlen(argv[i]); >size++; /* Separator space or NULL. */ > } > > @@ -66,36 +41,21 @@ grub_err_t > grub_create_loader_cmdline (int argc, char *argv[], char *buf, > grub_size_t size, enum grub_verify_string_type type) > { > - int i, space; > + int i; >unsigned int arg_size; > - char *c, *orig_buf = buf; > + char *orig_buf = buf; > >for (i = 0; i < argc; i++) > { > - c = argv[i]; > - arg_size = check_arg(argv[i], &space); > - arg_size++; /* Separator space or NULL. */ > + arg_size = grub_strlen(argv[i]); > > - if (size < arg_size) > + /* Separator space or NULL. */ > + if (size < arg_size + 1) > break; > > - size -= arg_size; > - > - if (space) > - *buf++ = '"'; > - > - while (*c) > - { > - if (*c == '\\' || *c == '\'' || *c == '"') > - *buf++ = '\\'; > - > - *buf++ = *c; > - c++; > - } > - > - if (space) > - *buf++ = '"'; > - > + grub_memcpy(buf, argv[i], arg_size); > + size -= arg_size + 1; > + buf += arg_size; >*buf++ = ' '; > } > > > ___ > Grub-devel mailing list > Grub-devel@gnu.org > https://lists.gnu.org/mailman/listinfo/grub-devel ___ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel