Re: Enable compgen even when programmable completions are not available?
Date:Fri, 30 Jun 2023 20:21:20 -0400 From:Eli Schwartz Message-ID: I suggested using set partly because it should work in any shell. The need, or even ability, to set posix mode is likely to differ, but that is minor. In general I vastly prefer highly portable solutions to all problems .. those tend to fail far less often, even when just considering the evolution of a single application, with no porting to a different one even considered. But moving on | Suggesting ways that future versions of bash 5.3 and above can do this | *without* hacks, would accomplish something. It would accomplish the | ability to migrate away from hacks and over to proper APIs. | | Maybe we should focus on that. That's reasonable, but I would have thought a simple solution obvious, and one which probably does not need to wait for a new bash version (this assumes that solutions which only work in bash are acceptable). Simply create a loadable builtin which just prints all the existing variable names. If desired it could have a few options to filter which are printed (by type, value, name, attributes...) but doesn't need any of that. I personally doubt there is enough demand for this functionality for it to be a permanent builtin (until this discussion I had never seen or heard of an actual script with a need for that exact mechanism) but having a loadable builtin to do it sounds entirely reasonable. Filtering the output in the builtin might make scripts slightly more efficient, but only when the filtering offered is exactly what the application requires, and that is something that is only discovered after actual use. Everything likely to be required can already be done given just the var name, and suitable scripting. You could probably even make one which could be built for, and work in, versions of bash back to whenever the loadable builtin facility first appeared. (Different compiled builtins for different bash versions, obviously.) kre
Re: Enable compgen even when programmable completions are not available?
On 6/30/23 3:25 PM, Robert Elz wrote: > Date:Fri, 30 Jun 2023 18:35:34 +0100 > From:Kerin Millar > Message-ID: <20230630183534.85da7986a24855126bfea...@plushkava.net> > > | This can be trivially foiled. > > You mean it doesn't give you all the variable names? Why not? > Does bash have a bug in this area that I am unaware of? > > Or do you mean that it will sometimes (due to newlines in the values) > be able to be persuaded to give you more than just the var names? > > If the latter, why do you care? The processing can check for each variable > name produced whether it is set or not, and simply skip anything that looks > like it is a variable name, but actually isn't set in the shell. If it is > important that each var name be included just once, just run the output > through sort -u so any duplicates (which will only be caused by that kind > of newline in the value stuff) get removed. > > You're going to need a more sophisticated sed script (but not all that > much more) to remove any "obviously not a variable name followed by =" > lines, but you should be getting all of the variable names, which is the > part that really matters, I am guessing. That is, making sure none are > missing is the challenging part, weeding out fakes is simple. You have very helpfully described how one can design a rube goldberg machine in bash to achieve some important goal. But it rather misses the point that this thread wasn't asking for an answer. The topic is "should compgen be enabled when readline / progcomp is not". The initial premise of the thread described your solution except better -- declare -p, as mentioned in passing by my thread starter and in more depth at the linked gentoo bug. And declare -p is better because it returns the same results as set, except you get even more detail because you can also grep for a leading `^declare`. It was deemed undesirable, nonetheless. Yet another non-compgen hack is described in the gentoo bug. It's still a hack. Brainstorming additional workarounds doesn't accomplish anything above and beyond the first workaround -- which is another way of saying, brainstorming additional workarounds doesn't accomplish anything at all, at the moment. Especially when they are incomplete and come with the caveat that you'll need to do additional sed postprocessing, "check for this and that", and various forms of English prose description about how to accomplish a task... .. that was already worked around with a hack and has a fully implemented patch to do so on the gentoo ticket. Suggesting ways that future versions of bash 5.3 and above can do this *without* hacks, would accomplish something. It would accomplish the ability to migrate away from hacks and over to proper APIs. Maybe we should focus on that. -- Eli Schwartz
Re: [PATCH] fix printing command after group with heredoc
On 6/29/23 4:05 AM, Grisha Levit wrote:
If the last redirection list in a group / subshell / substitution has
a heredoc and the following connector is a semicolon, the connector is
incorrectly skipped over.
Try this patch instead:
*** ../bash-20230628/print_cmd.cMon Jun 26 16:56:46 2023
--- print_cmd.c Fri Jun 30 19:35:35 2023
***
*** 346,349
--- 352,356
PRINT_DEFERRED_HEREDOCS ("");
cprintf (" )");
+ was_heredoc = 0; /* last wasn't heredoc/newline */
break;
***
*** 699,702
--- 706,710
cprintf ("}");
+ was_heredoc = 0;/* last wasn't heredoc/newline */
group_command_nesting--;
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: maybe a bug in bash?
On Sat, Jul 01, 2023 at 05:47:33AM +0900, Dominique Martinet wrote:
> hm, this has the password show up in ps on the box executing ssh;
> depending on the context that can be bad.
Fair enough. We have no way to judge the OP's risk analysis, because
they've kept almost everything secret so far.
I place an extremely high value on avoiding solutions that embed
passwords inside dynamically generated scripts. Other people may
have a different set of priorities.
> It does simplify the content of the here-doc a bit because it doesn't
> require escaping, but the password itself still needs one layer of
> escaping (so in his example not ${password@Q} but ${initial_password@Q}
> or $password), and we don't know enough to know if showing up in ps can
> be important but passwords have generally been recommended to be passed
> through stdin
Sure, that would be preferred in most cases. But the OP is specifically
giving us a situation where stdin has been tied up sending an ad hoc
shell script to be interpreted on the remote host.
There are many other ways this problem could be approached. If the script
isn't really ad hoc, but is instead something that's going to be executed
periodically, it could be stored permanently on the server. Then you
could use something like
printf '%s\n' "$password" | ssh user@host /opt/foo/bin/myscript
where /opt/foo/bin/myscript reads the password from stdin. You could
send additional parameters as well, if needed, to prevent needing to
send a slightly modified script every time.
If you *really* want to embed a password inside a dynamically generated
script, perhaps an approach like this would be less dangerous:
start="IFS= read -r pass <<'EOF'"
end='
EOF
The rest of
the script
goes here
and you can'\''t easily
use single quotes.
'
printf '%s\n%s\n%s' "$start" "$password" "$end" |
ssh user@host bash
The definition of "end" could be loaded into a variable from a here
document to avoid the '\'' issue, if you wish. The extra blank line
between the password and EOF doesn't matter, because read only reads
the first line.
I haven't tested this method, but it looks... less bad than some others.
Oh! And you have to make sure the password isn't EOF. Or else use
a heredoc sentinel that begins with a character that's not allowed
in the password. Or is a password that would be rejected by policy
(which, to be fair, EOF would probably be rejected due to shortness).
Or something like that.
Re: Enable compgen even when programmable completions are not available?
On 6/30/23 12:53 PM, Chet Ramey wrote: >> And because this is conditional on readline, which is usually an >> external library dependency (a global system policy decision), reducing >> the number of shared libraries the dynamic loader has to deal with might >> be especially interesting. > > The dynamic loader has to know where the library is. If you don't call > readline, it shouldn't ever have to actually map it into the process. >> (This is all theorizing -- I quite like bash as an interactive shell and I'm not certain how to respond to this other than "pretty sure that's not how it works". We're not talking about dlopen here. Note also that even with lazy binding (which is frequently disabled as a system-wide security policy) the library is still opened, it just doesn't do relocations until they are used. But this feels like getting into the weeds -- it's not really related to the goal of this conversation to debate exactly how efficient it is to link to a library when that library's functionality is only used by some parts of the program and never used on certain machines at all. >> have no intention of building systems with readline disabled. It is >> nonetheless true that the topic came up because there are Gentoo users >> who apparently decided to try to do so.) > > Yes, but the question is whether or not that makes sense in the modern age, > and whether there should be extra features to accommodate that decision. ... instead, I would like to discuss the fact that it *is* possible. If it doesn't make sense in the modern age, why not get rid of the option and make readline mandatory? If it makes sense to keep the option, then why does it *matter* what people's justifications are for using the option? Writing robust software means taking into account that it may be getting toggled. > But this isn't right. You have to explicitly disable those configuration > options -- they're on by default. You don't have to do anything to get > readline support compiled into bash. You have to do things to disable it. > If you take that extra configuration step to disable it, there are going > to be consequences. "There will be consequences if you use this option". What does this mean? Is it unsupported to do so? Defaults are just defaults, not prescriptive instructions about what is reasonable or sensible. Saying that something defaults to on, isn't a good rationale for whether people toggling the default constitute an unsupported state prone to not working. > You can always check whether compgen is available and fall back to some > other solution if it's not. > > compgen -v >/dev/null 2>&1 && have_compgen=1 This makes zero sense, sorry. If I have to handle the case where compgen doesn't exist, and test that "some other solution" exists and returns correct results, then at that point I have no need of compgen and can just not bother using it at all. It no longer provides added value. The reason this is problematic is because there is a degree of skepticism regarding whether such a fallback is correct at all. But if I have a correct solution and am willing to support the hacks it requires, then... mission accomplished? >> Should I document in the project readme that in addition to needing a >> certain minimum version of bash, "you also need to make sure that >> programmable completions are enabled when compiling your bash binary"? > > No. You need to say that users should make sure they haven't disabled > them when compiling their bash binary. This is splitting hairs. "You need to do X." "You need to not not do X." >> Should I eschew compgen and rely on eval-using hacks like the one Kerin >> described? > > It's your call, of course. You just have to decide whether or not it's > worth the effort to accommodate non-default option choices. What about > aliases? Arrays? Brace expansion? Process substitution? Extglobs? All of > those can be compiled out. What's the `bash core' you're going to assume? This feels like a deeply, deeply, deeply unfair comparison. I do not even understand why those options exist. It's like having an option to compile GCC without support for for loops. What utility role is such a bash supposed to perform? Readline/progcomp support has a pretty clear scope. Bash is two things: - an interactive console - a valid script interpreter and people using bash for a script interpreter have an easily understood rationale for disabling ***library dependencies*** whose only effects are to enable interactive functionality. The expectation is going to be, that disabling this functionality will have no effect on use as a script interpreter, and it's okay to disable use as an interactive console because the user would, well, *interactively* notice problems which happen as a result. And that's kind of my whole question here to begin with. It's not clear to me that it's a good idea to allow bash to disable part of the bash scripting language at all. The "bash core" I want t
Re: maybe a bug in bash?
Greg Wooledge wrote on Fri, Jun 30, 2023 at 10:36:36AM -0400:
> > then as Greg suggested pass password to bash as argument instead;
> > assuming password has been quoted once as previously:
> > ssh user@machine << EOF
> > bash -c 'echo \$1' -- $password
> > EOF
>
> I never suggested embedding the password inside an unquoted here document.
> My suggestion was more like this:
>
> ssh user@machine bash -s "${password@Q}" <<'EOF'
> echo "$1"
> EOF
hm, this has the password show up in ps on the box executing ssh;
depending on the context that can be bad.
It does simplify the content of the here-doc a bit because it doesn't
require escaping, but the password itself still needs one layer of
escaping (so in his example not ${password@Q} but ${initial_password@Q}
or $password), and we don't know enough to know if showing up in ps can
be important but passwords have generally been recommended to be passed
through stdin
(The here-doc can use a file as backing so it's not necessarily great
either, but hopefully /tmp will be tmpfs and it'll only be readable by
root as opposed to readable by anyone who can read /proc which is pretty
much everyone by default)
Anyway, I agree on need-more-info and I'll probably stop replying to
this; there's been enough helpful answers.
--
Dominique Martinet | Asmadeus
Re: [PATCH] printing multiple heredocs in list
On Fri, Jun 30, 2023, 16:02 Chet Ramey wrote: > On 6/29/23 4:01 AM, Grisha Levit wrote: > > If there are multiple commands in a row that each require printing the > > connector prior to the heredoc body, the connector ends up in the wrong > > place for commands after the first: > > OK. The parser builds lists (connections) left-side heavy. What do you > think of replacing the test for was_heredoc in your patch with > (printing_connection == 1), so if we're called recursively to print a > second connection on the left side (Connector->first), we print the here- > document body after our caller has been able to add the connector in > the right place. Oh yeah that seems like a much better solution. That way you shouldn't need to print any deferred > here-documents in make_command_string(). > Great, this was the part that made me think something was wrong with my approach. >
Re: Enable compgen even when programmable completions are not available?
On Sat, 01 Jul 2023 02:25:33 +0700 Robert Elz wrote: > Date:Fri, 30 Jun 2023 18:35:34 +0100 > From:Kerin Millar > Message-ID: <20230630183534.85da7986a24855126bfea...@plushkava.net> > > | This can be trivially foiled. > > You mean it doesn't give you all the variable names? Why not? > Does bash have a bug in this area that I am unaware of? > > Or do you mean that it will sometimes (due to newlines in the values) > be able to be persuaded to give you more than just the var names? > > If the latter, why do you care? The processing can check for each variable Well, I don't in particular. However, it is also the case that: - every single solution posted in this thread has been broken in its initial form, save for that which I had devised even before Eli posted - I have to yet to see one working solution that provides any tangible advantage over mine - I am not soliciting solutions in any case; they have been nothing but a distraction - I am tired of being told things that I already know and am frankly now tired of this entire thread; nothing is going to come of it -- Kerin Millar
Re: [PATCH] printing multiple heredocs in list
On 6/29/23 4:01 AM, Grisha Levit wrote: If there are multiple commands in a row that each require printing the connector prior to the heredoc body, the connector ends up in the wrong place for commands after the first: OK. The parser builds lists (connections) left-side heavy. What do you think of replacing the test for was_heredoc in your patch with (printing_connection == 1), so if we're called recursively to print a second connection on the left side (Connector->first), we print the here- document body after our caller has been able to add the connector in the right place. That way you shouldn't need to print any deferred here-documents in make_command_string(). -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Enable compgen even when programmable completions are not available?
Date:Fri, 30 Jun 2023 18:35:34 +0100 From:Kerin Millar Message-ID: <20230630183534.85da7986a24855126bfea...@plushkava.net> | This can be trivially foiled. You mean it doesn't give you all the variable names? Why not? Does bash have a bug in this area that I am unaware of? Or do you mean that it will sometimes (due to newlines in the values) be able to be persuaded to give you more than just the var names? If the latter, why do you care? The processing can check for each variable name produced whether it is set or not, and simply skip anything that looks like it is a variable name, but actually isn't set in the shell. If it is important that each var name be included just once, just run the output through sort -u so any duplicates (which will only be caused by that kind of newline in the value stuff) get removed. You're going to need a more sophisticated sed script (but not all that much more) to remove any "obviously not a variable name followed by =" lines, but you should be getting all of the variable names, which is the part that really matters, I am guessing. That is, making sure none are missing is the challenging part, weeding out fakes is simple. kre
Re: +=() can be used to set illegal indices
On Fri, Jun 30, 2023 at 12:16:46PM -0400, Chet Ramey wrote:
> > What I would have expected was something like this:
> >
> >$ x=([9223372036854775805]=foo)
> >$ x+=( {1..5} ); echo "this won't run"
> >bash: some "invalid assignment" error
> >$ declare -p x # no value gets appended since there was an error
>
> No. Why wouldn't the valid assignments be accepted? If you want
>
> a=(1 2 3)
>
> to be as close as possible to
>
> a[0]=1
> a[1]=2
> a[2]=3
>
> you have to do it that way.
Hmm, I have never noticed that behaviour to be honest.
I would have expected:
$ a=([1]=hi [100]=foo [-1002]=bar boo [200]=baz); echo "won't run"
bash: [-1002]=bar: bad array subscript
$ declare -p a
bash: declare: a: not found
But it seems bash actually behaves like so:
$ a=([1]=hi [100]=foo [-1002]=bar boo [200]=baz); echo "will run"
bash: [-1002]=bar: bad array subscript
will run
$ declare -p a
declare -a a=([1]="hi" [100]="foo" [101]="boo" [200]="baz")
So it simply skips and prints a warning for invalid indices, and still
sets all the other valid indices, without triggering an error for the
assignment; even though a[-1002]=bar on its own would have triggered
an error:
$ a[1]=hi a[100]=foo a[-1002]=bar a[200]=baz; echo "won't run"
bash: [-1002]=bar: bad array subscript
$ declare -p a
declare -a a=([1]="hi" [100]="foo")
The ksh behaviour is more similar to a sequence of a[k1]=v1 a[k2]=v2
assignments; only the assignments before the first invalid index are
performed, and the assignment triggers an error (n.b. you need to use
typeset -a with ksh93 otherwise it creates an associative array if you
use a=([k1]=v1 [k2]=v2) ):
$ typeset -a a=([1]=hi [100]=foo [-1002]=bar [200]=baz); echo "won't run"
ksh: a: subscript out of range
$ typeset -p a
typeset -a a=([1]=hi [100]=foo)
Anyhow, if x+=() should behave similarly to x=( ... ), I guess my
example should work like so:
$ x=([9223372036854775805]=foo)
$ x+=( {1..5} ); echo "will run"
bash: x[-9223372036854775808]: bad array subscript
bash: x[-9223372036854775807]: bad array subscript
bash: x[-9223372036854775806]: bad array subscript
will run
$ echo "${x[@]}"
foo 1 2
o/
emanuele6
Re: Enable compgen even when programmable completions are not available?
On 6/30/23 1:19 PM, Robert Elz wrote: > Date:Thu, 29 Jun 2023 23:05:38 +0100 > From:Kerin Millar > Message-ID: <20230629230538.cbef14a75694143ccf034...@plushkava.net> > > | The thing is that portage also has a legitimate stake in needing > | to enumerate all variable names. > > Why isn't "set" good enough for that? > > (set -o posix; set) | sed 's/=.*$//' | whatever_processes_them > > You need posix mode, so bash doesn't dump functions as well (which IMO > it shouldn't do with set, ever, but has probably been doing it since > prehistoric times, and somewhere, there's probably 1 user depending upon > it...) So then you get what? The exact same thing, with all the attendant flaws, as declare -p which is already known not to work? You still have to sed/grep the contents, and you still have the original declare -p problem that you need to parse the value of the variable in order to figure out *where* it is so you can discard and ignore it... which is impractical without an actual shell parser. -- Eli Schwartz
Re: Enable compgen even when programmable completions are not available?
On Sat, 01 Jul 2023 00:19:41 +0700 Robert Elz wrote: > Date:Thu, 29 Jun 2023 23:05:38 +0100 > From:Kerin Millar > Message-ID: <20230629230538.cbef14a75694143ccf034...@plushkava.net> > > | The thing is that portage also has a legitimate stake in needing > | to enumerate all variable names. > > Why isn't "set" good enough for that? > > (set -o posix; set) | sed 's/=.*$//' | whatever_processes_them This can be trivially foiled. -- Kerin Millar
Re: Enable compgen even when programmable completions are not available?
Date:Thu, 29 Jun 2023 23:05:38 +0100 From:Kerin Millar Message-ID: <20230629230538.cbef14a75694143ccf034...@plushkava.net> | The thing is that portage also has a legitimate stake in needing | to enumerate all variable names. Why isn't "set" good enough for that? (set -o posix; set) | sed 's/=.*$//' | whatever_processes_them You need posix mode, so bash doesn't dump functions as well (which IMO it shouldn't do with set, ever, but has probably been doing it since prehistoric times, and somewhere, there's probably 1 user depending upon it...) kre
Re: Enable compgen even when programmable completions are not available?
On 6/29/23 11:16 PM, Eli Schwartz wrote: I assume that, given the option exists in the configure script, there are people who will want to use the option made available to them. One reason might be because they are configuring it for a system that isn't fussed about using bash for an interactive shell (either it is administrated via non-interactive means, or simply because the preferred interactive site shell is e.g. zsh). In such cases, a rationale that readily comes to mind is "this user wanted a smaller, leaner bash binary by disabling unimportant bits that they do not use". Maybe. I don't think it's that big a win. And because this is conditional on readline, which is usually an external library dependency (a global system policy decision), reducing the number of shared libraries the dynamic loader has to deal with might be especially interesting. The dynamic loader has to know where the library is. If you don't call readline, it shouldn't ever have to actually map it into the process. (This is all theorizing -- I quite like bash as an interactive shell and have no intention of building systems with readline disabled. It is nonetheless true that the topic came up because there are Gentoo users who apparently decided to try to do so.) Yes, but the question is whether or not that makes sense in the modern age, and whether there should be extra features to accommodate that decision. The thing is, does it really matter? I think there's a larger issue here, which I mentioned in the Gentoo bug report but probably makes sense to copy/paste here: The problem with compgen is that it is only available for use when bash is configured with --enable-progcomp / --enable-readline, which feels like a powerful argument that script authors are not allowed to assume that it will exist, regardless of how useful it may be in non-programmable-completion contexts. Maybe the answer is to ask that it always be available as a builtin, even when the programmable completion system isn't enabled. But this isn't right. You have to explicitly disable those configuration options -- they're on by default. You don't have to do anything to get readline support compiled into bash. You have to do things to disable it. If you take that extra configuration step to disable it, there are going to be consequences. So: can I? Are my bash scripts valid scripts if they use compgen, or should I be concerned that when I publish them for wide adoption, people are going to report bugs to me that it is broken on their niche system configuration which they are positive they have a good reason for? You can always check whether compgen is available and fall back to some other solution if it's not. compgen -v >/dev/null 2>&1 && have_compgen=1 Should I document in the project readme that in addition to needing a certain minimum version of bash, "you also need to make sure that programmable completions are enabled when compiling your bash binary"? No. You need to say that users should make sure they haven't disabled them when compiling their bash binary. Should I eschew compgen and rely on eval-using hacks like the one Kerin described? It's your call, of course. You just have to decide whether or not it's worth the effort to accommodate non-default option choices. What about aliases? Arrays? Brace expansion? Process substitution? Extglobs? All of those can be compiled out. What's the `bash core' you're going to assume? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: +=() can be used to set illegal indices
On 6/29/23 6:24 AM, Emanuele Torre wrote:
The fix in the most recent push did not fix the issue:
$ x=([9223372036854775805]=foo)
$ x+=( {1..5} )
$ echo "${x[@]}"
3 4 5 foo 1 2
It behaves exactly the same way: no error is printed, some elements are
prepended instead of appended, and there are invalid indices in the
output of declare -p .
Thanks, good catch. The check really needs to be in the loop that goes
through the words in the compound assignment.
What I would have expected was something like this:
$ x=([9223372036854775805]=foo)
$ x+=( {1..5} ); echo "this won't run"
bash: some "invalid assignment" error
$ declare -p x # no value gets appended since there was an error
No. Why wouldn't the valid assignments be accepted? If you want
a=(1 2 3)
to be as close as possible to
a[0]=1
a[1]=2
a[2]=3
you have to do it that way.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Enable compgen even when programmable completions are not available?
On 6/29/23 6:05 PM, Kerin Millar wrote: On Thu, 29 Jun 2023 16:39:52 -0400 Chet Ramey wrote: On 6/25/23 2:38 PM, Eli Schwartz wrote: compgen is a useful builtin for inspecting information about the shell context e.g. in scripts -- a good example of this is compgen -A function or compgen -A variable. But it's not always available depending on how bash is built, which results in people lacking confidence that it can / should be used in scripts. See e.g. https://bugs.gentoo.org/909148 It's dependent on programmable completion and readline, which are features that are enabled by default. Who builds a version of bash with those turned off? What's the rationale for doing that? This is discussed in the referenced bug. Yes, the fact that gentoo uses a bash built without readline is there, but no hint about why it's built that way. To begin with, readline is disabled during gentoo's bootstrapping stage. I'm not one of its release engineers, but I presume that there is a cromulent reason for doing so. Sam might be able to chime in on this. Excellent use of `cromulent'. Now for my part in this. Without going into the details of portage's internals, some of the code concerning the processing of the ebuild environment is ... well, let me just put it politely by saying that it is far from ideal. I have been working on cleaning much of this up. Consequently, I have come to rely upon compgen quite heavily, duly compelling me to pose the same question: what is the reason for disabling readline support in the first place? In fact, not only is it unavailable during bootstrapping but users do also have the option to disable it. Admittedly, they never do in practice; it's an option that is made deliberately difficult to take advantage of. In any case, comments #7 and #8 were the catalyst for bringing the matter up here on the list, if you're interested. I skimmed the bug report. The option to disable readline during a bash build is there because users asked for it long ago. I agree that it doesn't make much sense in this day and age, and no one ever does it. The question then becomes what advantage, in this modern era, is gained by compiling bash without readline. The speed and space advantage does not accrue as much as it once did, especially if you link readline against a system shared library. So before you propose work for me to do whose purpose is to protect gentoo from the consequences of its decision, it would help to explore the reasons for that decision. If those reasons have been lost to time, maybe it's time to reevaluate. As it happens, I have submitted a patch that replaces the use of "compgen -A function" with a "declare -F" parser, which is all well and good. The thing is that portage also has a legitimate stake in needing to enumerate all variable names. To that end, I had been using "compgen -A variable" in my fork. The implication of the bug is that I must refrain from using it, at which point the prospect of parsing "declare -p" is a tremendously unappealing one, to say the least. The output of declare -p is intended for the shell to parse -- so it can be reused as input. That's always going to be its primary purpose. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: maybe a bug in bash?
On Fri, Jun 30, 2023 at 11:17:06PM +0900, Dominique Martinet wrote:
> Sebastian Luhnburg wrote on Fri, Jun 30, 2023 at 03:47:57PM +0200:
> > p.s.: in the final script, it is only one SSH:
> >
> > ssh user@machine << EOF
> > /bin/bash -c "do something with the password"
> > EOF
I wish you would remove the layers of obfuscation here, and just state
clearly what you're doing.
Why are you running /bin/bash inside the script which is already
being executed by the remote user's shell? Is this is a stand-in for
some other program? Is it just a bad design? We can't tell!
> then as Greg suggested pass password to bash as argument instead;
> assuming password has been quoted once as previously:
> ssh user@machine << EOF
> bash -c 'echo \$1' -- $password
> EOF
I never suggested embedding the password inside an unquoted here document.
My suggestion was more like this:
ssh user@machine bash -s "${password@Q}" <<'EOF'
echo "$1"
EOF
Or if the remote user's shell isn't bash, then use "sh quoting" instead of
bash's @Q quoting.
q=\' b=\\
ssh user@machine bash -s "'${password//$q/$q$b$q$q}'" <<'EOF'
echo "$1"
EOF
In all cases, the here document is quoted ('EOF' instead of EOF) so no
substitutions occur within it. It's passed to the remote system as
written. The password is passed as an argument to the command which is
executed by sshd -- in this case, bash, which will interpret the
script passed in by stdin, and which now has the password as its "$1"
parameter.
Therefore, you can use "$1" inside the script (which is inside the quoted
here document) to refer to the password.
Everything I've written here is boilerplate. You can take it exactly as
I've written it, and replace the echo "$1" with your actual commands.
Telling us more about the setup you're working with (is the remote user's
shell sh or bash, for example) would help us simplify the advice we
give, by removing unneeded "but if..." scenarios.
Re: maybe a bug in bash?
Sebastian Luhnburg wrote on Fri, Jun 30, 2023 at 03:47:57PM +0200:
> /bin/bash -c "echo 'password in subshell in here document: ' ${password@Q}"
${password@Q} is still within double quotes in your here-document here;
these quotes are breaking the escaping you used.
This would be out of quotes:
bash < p.s.: in the final script, it is only one SSH:
>
> ssh user@machine << EOF
> /bin/bash -c "do something with the password"
> EOF
then as Greg suggested pass password to bash as argument instead;
assuming password has been quoted once as previously:
ssh user@machine << EOF
bash -c 'echo \$1' -- $password
EOF
(note once again it should not be quoted, printf %q or ${@Q} did that)
And... if that is all there is to it ssh already runs things in a shell,
so you don't need bash -c at all and can just use $password directly,
quoted exactly once.
--
Dominique
Re: maybe a bug in bash?
On Fri, Jun 30, 2023 at 03:49:23PM +0200, Sebastian Luhnburg wrote:
> You suggest to use an argument, to pass the password. Please correct me if I
> be wrong, but to write one or more password(s) in clear text as an argument
> in the terminal (./myscript.sh password1 password2) is not the best practice
> (the Bash history save the last x commands). To get the input via an
> password manager (like Bitwarden CLI) is at this point the better way I
> think (code injection stay be a problem, if I use the passwords in an
> awkward way in my script).
I described a way for your *script* to send the password to the remote
system, for use by whatever remote command needs it. How your script
gets that password in the first place is a separate question.
In one model, you could simply prompt the user for it. Then the user
can retrieve the password from your password manager, and paste it.
#!/bin/bash
read -rsp 'Mysql password: ' pass
echo
ssh remoteuser@remotehost bash -s "${pass@Q}" <<'EOF'
echo 'select something from sometable ...' |
mysql -p"$1" mydatabase
EOF
Here, the (shell-quoted) password is passed as an argument to ssh.
Ssh concatenates "bash", "-s" and the password argument together to
form a command, which is passes to the remote host for execution.
This is one of the ways that you can send a password to a remote system
over an ssh connection, without fear of mangling it, or causing an
unwanted code injection. It's the way that I would choose if you are
in fact feeding this password to some arbitrary program (such as mysql)
on the remote host.
If you are using the password for ssh to create the connection in the
first place, then everything changes. I've already stated my opinions
about that in my previous email, and I won't cover it here.
Re: maybe a bug in bash?
Hello Greg,
thank you for your quick answer, too!
First, I am not a Bash mastermind or have deeper knowledge about secure
programming. Your suggestion about the risk of coding injection is very
interesting! But I will read the full text after my holidays, which will
begin after this E-Mail :)
But I will explain what I want to do and why I think it is a good way
(it may be a little bit off topic for this mailing list).
First, in my LPIC-1 course the lecturer tell me it is better (not
binding) to deny SSH login for root users (especially for the user with
the name root). The reason is simple: decrease the attack surface. Yes,
a secure password needs a lot of time to be cracked via brute force, but
if the attacker did not know the username, which is needed to login, the
attacker must get two things. For my opinion, the decrease the attack
surface is a good approach.
Why I did not use SSH keys instead of SSH password login? We use a
self-hosted password manager (precise Bitwarden). So, we have a central
collection point of credentials and sensitive data. Every server have a
unique password. If I use SSH keys, it is a decentral approach. Every
user must manage his keys, which allows to connect to the servers. E.g.
if the user change his computer, the WSL instance etc pp, all keys must
secure separately from decentralized places. Yes, an SSH jump host could
be a solution, but this is a future project. At the moment, I think the
use of a password manager is an easier way to do this.
You suggest to use an argument, to pass the password. Please correct me
if I be wrong, but to write one or more password(s) in clear text as an
argument in the terminal (./myscript.sh password1 password2) is not the
best practice (the Bash history save the last x commands). To get the
input via an password manager (like Bitwarden CLI) is at this point the
better way I think (code injection stay be a problem, if I use the
passwords in an awkward way in my script).
Mit freundlichen Grüßen
Sebastian Luhnburg
IT
--
swp software systems GmbH & Co. KG
Königsbrücker Straße 124
01099 Dresden
Tel: 0351-492850
Fax: 0351-4928550
www:https://www.vi-bim.de
Kennen Sie schon unsere FAQ-Wissensdatenbank? Einfach hier
klicken:https://faq.vi-bim.de
Unsere Datenschutzerklärung finden Sie unterhttps://datenschutz.vi-bim.de
Registergericht: Amtsgericht Dresden HRA 3008
persönlich haftender Gesellschafter:
swp Beteiligungs GmbH
Registergericht: Amtsgericht Dresden HRB 15 20 9
Geschäftsführer: Holger Schönemann, Stefan Urlberger
Am 29.06.23 um 15:42 schrieb Greg Wooledge:
On Thu, Jun 29, 2023 at 11:33:15AM +0200, Sebastian Luhnburg wrote:
I want to use this password in an here document, which inject some
commands into a SSH-Connection (for simplification, I use the bash instead
of ssh in the test script below).
Skipping the "is it a bug or not" part, which has already been addressed
as a mis-quoting issue, let's talk about the goal.
First of all, is this the password *of* the ssh connection itself? If
so, that can be addressed by switching to key-based authentication (and
either using a key with no passphrase on it, or an ssh-agent to hold
your passphrase).
Second, are you trying to do something like this:
sshnonr...@some.host 'sudo some command'
where the password is for "sudo" because you aren't logging in directly
as root? The answer to this is so blastedly obvious that I shouldn't
have to state it, but nonetheless, here we are: just ssh in as root
instead of nonroot + sudo.
Some folks will scream that this is a bad idea, horrible practice, can't
do it, etc. These folks are idiots. Ssh can be configured to allow
root logins only when using key authentication. That's as secure as you
could ask for. Certainly it's at *least* as secure as throwing a password
around and using sudo and invoking layers of quoting hell.
But let's say it's not either of these things. Maybe it's a database
access password or something, and you're actually doing something like:
sshnonr...@some.host 'mysql -p mydatabase ...'
The obvious play here is to send the password on stdin. But you've
introduced another layer of complication, because you're actually doing
something like:
sshnonr...@some.host bash <<-EOF
some command
another command
mysql -p"$injected_password" mydatabase
EOF
where stdin is already tied up sending the script for bash to execute,
and therefore can't be used to send the database password. This is NOT
going to work.
What you want to do instead is pass the password as an *argument* to
bash -s. You've already got the @Q (or printf %q) part, which is a
necessary step. You just need the rest of it.
sshnonr...@some.host bash -s "${password@Q}" <<-'EOF'
some command
another command
mysql -p"$1" mydatabase
EOF
This is also described in a bit more detail at
Re: maybe a bug in bash?
Thank you very much for your quick answer!
I am not sure what do you mean with "open the quote"? I try to copy your
example, but my copy did not work. I wrote this:
#!/usr/bin/env bash
initial_password='$abc&xyz'
echo "initial password: " $initial_password
among_password="${initial_password@Q}"
echo "among password: " $among_password
password="${among_password@Q}"
echo "end password: " $password
bash << EOF
echo "password in here document: " ${password@Q}
/bin/bash -c "echo 'password in subshell in here document: ' ${password@Q}"
EOF
But I got this:
initial password: $abc&xyz
among password: '$abc&xyz'
end password: ''\''$abc&xyz'\''' # single quotes!
password in here document: ''\''$abc&xyz'\'''
password in subshell in here document: ''\''&xyz'\'''
p.s.: in the final script, it is only one SSH:
ssh user@machine << EOF
/bin/bash -c "do something with the password"
EOF
Mit freundlichen Grüßen
Sebastian Luhnburg
IT
--
swp software systems GmbH & Co. KG
Königsbrücker Straße 124
01099 Dresden
Tel: 0351-492850
Fax: 0351-4928550
www:https://www.vi-bim.de
Kennen Sie schon unsere FAQ-Wissensdatenbank? Einfach hier
klicken:https://faq.vi-bim.de
Unsere Datenschutzerklärung finden Sie unterhttps://datenschutz.vi-bim.de
Registergericht: Amtsgericht Dresden HRA 3008
persönlich haftender Gesellschafter:
swp Beteiligungs GmbH
Registergericht: Amtsgericht Dresden HRB 15 20 9
Geschäftsführer: Holger Schönemann, Stefan Urlberger
Am 29.06.23 um 13:00 schrieb Dominique Martinet:
Sebastian Luhnburg wrote on Thu, Jun 29, 2023 at 11:55:12AM +0200:
initial_password="\$abc&xyz"
echo "initial password: " $initial_password
printf -v password '%q' $initial_password
echo "initial password with escaped characters: " $password
bash << EOF
echo "password in here document: " ${password@Q}
/bin/bash -c "echo 'password in subshell in here document: ' ${password@Q}"
password here is \$abc\&xyz, and the @Q escape just makes
${password@Q} '\$abc\&xyz' (adds single quotes).
What running it in a subshell (ssh, or sh/bash -c) changes here is the
order in which things are unescaped.
Note the here-doc substitutes ${password@Q}, so this is the same as
running a file with the content:
```
echo "password in here document: " '\$abc\&xyz'
/bin/bash -c "echo 'password in subshell in here document: ' '\$abc\&xyz'"
```
Like this, you can see that in the former case, the echo command gets
the password in single quotes, while the subshell's echo is first
interpreted in double-quotess.
In double-quotes, \$ is simplified to $ (as the lack of backslash would
just get you the variable)
Note that since the here-doc first expands the variable, you cannot
solve this by using single quotes: the variable would just end the
quoting!
... At which point all you need to do is just to open the quote,
e.g. the following:
a='$foo&bar'
b="${a@Q}"
c="${b@Q}"
ssh localhost <
OpenPGP_0x1E7D455B730DAD17.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
Re: [PATCH] sleep builtin signal handling
On Fri, Jun 30, 2023 at 08:53:16AM +0200, Phi Debian wrote: > Well > > ┌─none:/home/phi > └─PW$ type sleep > sleep is hashed (/usr/bin/sleep) > > So sleep is not a builtin here. This patch is for the sleep loadable builtin example distributed with bash. You can activate it using enable -f "$(pkg-config --variable=loadablesdir bash)"/sleep sleep On most linux distributions, it is installed with bash, on debian you need to install the bash-builtins package to have it. o/ emanuele6
