Re: Potential Bash Script Vulnerability
if you wanted this for your script - read all then start semantics, as
opposed to read-as-you-execute - would it work to rewrite yourself inside a
function?
function main() { ... } ; main
On Sun, Apr 7, 2024, 22:58 Robert Elz wrote:
> Date:Mon, 8 Apr 2024 02:50:29 +0100
> From:Kerin Millar
> Message-ID: <20240408025029.e7585f2f52fe510d2a686...@plushkava.net>
>
> | which is to read scripts in their entirety before trying to execute
> | the resulting program. To go about it that way is not typical of sh
> | implementations, for whatever reason.
>
> Shells interpret their input in much the same way, regardless of
> from where it comes. Would you really want your login shell to
> just collect commands that you type (possibly objecting to those
> with syntax errors) but executing none of them (including "exit")
> until you log out (send EOF) ?
>
> kre
>
>
>
>
Re: Light weight support for JSON
interfacing with an external tool absolutely seems like the correct answer
to me. a fact worth mentioning to back that up is that `jq` exists. billed
as a sed/awk for json, it fills all the functions you'd expect such an
external tool to have and many many more. interfacing from curl to jq to
bash is something i do on a near daily basis.
https://stedolan.github.io/jq/
On Sun, Aug 28, 2022, 09:25 Yair Lenga wrote:
> Hi,
>
> Over the last few years, JSON data becomes a integral part of processing.
> In many cases, I find myself having to automate tasks that require
> inspection of JSON response, and in few cases, construction of JSON. So
> far, I've taken one of two approaches:
> * For simple parsing, using 'jq' to extract elements of the JSON
> * For more complex tasks, switching to python or Javascript.
>
> Wanted to get feedback about the following "extensions" to bash that will
> make it easier to work with simple JSON object. To emphasize, the goal is
> NOT to "compete" with Python/Javascript (and other full scale language) -
> just to make it easier to build bash scripts that cover the very common use
> case of submitting REST requests with curl (checking results, etc), and to
> perform simple processing of JSON files.
>
> Proposal:
> * Minimal - Lightweight "json parser" that will convert JSON files to bash
> associative array (see below)
> * Convert bash associative array to JSON
>
> To the extent possible, prefer to borrow from jsonpath syntax.
>
> Parsing JSON into an associative array.
>
> Consider the following, showing all possible JSON values (boolean, number,
> string, object and array).
> {
> "b": false,
> "n": 10.2,
> "s: "foobar",
> x: null,
> "o" : { "n": 10.2, "s: "xyz" },
> "a": [
> { "n": 10.2, "s: "abc", x: false },
> { "n": 10.2, "s": "def" x: true},
> ],
> }
>
> This should be converted into the following array:
>
> -
>
> # Top level
> [_length] = 6# Number of keys in object/array
> [_keys] = b n s x o a# Direct keys
> [b] = false
> [n] = 10.2
> [s] = foobar
> [x] = null
>
> # This is object 'o'
> [o._length] = 2
> [o._keys] = n s
> [o.n] = 10.2
> [o.s] = xyz
>
> # Array 'a'
> [a._count] = 2 # Number of elements in array
>
> # Element a[0] (object)
> [a.0._length] = 3
> [a.0._keys] = n s x
> [a.0.n] = 10.2
> [a.0.s] = abc
> [a.0_x] = false
>
> -
>
> I hope that example above is sufficient. There are few other items that are
> worth exploring - e.g., how to store the type (specifically, separate the
> quoted strings vs value so that "5.2" is different than 5.2, and "null" is
> different from null.
>
> I will leave the second part to a different post, once I have some
> feedback. I have some prototype that i've written in python - POC - that
> make it possible to write things like
>
> declare -a foo
> curl http://www.api.com/weather/US/10013 | readjson foo
>
> printf "temperature(F) : %.1f Wind(MPH)=%d" ${foo[temp_f]}, ${foo[wind]}
>
> Yair
>
Re: Prefer non-gender specific pronouns
Léa, I see that in the section Ilkka quoted you were using it in the
plural. However Ilkka is exactly right; despite "they" being technically
plural, using it for somebody of undetermined gender has been in the
mainstream since long before inclusive language. "Someone left *their*
book, there's no name and I don't know who to call."
The AP and Chicago style guides, hardly reckless proponents of any
progressive vanguard, endorse this usage, though they recommend working
around it if possible ("Somebody left *a* book"). However they do
unequivocally endorse using it for somebody who declares "they" to be their
pronoun (though for now that may not have much bearing on the manual).
On Sun, Jun 6, 2021, 07:49 Léa Gris wrote:
> Le 06/06/2021 à 11:33, Ilkka Virta écrivait :
> > In fact, that generic 'they' is so common and accepted, that you just
> used
> > it yourself
> > in the part I quoted above.
>
> Either you're acting in bad faith, or you're so confused by your
> gender-neutral delusion that you don't remember that in normal people's
> grammar, "they" is a plural pronoun.
>
> --
> Léa Gris
>
>
>
Re: Prefer non-gender specific pronouns
I can see a couple reasons why it would be a good thing, and in the con column only "I personally don't have time to go through the manual and make these changes". but I'd happily upvote a patch from somebody that does. On Sat, Jun 5, 2021, 09:24 Vipul Kumar wrote: > Hi, > > Isn't it a good idea to prefer non-gender specific pronoun (like "their" > instead of "his") at following places in the reference manual? > > > leaving the user in a non-writable directory other than his home > directory after login, [1] > > > his home directory. The name of this file is taken from the value of > the shell variable [2] > > [1]: > > https://www.gnu.org/software/bash/manual/html_node/The-Restricted-Shell.html#The-Restricted-Shell > [2]: > > https://www.gnu.org/software/bash/manual/html_node/Readline-Init-File.html#Readline-Init-File > > > Cheers, > Vipul >
Re: local failure
I think the underlying question here is not exactly "how do I gather this from the docs" as much as it is "how was I supposed to know about this and act on it before I had to debug it?" The bash manual is always "adequate" in the sense that almost any question can be answered by carefully consulting it -- but that doesn't help the person who either is unpracticed at the occasionally arcane art of *reading* the bash manual, or who doesn't even know they need to ask a question (let alone how to articulate it). I've been consulting the bash manual for nearly a decade but I still find myself in this position every now and then. That's why I want to plug linting your shell scripts, preferably in your text editor. You'll catch a lot of problems before they happen and just get better at shell scripting. My preferred tool, ShellCheck ( https://github.com/koalaman/shellcheck), alerted me to this problem a couple of years ago, and lots, lots, lots more. It incorporates into Vim easily via the plugin "Syntastic"; I'm sure it's easy to incorporate for other text editors as well. On Sun, May 31, 2020 at 12:37 PM Robert Elz wrote: > Date:Sun, 31 May 2020 10:29:27 +0100 > From:Laurent Picquet > Message-ID: 0wg33x5paj...@mail.gmail.com> > > | This behaviour is not fully documented > > Nothing is ever "fully" documented, as it is always possible to > write even more text about anything at all - and if that were done > then it is clear that what existed before was not "full". > > So, instead of "fully documented" let's use "adequately documented" > and for that it is. > > But you do need to understand how things fit together to follow. > > First, the exit status is only ever set as the result of executing > a command - and only a command in the current execution environment. > >? Expands to the exit status of the most recently executed > foreground pipeline. > > [elsewhere it is noted that a simple command is a degenerate pipeline, > so is a compound command for that matter.] > > "local" is such a command, and like all such commands, its exit status > is documented... > > The return status is 0 unless local is used outside a > function, an invalid name is supplied, or name is a readonly > variable. > > (that text has been previously quoted in this thread). > > Command substitutions execute in a different shell execution environment > (nothing that happens in one has any effect upon the shell that contains > them - except that stdout from the command substitution is inserted into > the current command line (and then potentially subject to field splitting, > pathname expansion, ...) -- it becomes a part of some other command. > > There is one exception to the case that only commands in the current > execution environment ever set the exit status, which is documented in > bash(1) as ... > >If there is a command name left after expansion, execution proceeds > as >described below. Otherwise, the command exits. If one of the >expansions contained a command substitution, the exit status of the >command is the exit status of the last command substitution > performed. >If there were no command substitutions, the command exits with a > status >of zero. > > None that neither redirects nor variable assignments are commands (but > almost everything else that you would write in a sh script is, including > "local"). The "as described below" involves locating the command, running > it, and obtaining its exit status - and is what is done when there is a > command. > > When there isn't - which is what happens when all there was was > variable assignments &/or redirects - then, and only then, does the: > > If one of the expansions contained a command substitution, the > exit status of the command is the exit status of the last command > substitution performed. > > This is the only way that the exit status of a command substitution can > ever be observed in the parent shell. But you have to read the entire > man page to observe that nowhere else is the status of a command > substitution > ever reported. > > So, the doc is all there, and it is possible to work out what happens. > > As above, it is always possible to write more - but write too much, and > instead of a manual page, what would be left would be a tutorial, or > perhaps > even a book (but there's already one, perhaps more than one, of those). > > There is certainly nothing special about the local command in this context, > any command where there happens to be a command substitution on the command > line somewhere works exactly the same way, so it would be wrong to make any > special note of this in the doc for "local" without also putting the same > extra info in the descriptions of every other command. > > kre > > >
[PATCH] builtin_read: count null character toward -n/-N limit
When the read builtin is invoked with -n/-N , the documentation
specifies that at most characters will be read from stdin. This
statement is not true when stdin emits null characters: read discards
the null character and keeps reading without incrementing its counter,
continuing until it has consumed non-null characters. This
leads to infinite running time when the input stream has infinite null
characters and not enough non-null characters: for example, `read -N 1
maximum.
It also updates the documentation to describe explicitly how the read
builtin behaves when it encounters null characters - both the fact that
it discards them, and that the null characters (as of this patch) will
count toward the -n/-N maximum.
---
Notes:
I earlier tried to submit a bug report using bashbug but I am not sure if it
came through. Sending as a patch now. Sorry for the duplicate if it is that.
If I included a patch in the bug report, this one is the same except that it
also updates documentation, so I suggest using this instead.
builtins/read.def | 3 ++-
doc/bash.1| 3 +++
tests/read.right | 1 +
tests/read3.sub | 6 ++
4 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/builtins/read.def b/builtins/read.def
index b57c8c398e18..5713766e4c3a 100644
--- a/builtins/read.def
+++ b/builtins/read.def
@@ -672,7 +672,7 @@ read_builtin (list)
break;
if (c == '\0' && delim != '\0')
- continue; /* skip NUL bytes in input */
+ goto increment_byte_count; /* skip NUL bytes in input */
if ((skip_ctlesc == 0 && c == CTLESC) || (skip_ctlnul == 0 && c ==
CTLNUL))
{
@@ -713,6 +713,7 @@ add_char:
}
#endif
+increment_byte_count:
nr++;
if (nchars > 0 && nr >= nchars)
diff --git a/doc/bash.1 b/doc/bash.1
index e6cd08db3867..f1a72e1ae4cf 100644
--- a/doc/bash.1
+++ b/doc/bash.1
@@ -9138,6 +9138,7 @@ are used to split the line into words using the same
rules the shell
uses for expansion (described above under \fBWord Splitting\fP).
The backslash character (\fB\e\fP) may be used to remove any special
meaning for the next character read and for line continuation.
+Null characters are discarded.
Options, if supplied, have the following meanings:
.RS
.PD 0
@@ -9178,6 +9179,7 @@ buffer before editing begins.
\fBread\fP returns after reading \fInchars\fP characters rather than
waiting for a complete line of input, but honors a delimiter if fewer
than \fInchars\fP characters are read before the delimiter.
+Null characters count toward this limit.
.TP
.B \-N \fInchars\fP
\fBread\fP returns after reading exactly \fInchars\fP characters rather
@@ -9189,6 +9191,7 @@ not treated specially and do not cause \fBread\fP to
return until
The result is not split on the characters in \fBIFS\fP; the intent is
that the variable is assigned exactly the characters read
(with the exception of backslash; see the \fB\-r\fP option below).
+Null characters count toward this limit.
.TP
.B \-p \fIprompt\fP
Display \fIprompt\fP on standard error, without a
diff --git a/tests/read.right b/tests/read.right
index 73cb7042fbca..839c532f7c67 100644
--- a/tests/read.right
+++ b/tests/read.right
@@ -45,6 +45,7 @@ abcde
abc
ab
abc
+ac
#
while read -u 3 var
do
diff --git a/tests/read3.sub b/tests/read3.sub
index af41e3f27930..02334fdad6d2 100644
--- a/tests/read3.sub
+++ b/tests/read3.sub
@@ -20,5 +20,11 @@ echo abc | {
echo $foo
}
+# does not consume too many characters
+echo abcde | tr b '\0' | {
+ read -N 3 foo
+ echo $foo
+}
+
read -n 1 < $0
echo "$REPLY"
--
2.20.1
read -N 1
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin17.5.0
Compiler: clang
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='darwin17.5.0' -DCONF_MACHTYPE='x86_64-apple-darwin17.5.0'
-DCONF_VENDOR='apple' -DLOCALEDIR='/usr/local/Cellar/bash/4.4.23/share/locale'
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DMACOSX -I. -I. -I./include
-I./lib -I./lib/intl -I/private/tmp/bash-20180602-817-1lkxzgr/bash-4.4/lib/intl
-DSSH_SOURCE_BASHRC -Wno-parentheses -Wno-format-security
uname output: Darwin pop-jpas-mbpr-4.local 17.7.0 Darwin Kernel Version 17.7.0:
Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin17.5.0
Bash Version: 4.4
Patch Level: 23
Release Status: release
Description:
The `read` builtin skips null characters and does not count
them toward the limit specified by -n/-N. This leads to
surprising behavior with /dev/zero: read -N 1 0 && nr >= nchars)
diff --git a/tests/read.right b/tests/read.right
index 73cb7042fbca..649c78a568c5 100644
--- a/tests/read.right
+++ b/tests/read.right
@@ -45,6 +45,7 @@ abcde
abc
ab
abc
+ac
#
while read -u 3 var
do
diff --git a/tests/read3.sub b/tests/read3.sub
index af41e3f27930..02334fdad6d2 100644
--- a/tests/read3.sub
+++ b/tests/read3.sub
@@ -20,5 +20,11 @@ echo abc | {
echo $foo
}
+# does not consume too many characters
+echo abcde | tr b '\0' | {
+ read -N 3 foo
+ echo $foo
+}
+
read -n 1 < $0
echo "$REPLY"
Verbose error output in interactive mode
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: darwin15.0.0 Compiler: clang Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='darwin15.0.0' -DCONF_MACHTYPE='x86_64-apple-darwin15.0.0' -DCONF_VENDOR='apple' -DLOCALEDIR='/usr/local/Cellar/bash/4.3.42/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DMACOSX -I. -I. -I./include -I./lib -I./lib/intl -I/private/tmp/bash20150913-75517-1rfmxak/bash-4.3/lib/intl -DSSH_SOURCE_BASHRC uname output: Darwin john-zeconomy.local 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64 Machine Type: x86_64-apple-darwin15.0.0 Bash Version: 4.3 Patch Level: 42 Release Status: release Description: In an interactive session, when attempting file completion via when a certain type of ill-formed command substitution is present, bash gives verbose error output relating to "unexpected end of file". Naturally you can work around this by never making this mistake when using command substitution, but it seems to me bash can probably guard for this and avoid disruptive error messages. Repeat-By: In an interactive session, type the following: $ echo "$(echo " ~/ (Note the missing close-parenthesis.) With the cursor at the end of this line (i.e. after the "/",), enter a tab. Bash should prompt with a listing of the given directory. Instead, the following is shown: $ echo "$(echo " ~/-bash: command substitution: line 87: unexpected EOF while looking for matching `"' -bash: command substitution: line 88: syntax error: unexpected end of file -bash: command substitution: line 87: unexpected EOF while looking for matching `"' -bash: command substitution: line 88: syntax error: unexpected end of file -bash: command substitution: line 87: unexpected EOF while looking for matching `"' -bash: command substitution: line 88: syntax error: unexpected end of file -bash: command substitution: line 87: unexpected EOF while looking for matching `"' -bash: command substitution: line 88: syntax error: unexpected end of file -bash: command substitution: line 87: unexpected EOF while looking for matching `"' -bash: command substitution: line 88: syntax error: unexpected end of file I apologize that I'm unable to offer a solution.
