Re: array subscripts act differently for integers(ie. let)
> > > also this variant does the same: > > $ (('ar[$idbad2]+=11')) > Because (( and let are essentially equivalent. I think the more important question isn't "why does (( behave this way?", but rather "should (( behave this way?". It's probably not reasonable to expect the author to know and take into account that (( arr[$key] )) treats key's data as bash code. Really, the behaviour of ((' arr[$key] ')) makes more sense as a default behaviour. The former is broken (as the expectation that $key expands data is not met).
printf and the $ modifier
man 3 printf describes the following: o An optional field, consisting of a decimal digit string followed by a $, specifying the next argument to access. If this field is not provided, the argument following the last argument accessed will be used. Arguments are numbered starting at 1. If unaccessed arguments in the format string are interspersed with ones that are accessed the results will be indeterminate. This is a useful feature, allowing you to decouple the order of your arguments with from the format. This is useful for localization and a few other cases. In this example, I'd like to use it to simplify a regex replace statement: # s/(.*)foo(.*)/$2bar$1/ [[ 123foo567 =~ (.*)foo(.*) ]] && printf '%2$sbar%1$s' "${BASH_REMATCH[@]:1}" Is there a particular reason why bash's built-in printf does not support this format modifier? Does bash re-implement printf or does it use the Standard C Library's printf? (If the former; why?) -- *Maarten Billemont* (lhunath) me: http://www.lhunath.com – business: http://www.lyndir.com – http://masterpasswordapp.com
Inconsistent behaviour of +=() and existing array keys
So, referring to man bash, this is the description of +=: When += is applied to an array variable using compound assignment (see Arrays below), the variable's value is not unset (as it is when using =), and new values are appended to the array beginning at one greater than the array's maximum index (for indexed arrays) or added as additional key-value pairs in an associative array. This description does not help us understand what should happen when this syntax explicitly mentions keys and more specifically, ones that already exist within the array. Intuition would likely have us believe that since +=() mutates the array by adding the elements to it, consistency would imply that when you specify elements by key, the array would either gain an element for that key or change the existing element at that key. So if we were to run this code: declare -a indexed_array=( [0]=a ) declare -A associative_array=( ["x"]=a ) indexed_array+=( [0]=b [1]=c ) associative_array+=( ["x"]=b ["y"]=c ) declare -p indexed_array associative_array Here's what I would expect to see: declare -a indexed_array='([0]="b" [1]="c")' declare -A associative_array='([x]="a" [y]="c" )' Instead, we get this: declare -a indexed_array='([0]="ab" [1]="c")' declare -A associative_array='([x]="ab" [y]="c" )' So the current behaviour appears to change the meaning of +=() as either mutating the array or mutating the array elements, depending on whether the specified key already exists within the array. I believe this behaviour to not only be counter-intuitive but also dangerous: the same syntax now has two distinct meanings depending solely on what data already exists within the array. Even more interestingly, it is apparently also legal to use the += operator within the element assignment syntax: indexed_array+=( [0]+=b [1]+=c ) This appears to behave exactly as a normal = would there. What I would expect is for += inside +=() to behave as = does now, and = to behave as it does outside of +=(), which is to "set" the value, not append. Ergo: declare -a indexed_array=( [0]=a ) indexed_array=( [0]=b ) #=> [0]=b -- because we unset the array and set the element so those given. indexed_array+=( [0]=b ) #=> [0]=b -- because we mutate the array by setting the element to that given. indexed_array=( [0]+=b ) #=> [0]=b -- because we unset the array and set the element by appending the given to nothing. indexed_array+=( [0]+=b ) #=> [0]=bb -- because we mutate the array by appending the given to what is already there. -- *Maarten Billemont* (lhunath) me: http://www.lhunath.com – business: http://www.lyndir.com – http://masterpasswordapp.com
Re: Arithmetic + array allows for code injection
On Jun 2, 2014, at 9:34 AM, Greg Wooledge wrote: > On Mon, Jun 02, 2014 at 03:08:17PM +0200, Andreas Schwab wrote: >> Greg Wooledge writes: >> >>> imadev:~$ : $((a[$x])) >>> bash: Mon Jun 2 08:06:39 EDT 2014: syntax error in expression (error token >>> is "Jun 2 08:06:39 EDT 2014") >>> >>> There's the code-injection problem that started the thread. >> >> Here the index is '$(date)'. >> >> *Note (bash) Arithmetic Expansion:: ... All tokens in the expression >> undergo parameter and variable expansion, command substitution, and >> quote removal. The result is treated as the arithmetic expression to be >> evaluated. > > Ah. And this is copied almost verbatim from POSIX: > > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04 > > $((expression)) > > The expression shall be treated as if it were in double-quotes, except that a > double-quote inside the expression is not treated specially. The shell shall > expand all tokens in the expression for parameter expansion, command > substitution, and quote removal. > > So the reason it acts this way is because POSIX said so. Now it starts > to make sense! > > (One could argue that POSIX's wording doesn't require the command > substitution be done in a second pass AFTER the parameter expansion. > But apparently it has been interpreted this way.) As such, the bug is that the expansions triggered by $(( )) are IMPLICITLY re-evaluated by [ ]. To summarize, index='$( echo >&2 foo )' # Literal shell code should never be evaluated unless an 'eval' is involved. echo ${array[ $index ]} # [] expands $index, results in a literal that [] does not re-evaluate. echo $(( $index )) # (( )) expands $index, results in a literal that (( )) does not re-evaluate. echo $(( array[ $index ] )) # (( )) expands $index, results in a literal that [] DOES re-evaluate. Whether or not the documentation justifies the above behaviour, I think we can agree on the fact that no user will expect or even desire the behaviour of the latter, never mind that it is dangerous. As such, it certainly is a bug and should be corrected. I think it's fair to trust, as an author, that shell code will only be re-evaluated when there's an explicit eval, source, or similar statement declaring such behaviour. As a side note, expanding literal words as variables such as happens in ${array[ index ]} or the above code with index=foo is probably acceptable so long as the value that results from this expansion is treated purely as data (though bash does allow recursion on the expansion of literal variable names here). The point being that in my opinion, bash should as a matter of principle prohibit concealed code execution wherever possible; failing that it will be nearly impossible to write safe shell code - or at least trust that your shell code is safe. On Jun 2, 2014, at 10:28 AM, Andreas Schwab wrote: > If you want to write robust scripts, don't use shell. > > Andreas. What does this comment justify? Other than stopping all maintenance on bash today.
Re: Bug when trapping exit from subshell
the exit handler is called, you just muted it. remove the redirections, and you'll also notice you need a space after your ! On 16 May 2014 12:41, Mark Ferrell wrote: > The following script properly reports the exit code to the calling > environment, , but the exit handler is not called if a function > triggers the exit vs an external command. > > Script executed via: > bash
EXIT traps in interactive shells
Trapping on EXIT is a really convenient way of cleaning up after yourself when your script ends. The nice thing about it is that you don't need to handle all sorts of signals, handle them, and re-throw them. It makes for a single clean point to define your exit handlers. Unfortunately, I've noticed that EXIT handlers don't play so nice in interactive shells. This is of particular interest when you background a subshell from them: { trap 'rm tempfile' EXIT dowork tempfile } & ... kill $! This works fine from a script (non-interactive) but not when performed from the prompt. Here's some demonstrational code: interactive: ~ $ { trap 'echo exiting' EXIT; sleep 5; } & sleep 1; kill $! [1] 59061 [1]+ Terminated: 15 { trap 'echo exiting' EXIT; sleep 5; } non-interactive: ~ $ bash -c '{ trap "echo exiting" EXIT; sleep 5; }' & sleep 1; kill $! [1] 59168 exiting [1]+ Terminated: 15 bash -c '{ trap "echo exiting" EXIT; sleep 5; }' So, why is this, is this a bug, and we have this fixed? I like being able to rely on my EXIT handler. Cheers, Maarten. smime.p7s Description: S/MIME cryptographic signature
Re: Severe Bash Bug with Arrays
This mail has gone slightly off-topic. Hit mark as read if you don't care for that sort of thing. On 27 Apr 2012, at 21:02, Linda Walsh wrote: > Greg Wooledge wrote: > >> NEVER use eval plus a brace expansion to generate a list. That's just >> freakin' evil. And unsafe: > > But I _like_ evil sometimes! >:-> I don't care if you like evil in your sexual play or in your vendettas. When you write code, evil is nothing worth liking. In programming, evil is defined as something that is stupid and not thought through. Doing evil has harmful side effects. It's not cool to be evil, it's short-sighted. > But reality is things like that save me time in typing in stuff or writing > maint scripts on my machine -- they aren't meant for handling user input. > > They take hand editing to gen the paths I want and don't take user input, > so it's a specialized usage in that case It's fine to do short-sighted hacks on the prompt, where you know your input well and will notice breakage so that you can recover. It's not fine to do said things in scripts that run on unpredictable input and where you don't always notice breakage until it's caused irreparable harm. I don't really care what lame justifications you come up with to try and justify broken system code to yourself, just don't try to convince us with them. > Maarten Billemont wrote: > >> On 26 Apr 2012, at 01:18, Linda Walsh wrote: >>> Ishtar:> echo "${b[*]}" >> Please note that expanding array elements using [*] is usually NOT what >> anyone wants. > > It was exactly what I wanted for my example to work. Please don't suggest > non-working solutions to the example code I provided. ${b[@]} Will NOT > work as a replacement for [*]. It isn't not BETTER, it is DIFFERENT. > That's why there are both. I didn't say never to use "${b[*]}". I said it's usually not what anyone wants. It's mostly only useful for merging arrays into a single string such as for user display of said array. I explained that. You are correct, of course, there are two syntaxes, they both do something different (when quoted), and that's fine. They each have their use case. What I *did* say, is never to use ${b[*]}. And it also so happens that ${b[@]} is not different from ${b[*]}. And I stick with that point. Never use unquoted expansion of arrays, not with [*], not with [@]. >> your array like above, you then follow up by re-splitting the whole thing >> using standard wordsplitting, not to mention pathname expanding the >> resulting words. > > I followed up? I don't understand this. When you leave ${b[*]} unquoted, you expand each element of the array, then ask bash to split the whole thing, and then pathname expand the result. Not only did you just throw away any distinction between the different array elements, you also made an entirely new distinction (whitespace), and then you're asking bash to search the filesystems for any pathnames that match each of the words. Seriously, nobody, ever, wants, this. If they did, they'd have used a string instead. And even then I question the sanity behind it. Word-splitting is useful in a primitive POSIX shell that has no concept like arrays etc. to compensate. In bash, it is the major cause of bugs, and best avoided. > I would assume they'd have the > intelligence to know when to use * or @ and I wouldn't have to talk down to > them > and cover basics. I can't help notice the slight irony in my having to explain to you the necessity of quotes with array expansion. >> You should always recommend the "${b[@]}" variant. > > I shouldn't ALWAYS do anything. It's dangerous. > You can do so for your examples... It won't work in the next > two any more than the last one of the previous... What? Anyway, think about this: An array holds elements in a way that each is separated from the other. To put these elements in an array is to want each element to be distinct from the others. Therefore, the default way of expanding these elements should preserve that distinction. Only in rare cases is the actual INTENT of the author to MERGE these elements into a single unit, a single string, thus throwing away the distinction between the elements. And in THOSE cases and those cases ALONE should the author use "${array[*]}". So yes, always recommend "${b[@]}", unless you know for a fact that the author is looking to merge the elements. > sort an arg list: > > > args="-dpi -xinerama -clipboard -unixkill -nowinkill -ac +bs -fp > > -multiwindow" > > > readarray -t dflts< <( a=( $args ); IFS=
Re: Is it possible or RFE to expand ranges of *arrays*
On 26 Apr 2012, at 06:30, John Kearney wrote: > Am 26.04.2012 06:26, schrieb Linda Walsh: >> I know I can get >> a="abcdef" echo "${a[2:4]}" = cde >> >> how do I do: >> typeset -a a=(apple berry cherry date); then get: >> >> echo ${a[1:2]} = "berry" "cherry" ( non-grouped args) >> >> I tried to do it in a function and hurt myself. >> >> >> > echo ${a[@]:1:2} > I see little reason to ask bash to wordsplit the elements after expanding them. You ought to quote that expansion. smime.p7s Description: S/MIME cryptographic signature
Re: Severe Bash Bug with Arrays
On 26 Apr 2012, at 01:18, Linda Walsh wrote: > > Ishtar:> echo "${b[*]}" Please note that expanding array elements using [*] is usually NOT what anyone wants. Be careful about recommending it to anyone. "${b[*]}" # This throws away any usefulness of the array by merging all the array elements into one single string (argument), concatenating the elements using the first character of IFS. ${b[*]} # Never do this. On top of throwing away the usefulness of your array like above, you then follow up by re-splitting the whole thing using standard wordsplitting, not to mention pathname expanding the resulting words. "${b[@]}" # Expands each array element as a separate argument, protecting it from wordsplitting and pathname expansion mutilation. ${b[@]} # Identical to and just as broken as ${b[*]}. Never do this. You should always recommend the "${b[@]}" variant. "${b[*]}" is rarely useful in the event that your intent is to merge the array into a single string, eg. for displaying elements to a user: dirs=(*/) (IFS=,; echo "The dirs are: ${dirs[*]}") But anyway, that's an aside from this topic, which has very little to do with this. smime.p7s Description: S/MIME cryptographic signature
Re: how are aliases exported?
On 16 Apr 2012, at 11:43, Linda Walsh wrote: > > > But it won't work for files below the top level of the library directory. > > if I have include system/errno.shh, Source is crippled to not look in PATH. When there's a slash in the file, it stops searching PATH. So the reason you're not doing source /lib/system/errno.sh is because you want it to search multiple possible lib dirs for a system subdir? smime.p7s Description: S/MIME cryptographic signature
Re: how are aliases exported?
On 16 Apr 2012, at 09:54, Maarten Billemont wrote: > On 15 Apr 2012, at 22:52, Linda Walsh wrote: >> >> But I want the syntax >> >> include file.shh >> >> to just 'work', i.e. first time, it would call my include file, which defines >> the function... > > I'm sorry, tell me again why you don't just write "source file.sh" instead of > having to invent your own "include file.sh"? I just realized, you appear to be looking for a way to define a "source path" to search for the file. PATH=/foo:/bar:/etc source file or PATH=$SOURCEPATH source file I know it's not as "clean" as "include file", but it's certainly more clear as to what it does. And clarity is more important, IMO. Also, it conveniently also doesn't require insane hacks. smime.p7s Description: S/MIME cryptographic signature
Re: how are aliases exported?
On 15 Apr 2012, at 22:52, Linda Walsh wrote: > > But I want the syntax > > include file.shh > > to just 'work', i.e. first time, it would call my include file, which defines > the function... I'm sorry, tell me again why you don't just write "source file.sh" instead of having to invent your own "include file.sh"? If you really want to create custom syntax, write and enable your own built-in. As a side note, you should probably have a slash in there somewhere unless you intend for bash to search PATH for the file. smime.p7s Description: S/MIME cryptographic signature
Re: inconsistency with "readonly" and scope
On 15 Apr 2012, at 03:21, Chet Ramey wrote: > > If I declare a variable readonly at global scope, I wouldn't expect it > to be overridden by some local namespace pollution. I think that's fine; in the local context, your variable has a different meaning; it's another variable with the same name. Outside of the local scope, the original readonly variable remains unmodified.
Re: Exit status of "if" statement?
On 10 Apr 2012, at 08:26, Barry Margolin wrote: > > if [ $status = 0 ] As an aside, I don't think it's a good idea to recommend comparing numbers using string comparison operators. if (( status == 0 )) But if you wanted to treat them as strings for the sake of code simplicity, just go the case route: case $status in 0) ... ;; 1) ... ;; *) ... ;; esac smime.p7s Description: S/MIME cryptographic signature
Re: status on $[arith] for eval arith vsl $((arith))??
On 09 Apr 2012, at 11:46, Linda Walsh wrote: > > Like it costs how much? I would **bet** that it cost more code to support > (()) than to support [] as arith ops .. so if you want my opinion, lets toss > (())... > (like that'll happen)... Just thought I'd add that I personally prefer $((...)) mostly because of consistency with other syntax. $ is always an indicator of expansion, and since we have (( ... )) to perform an arithmetic evaluation (on its own as a statement or in a for etc.), I'm very happy that putting a $ in front of that is all it takes to expand the result of it. Just like putting a dollar in front of a subshell construct expands the output of it and putting a dollar in front of a parameter name expands the contents of it. If we were to move to $[ ... ], then to keep that consistency, we'd have to change (( ... )) to [ ... ], and it so happens that this command is already taken for a different type of test entirely. (Don't give me the spiel about how [...] is already arithmetic evaluation inside array indices, that's a different syntax entirely, and perfectly fine - unless you'd prefer to turn it into arr[ (( ... )) ], which is rather overkill.) smime.p7s Description: S/MIME cryptographic signature
Re: semicolon at beginning of line
On 10 Apr 2012, at 06:03, Elliott Forney wrote: > Here is another example that appears to defy my expectations. In this > case, the semicolon is allowed: > > sine:~$ hello='echo hello' > sine:~$ world='echo world' > sine:~$ ${hello};${world} > hello > world > sine:~$ unset hello > sine:~$ ${hello};${world} > world > sine:~$ unset world > sine:~$ ${hello};${world} Those are not empty statements. It's important to understand the difference between: $var and: eval "$var" People should stop trying to execute code by parameter expansion, and specifically stop thinking that parameter-expanded words are evaluated as bash code. > > Thanks! > Elliott Forney > > On Mon, Apr 9, 2012 at 10:02 PM, Elliott Forney > wrote: >> Sure, a comment can be used to place a line in your history but that >> doesn't really address the examples I had. Just seems to me like a >> lone semicolon could be treated as a newline/noop. I can't seem to >> think of anything that this would break but, of course, that doesn't >> mean it wouldn't. The end of a case in a switch statement is >> certainly an interesting one, hadn't thought of that, but it should be >> possible to handle that by checking for ;; as a token before ;. OK, so you're saying, let's change bash so that an empty statement becomes a noop statement. Except for when that empty statement is preceded by a semicolon and happens to have no whitespace, because then it could be a case delimiter. Frankly, what are you hoping to gain from this? This will just introduce new rules with new exceptions and inconsistencies. If it were possible to do a blanket rule: empty statements before a semicolon are noops, I might be OK with it, but if it requires adding additional addendums to the rule, "oh wait, except for this and that case", my vote is out. >> >> I might mention that ksh, zsh and tcsh all allow lines to begin with a >> semicolon. zsh even allows "; ; ; echo hello world ; ; ;" although >> ksh only allows a single ; at the beginning of a line. >> >> On Sun, Apr 8, 2012 at 6:48 PM, Joseph Fredette wrote: >>> Could also use a #, no? >>> >>> On Sun, Apr 8, 2012 at 8:46 PM, Steven W. Orr wrote: >>> On 4/7/2012 4:00 PM, Elliott Forney wrote: > I wish bash would happily execute lines that begin with a semicolon, > i.e., treat it as a no-op followed by a command. The following > examples come to mind: > > $ infloop& echo hello > [2] 11361 > hello > $ infloop&; echo hello > bash: syntax error near unexpected token `;' > > $ echo hello; echo world > hello > world > $ echo hello;; echo world > bash: syntax error near unexpected token `;;' > > $ ; echo hello world > bash: syntax error near unexpected token `;' > > Any thoughts? > > Thanks, > Elliott Forney > Just use a colon. : echo Hello world. I use it all the time to 'park' a command in my history. Then when I'm ready, I just back up to it and remove the colon. -- Time flies like the wind. Fruit flies like a banana. Stranger things have .0. happened but none stranger than this. Does your driver's license say Organ ..0 Donor?Black holes are where God divided by zero. Listen to me! We are all- 000 individuals! What if this weren't a hypothetical question? steveo at syslang.net > smime.p7s Description: S/MIME cryptographic signature
Re: status on $[arith] for eval arith vsl $((arith))??
On 08 Apr 2012, at 21:30, Chet Ramey wrote: > On 4/8/12 3:02 PM, Maarten Billemont wrote: > >> Any particular reason for not removing old undocumented functionality, or is >> that mostly the nature of this beast - dragging along and maintaining >> ancient code for the sake of compatibility?= > > Because, as Linda discovered, there is still working code out there using > it. Maybe we'll get to a point where it's all gone, but we're not there > yet. IMO, the working code out there that relies on $[...] either runs on older versions of bash, or if the sysadmin decided to upgrade bash, he can assume the responsibility to fix the code. I suppose a deprecation route would make such a scenario least painful. Though I don't think bash has ever taken this approach yet (removal of features), it will be unexpected and people will likely complain. In the end one must decide between a lean code base and keeping the 1% happy to upgrade. smime.p7s Description: S/MIME cryptographic signature
Re: status on $[arith] for eval arith vsl $((arith))??
On 08 Apr 2012, at 01:47, Chet Ramey wrote: > On 4/7/12 4:45 PM, Linda Walsh wrote: >> >> >> >> In modifying some released code on my distro,I ran into the extensive use >> of $[arith] as a means for returning arithmetic evaluations of the >> expression. >> >> I vaguely remember something like that from years ago, but never see any >> reference to >> it -- yet it works, and old code seems to rely on it -- and >> "$[(1+2)/3]" looks cleaner than "$(((1+2)/3))". So what's up with that? > > It dates from Posix circa 1990 (1003.2d9, of which I've lost my paper > copy). I implemented it after the Berkeley guys, mostly Marc > Teitelbaum, put it into Posix. It ended up getting dropped in favor > of the ksh $((...)) expansion, at which point everyone deprecated the > old $[...]. I removed it from the manual sometime later, but it still > works as it always has. > > Chet Any particular reason for not removing old undocumented functionality, or is that mostly the nature of this beast - dragging along and maintaining ancient code for the sake of compatibility? smime.p7s Description: S/MIME cryptographic signature
Re: Inconsistent quote and escape handling in substitution part of parameter expansions.
On 28 Feb 2012, at 23:23, Chet Ramey wrote: > On 2/28/12 5:18 PM, John Kearney wrote: >> On 02/28/2012 11:07 PM, Chet Ramey wrote: >>> On 2/28/12 4:28 PM, John Kearney wrote: On 02/28/2012 10:05 PM, Chet Ramey wrote: > On 2/28/12 12:26 PM, John Kearney wrote: > >> But that isn't how it behaves. "${test//str/""}" >> >> because str is replaced with '""' as such it is treating >> the double quotes as string literals. >> >> however at the same time these literal double quotes >> escape/quote a single quote between them. As such they are >> treated both as literals and as quotes as such >> inconsistently. > > I don't have a lot of time today, but I'm going to try and > answer bits and pieces of this discussion. > > Yes, bash opens a new `quoting context' (for lack of a better > term) inside ${}. Posix used to require it, though after > lively discussion it turned into "well, we said that but it's > clearly not what we meant." > > There are a couple of places in the currently-published version > of the standard, minus any corregendia, that specify this. The > description of ${parameter} reads, in part, > > "The matching closing brace shall be determined by counting > brace levels, skipping over enclosed quoted strings, and > command substitutions." > > The section on double quotes reads, in part: > > "Within the string of characters from an enclosed "${" to the > matching '}', an even number of unescaped double-quotes or > single-quotes, if any, shall occur." > > Chet yhea but I think the point is that the current behavior is useless. there is no case where I want a " to be printed and start a double quoted string? and thats the current behavior. Not so important how you treat it just need to pick 1. then you can at least work with it. Now you have to use a temp variable. as a side note ksh93 is pretty good, intuitive ksh93 -c 'test=teststrtest ; echo "${test//str/"dd dd"}"' testdd ddtest ksh93 -c '( test=teststrtest ; echo ${test//str/"dd '\''dd"} )' testdd 'ddtest >>> >>> The real question is whether or not you do quote removal on the >>> stuff inside the braces when they're enclosed in double quotes. >>> Double quotes usually inhibit quote removal. >>> >>> The Posix "solution" to this is to require quote removal if a >>> quote character (backslash, single quote, double quote) is used to >>> escape or quote another character. Somewhere I have the reference >>> to the Austin group discussion on this. >>> >> >> 1${A:-B}2 >> >> Logically for consistancy having double quotes at position 1 and 2 >> should have no effect on how you treat string B. > > Maybe, but that's not how things work in practice. Should the following > expansions output the same thing? What should they output? > > bar=abc > echo ${foo:-'$bar'} > echo "${foo:-'$bar'}" > > Chet Personally, I'd prefer it to behave in the same way bash deals with this sort of thing in other contexts. Eg. echo $(echo '$bar') echo "$(echo '$bar')" Both obviously result in <$bar> getting output. For the sake of consistency with the rest of bash, I would expect the same from your example. smime.p7s Description: S/MIME cryptographic signature
Re: Inconsistent quote and escape handling in substitution part of parameter expansions.
On 28 Feb 2012, at 18:52, John Kearney wrote: > On 02/28/2012 06:43 PM, Dan Douglas wrote: >> On Tuesday, February 28, 2012 06:38:22 PM John Kearney wrote: >>> On 02/28/2012 06:31 PM, Dan Douglas wrote: On Tuesday, February 28, 2012 05:53:32 PM Roman Rakus wrote: > On 02/28/2012 05:49 PM, Greg Wooledge wrote: >> On Tue, Feb 28, 2012 at 05:36:47PM +0100, Roman Rakus >> wrote: >>> And that means, there isn't way to substitute "something" >>> to ' (single quote) when you want to not perform word >>> splitting. I would consider it as a bug. >> >> imadev:~$ q=\' imadev:~$ input="foosomethingbar" imadev:~$ >> echo "${input//something/$q}" foo'bar > > I meant without temporary variable. > > RR ormaaj@ormaajbox ~ $ ( x=abc; echo ${x/b/$'\''} ) a'c >>> >>> ( x=abc; echo "${x/b/$'\''}" ) -bash: bad substitution: no >>> closing `}' in "${x/b/'}" >>> >>> >>> you forgot the double quotes ;) >>> >>> >>> I really did spend like an hour or 2 one day trying to figure it >>> out and gave up. >> >> Hm good catch. Thought there might be a new quoting context over >> there. > I think we can all agree its inconsistent, just not so sure we care?? > i.e. we know workarounds that aren't so bad variables etc. So essentially, we all agree it's buggy, but since we can work around the buggy behavior and since POSIX doesn't *require* it not be buggy, let's leave it be buggy? How about making it behave properly instead? And with properly, I mostly mean, make the quotes and backslashes behave as they do now inside, for example, a process substitution instead of a parameter expansion. smime.p7s Description: S/MIME cryptographic signature
Re: Another mapfile question.
On 05 Aug 2011, at 16:32, Steven W. Orr wrote: > On 8/5/2011 9:08 AM, Maarten Billemont wrote: >>> IFS= >>> aa=() >>> while read line >>> do >>> aa+=("$line") >>> done< fn >> >> You should really put that IFS= on your 'read', so as not to break the >> default wordsplitting for the rest of your script: >> > > For purposes of giving concise examples on this email list, I wrote it as > above. In practice, something like the above construct that modifies IFS > would be responsible for restoring IFS. I'm very much against changing the standard operation of the shell in a broader scope than where you need its effect. old_IFS=$IFS IFS=$'\n' while read line do stuff done < <(fn) IFS=$old_IFS The point here is to change IFS for 'read'. Unfortunately, you're also changing it for all of 'stuff'. Your scope is too broad, and as a result, stuff's behavior is non-standard. Making IFS local-scoped is really the same thing, still too broad. You wouldn't notice until you did something with IFS in 'stuff', and actually did notice the bug in that. It's dangerous. Do this: while IFS= read line do stuff done < <(fn) It's much cleaner and it's more correct. Same goes for set -f hackery. I'm always against it, because there's no way to cleanly scope it, it changes the way the shell works substantially, and there's usually constructs that express the logic more correctly. smime.p7s Description: S/MIME cryptographic signature
Re: Another mapfile question.
> IFS= > aa=() > while read line > do > aa+=("$line") > done < fn You should really put that IFS= on your 'read', so as not to break the default wordsplitting for the rest of your script: while IFS= read -r line > vs. > IFS=$'\n' > aa=($(< fn)) Don't leave expansions unquoted! you're still permitting pathname expansion here. IFS=$'\n' read -rd '' -a aa < <(fn) > vs. > mapfile -t aa < fn When compared to the above read, this will also read empty lines. The read operation will merge multiple newlines into a single delimiter (as it does for all types of whitespace in IFS). The while read works fine but is verbose. The mapfile is a bit more concise. The read -a is fine and concise so long as you can live with the lack of empty lines. I doubt mapfile is much use to you that while read doesn't already give you. smime.p7s Description: S/MIME cryptographic signature
Process substitution file consumed?
In the following snippet, I expect the two ls' to give the same output. However, it appears the process substitution's temporary file sometimes vanishes for the second. Can someone explain why and whether this is the desired behavior or not? A Linux (2.6.27.6) VM in a QEMU says this: # moo() { ls -l "$1"; ls -l "$1"; }; moo >(true) l-wx-- 1 root root 64 Aug 2 14:00 /dev/fd/63 -> pipe:[245] ls: cannot access /dev/fd/63: No such file or directory A Linux (2.6.32-5-amd64) machine says this: # moo() { ls -l "$1"; ls -l "$1"; }; moo >(true) l-wx-- 1 devel devel 1 Aug 2 16:03 /dev/fd/63 -> pipe:[527968] l-wx-- 1 devel devel 1 Aug 2 16:03 /dev/fd/63 -> pipe:[527968] My Mac says this: # moo() { ls -l "$1"; ls -l "$1"; }; moo >(true) prw-rw 0 lhunath staff 0 2 Aug 15:55 /dev/fd/63| prw-rw 0 lhunath staff 0 2 Aug 15:55 /dev/fd/63| And the latter two break again when I toss a (true) inbetween the two ls': # moo() { ls -al "$1"; (true); ls -al "$1"; }; moo >(true) l-wx-- 1 devel devel 1 Aug 2 16:06 /dev/fd/63 -> pipe:[528022] ls: cannot access /dev/fd/63: No such file or directory # moo() { ls -al "$1"; (true); ls -al "$1"; }; moo >(true) prw-rw 0 lhunath staff 0 2 Aug 16:07 /dev/fd/63| ls: /dev/fd/63: Bad file descriptor Surely neither case where the FD can no longer be accessed or its file has disappeared can be correct behavior? smime.p7s Description: S/MIME cryptographic signature
Re: Negative indexes in ${arr[@]:off:length}
On 30/06/11 14:15, Greg Wooledge wrote: > On Wed, Jun 29, 2011 at 06:41:25PM +0200, Maarten Billemont wrote: >> For those to whom a cursor between elements seems confusing: think of your prompt's cursor and imagine an element is a single character. > > No wonder this makes no sense to me. Have you actually looked at the > cursor in a terminal? I use rxvt, but xterm is the same: the cursor is > a black[1] rectangle that *overlies* one of the characters, not a skinny > little wedge/line that sits *between* characters. > > The only time I've ever seen a cursor that is *between* characters is > in GUI stuff, like a textarea in Firefox. But for people like me, > that's the oddball case, not the norm. I certainly wouldn't expect a > programming language to work like this. > > [1] Or white, if you are one of those people who use a dark background. > Or possibly amber or green, if you're *really* old school. My cursor is also a block, but perhaps you use vim. Write abcabc in a vim document, then select the second a. You move your cursor so that the block is on the a or the line is in front of it. You go into visual mode and go forward by one. ${arr[@]:0:1}. Now instead of going forward, go backward one (${arr[@]:0:-1}) and you'll have selected the c. It really doesn't matter whether your cursor is a block or a line. To select elements from a set, you need a starting point and a direction, and IMO, pretending that ${arr[@]:0:1} should be the same thing as ${arr[@]:0:-1} is by far the more confusing way to think about arrays.
Re: Negative indexes in ${arr[@]:off:length}
On 29 Jun 2011, at 15:01, Greg Wooledge wrote: > > On Wed, Jun 29, 2011 at 01:42:06PM +0200, Maarten Billemont wrote: >> If you expand ${arr[@]:1:2}, you get the following: >> values: [ a | b | c ] >> indexes: 0 1 2 3 >> expand: [ 1 2 ] => b c >>^ start from 1 >> ^ length 2 >> >> I propose we let a negative length iterate backward, so with ${arr[@]:1:-2}, >> you get the following: >> values: [ a | b | c ] >> indexes: 0 1 2 3 >> expand:1 ] [ 2 => a c >>^ start from 1 >> ^ length 2 Firstly, I see my indenting of the "^ [label]" got whacked off target. Let me try to correct that, if my MUA allows: ${arr[@]:1:2} values: [ a | b | c ] indexes: 0 1 2 3 expand: [ 1 2 ] => b c ^ start from 1 ^ length 2 ${arr[@]:1:-2} values: [ a | b | c ] indexes: 0 1 2 3 expand:1 ] [ 2 => a c ^ start from 1 ^ length 2 > But if you're starting with element 1, why wouldn't it be "b a"? This is where preceding the element with the index comes in handy: We start with the index 1, which lays between element 0 and element 1. We go back 1, that gives us element 0 and we're now between element 0 and the last element of the array (assuming a circular view on the array). Go back one more and you now also get the last element in the array, and the cursor ends up between the last and the before last element. For those to whom a cursor between elements seems confusing: think of your prompt's cursor and imagine an element is a single character. Type this: abcabc Put your cursor before the second "a", imagine you're now at index 0: abc|abc Index 0 refers to element "a", because when you select 1 character (${arr[@]:0:1}), that'll be "a": abc[a]bc. Select 2 elements (${arr[@]:0:2}): abc[ab]c. Now try going backwards with your cursor to select 2 elements, instead of forwards (${arr[@]:0:-2}): a[bc]abc My example was ${arr[@]:1:-2}, so that would look like this: ab[ca]bc
Re: Negative indexes in ${arr[@]:off:length}
On 29 Jun 2011, at 14:05, Mart Frauenlob wrote: > > On 29.06.2011 13:42, Maarten Billemont wrote: >> On 27 Jun 2011, at 16:25, Chet Ramey wrote: >>> > [...] >> Exactly, let's draw the array in the example: >> >> arr=(a b c) >> values: [ a | b | c ] >> indexes: 0 1 2 3 >> > [...] > > 4 members? 3 members. The index cursor precedes its member, as is commonly done when depicting arrays. Apologies for mentioning the index 3 and somehow confusing you. Note that I prefixed the line with the word "indexes", not "values". I think we all know value a has the index 0 and value c has the index 2.
Re: Negative indexes in ${arr[@]:off:length}
> Since ${arr[-1]} already does exactly this, I figure it would merely add to > the syntax' consistency. Consider the fact that ${arr[x]} is in fact the equivalent of ${arr[@]:x:1}.
Re: Negative indexes in ${arr[@]:off:length}
On 27 Jun 2011, at 16:25, Chet Ramey wrote: > >> I don't even understand what the second one is supposed to mean at >> all -- the :1: means to start with "b" and the -2 means to go back 2 >> elements...? How do you derive "a c" from any possible interpretation >> of this? > > I assume that he wants to be able to treat an indexed array as a circular > buffer without having to do any of the work in a script. Exactly, let's draw the array in the example: arr=(a b c) values: [ a | b | c ] indexes: 0 1 2 3 If you expand ${arr[@]:1:2}, you get the following: values: [ a | b | c ] indexes: 0 1 2 3 expand: [ 1 2 ] => b c ^ start from 1 ^ length 2 I propose we let a negative length iterate backward, so with ${arr[@]:1:-2}, you get the following: values: [ a | b | c ] indexes: 0 1 2 3 expand:1 ] [ 2 => a c ^ start from 1 ^ length 2 Since ${arr[-1]} already does exactly this, I figure it would merely add to the syntax' consistency.
Re: Feature Request - Allow mapfile to handle NUL-delimited data
On 13 May 2011, at 20:07, DJ Mills wrote: > A -0 option, or even an option to use a different field separator, would be > an > excellent feature for mapfile. Currently, of course, you can use: > > while IFS= read -rd '' file; do array+=("$file"); done > > mapfile is considerably faster, however, and I think this would be quite > handy. seconded. I'm not sure why it only works for lines. Sure, on the shell data is often line-based, but more often than that, line-based parsing is bad and should be done with a safe delimiter rather than \n instead.
Re: BASH_SUBSHELL documentation misleading
On 23 Mar 2011, at 21:28, Chet Ramey wrote: > > OK. What wording would you like to see? I don't mind the wording he proposed: On 23 Mar 2011, at 17:12, Sam Liddicott wrote: >maybe it should say > >BASH_SUBSHELL > Incremented by one in each nested subshell or subshell > evironment. > It is always 0 when $BASH_PID=$$ > >
Re: BASH_SUBSHELL documentation misleading
On 23 Mar 2011, at 21:15, Sam Liddicott wrote: > > On 23/03/11 18:52, Chris F.A. Johnson wrote: >> >> On Wed, 23 Mar 2011, Sam Liddicott wrote: >> >>> >>> Configuration Information [Automatically generated, do not change]: >>> Machine: i686 >>> OS: linux-gnu >>> Compiler: gcc >>> Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' >>> -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' >>> -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DP$ uname output: Linux >>> sojo 2.6.35-28-generic-pae #49-Ubuntu SMP Tue Mar 1 14:58:06 UTC 2011 i686 >>> GNU/Linux >>> Machine Type: i686-pc-linux-gnu >>> >>> Bash Version: 4.1 >>> Patch Level: 5 >>> Release Status: release >>> >>> Description: >>> man page says: >>> >>> BASH_SUBSHELL >>> Incremented by one each time a subshell or subshell environment >>> is spawned. The initial value is 0. >>> >>> This suggests that: >>> >>> echo $BASH_SUBSHELL ; ( echo ) ; echo $BASH_SUBSHELL >>> >>> would not give the same answer for BASH_SUBSHELL >> >> No, it suggests that: >> >> echo $BASH_SUBSHELL ; ( echo $BASH_SUBSHELL ) >> >> would not give the same answer for BASH_SUBSHELL > > It should suggest that, because that is how it actually works. > > But the man page is misleading and does not actually suggest this. > >> >> In your example, the second "echo $BASH_SUBSHELL" is at the same >> depth as the first. >> > > Yes. But a new subshell environment has been spawned. Each time that happens > BASH_SUBSHELL should increase. > > Of course I know how it does work, but the man page isn't clear. It doesn't > say that the increase is only visible within the subshell and therefore it is > a measurement of subshell depth. > > Sam > I must concur. In the bug example at the second expansion of BASH_SUBSHELL occurs after "a subshell or subshell environment is spawned". The documentation does not specify that BASH_SUBSHELL is actually unmodified in the running shell and merely copied to the subshell and incremented there. What's more, the fact that this is not mentioned, grammatically suggests that BASH_SUBSHELL is modified, which is not actually the case.
Re: variable name and its' value are the same characters causes recursion error
On 10 Mar 2011, at 15:23, Chet Ramey wrote: > > On 3/10/11 8:14 AM, Greg Wooledge wrote: >> On Thu, Mar 10, 2011 at 10:18:26AM +0800, Clark J. Wang wrote: >>> Actually I don't like the recursion here. Does POSIX require that? >>> For example: unset a; declare a="a"; [[ a -lt 3 ]]; echo $? bash: [[: a: expression recursion level exceeded (error token is "a") 1 >> >> POSIX doesn't even have a [[ command. This is all bash. > > Not really. There is substantial agreement among shells that implement > arithmetic expansion. bash, ksh93, zsh, mksh (and other pdksh derivatives > that implement `[[') all behave the same way. For the most part, it's the > same way with `['; zsh is a notable exception there. > > Chet Personally, I would much rather see (( a )) fail if a doesn't contain a number rather than go search for a parameter named by its contents. If the parameter a contains a word that's not a number, I can't imagine any case where this would be an expected and wanted scenario, rather than a bug. If it were expected, the author would've used the indirection operator directly. Which leaves us with a bug, one that's often terribly hard to detect and diagnose. That's ignoring the fact that you're leaving the door wide open for user input to go and load any parameter it chooses in its stead. Really, whenever this happens, it's either breaking things in completely unexpected and often invisible ways or it's somebody exploiting your code to do something it wasn't supposed to or reveal something it doesn't want to show. Why would we want this "feature"?
Re: Why escape char `:' with `\' when auto completing filenames?
On 17 Feb 2011, at 23:02, Chet Ramey wrote: > >>> >>> "Clark J. Wang" writes: >>> I think char `:' is not special in bash. >>> >>> $ printf "%q\n" "$COMP_WORDBREAKS" >>> $' \t\n"\'><=;|&(:' >>> >>> >> I don't think that explain the issue. > > /* characters that need to be quoted when appearing in filenames. */ > rl_filename_quote_characters = " \t\n\\\"'@<>=;|&()#$`?*[!:{~"; > /*}*/ > > So that's problem. I don't think @=: need to be treated specially. Any reason? >>> >>> They are used as word break characters during completion. >>> >>> >> That's the way it's implemented but that does not mean that's reasonable. > > For pete's sake. If you don't think they should be word break characters, > modify the value of COMP_WORDBREAKS. For the record, @ causes a word break > so you can complete hostnames more easily, = breaks so you can complete > filenames on the rhs of assignment statements, and : breaks so you can > complete filenames in words that look like $PATH. > > Are we really spending this much time on a cosmetic issue? Why are we escaping all word break characters? rm file:name and rm file\:name are effectively identical, I'm not sure I see the need for escaping it.
Re: leading 0 inconsistence?
On 12 Feb 2011, at 11:57, Ralf Goertz wrote: > > Hi, > > I am wondering what the reasoning might be for this seeming > inconsistence. > >> i=08 >> if [ $i -lt 9 ] ; then echo ok; fi > ok >> if [ $((i)) -lt 9 ] ; then echo ok; fi > bash: 08: value too great for base (error token is "08") > > Why is 08 not tried to be interpreted as octal when *numerically* compared > using test? > > I stand by my general recommendation of always using (( when performing arithmetic tests, [[ when testing strings and files, and never [, unless your shebang is not actually bash. It will save you from a great many pitfalls and inconsistencies.
Re: Can someone explain this?
On 12 Feb 2011, at 09:28, Andreas Schwab wrote: > > Bob Proulx writes: > >> Dennis Williamson wrote: >>> Yes, do your quoting like this: >>> ssh localhost 'bash -c "cd /tmp; pwd"' >> >> I am a big fan of piping the script to the remote shell. >> >> $ echo "cd /tmp && pwd" | ssh example.com bash >> /tmp > > Even better: > > $ ssh example.com bash <<\EOF > cd /tmp && pwd > EOF > > That avoids having to watch out for ' vs " quoting. > > Andreas. The trouble with using stdin is that it becomes much harder to pass user data. If it's simple strings, one might be tempted to expand them instead: ssh example.com bash <
Re: Don't show help of `readonly' and `readarray' when I run `help read'
On 10 Feb 2011, at 15:21, Chet Ramey wrote: > > On 2/10/11 4:03 AM, Clark J. Wang wrote: >> help: help [-dms] [pattern ...] >> >> From my understanding the *pattern* here must be a glob-style pattern >> (wildcard) so `readonly' does not match the pattern `read'. > > The pattern is composed of the same characters as a glob pattern, but > it's treated more like 'grep ^pattern topic' if it doesn't contain any > special pattern matching characters. > > Kind of like the following: > > $ printf "%s\n" read readonly readarray | grep ^read /dev/stdin > read > readonly > readarray > > > Chet I must admit I personally dislike getting three pages of help output I don't care about when doing `help read`. If I wanted to learn about `readonly`, I'd do `help readonly`. I'm not sure the current behavior has any real merits over treating the pattern like glob pattern matching usually works (anchored to beginning and end).
Re: How can i specify the scripting language used for parsing a script?
On 05 Feb 2011, at 17:09, Andreas Schwab wrote: > > Maarten Billemont writes: > >> The comment is called a hashbang or shebang. It tells the kernel which >> program to start. Your script is passed over stdin to the interpreter. > > No, it isn't, it's passed as the argument. > > Andreas. Of course, otherwise ./myscript < file wouldn't work.
Re: How can i specify the scripting language used for parsing a script?
On 05 Feb 2011, at 13:57, ali hagigat wrote: > > #!/bin/sh > echo ppp > echo $SHELL > exit 2200 > > In the above script i tried to specify /bin/sh as my parser by a > comment. Is that OK? When I type ./scr2 , i want bash recognize > /bin/sh as the parser of ./scr2. > The comment is called a hashbang or shebang. It tells the kernel which program to start. Your script is passed over stdin to the interpreter. Note that on many system /bin/sh is a symlink to bash, so interpreting your code with /bin/sh would just start the bash interpreter in POSIXLY_CORRECT mode.
Re: Why sh does not return a false value?
On 05 Feb 2011, at 13:47, ali hagigat wrote: > > if (sh -c exit 34) then echo p;fi > p > The following condition should be false, because our exit value is > non-zero. but 'if' considers the condition as true and executes 'echo' > command. Why? > You are giving -c the argument 'exit' and setting "34" as the zero'th argument to the script. It's vital to understand what word splitting is. Quotes are used to keep words together so they are passed as a single argument. -c takes only one single argument, so to pass the command exit 34 to -c, you quote it.
Re: printf "%q" and $'...'
On 25 Nov 2009, at 14:58, Antonio Macchi wrote: > my goal is very very stupid, like this > > $ printf "%q" $( > to get a "ascii form" of a binary file (something like uuencode) > > > but, as you can see, it does not work only for two binary chars > > 0x00, and 0x0a That doesn't sound like a goal; more like an approach to achieve some other goal. 1. Remember to quote your expansions: "$(..)" 2. You can't put binaries in arguments for the very reason you just encountered: Arguments cannot hold NUL bytes. If you state the true goal (why are you trying to convert binary files into some sort of undefined form?) perhaps a more appropriate suggestion can be made. printf doesn't read from a stream, so you can't use it to %q-encode your binary data directly. Though %q-encoding binary data sounds like an odd requirement. You did mention uuencode so I assume you considered that already as well as base64 encoding and found it insufficient. Which leaves me wondering what you could possibly be trying to do.
Re: printf "%q" and $'...'
You can "output" $'\x00' just fine: $ printf '\0' Note that -d $'\x00' is the same thing as -d '', for the reason I mentioned earlier. The argument to the -d option that "read" takes is a C-string. Understand the difference between *strings* and *streams*. A stream (standard output of printf '\0' or find . -print0) can contain any byte. A C-string can not contain NUL bytes. If you want strings with NUL bytes, you need Pascal-strings. You can *not*, however, output a NUL byte by using $'\x00' as an argument. Because arguments can't contain NUL bytes (they are C-strings). So outputting NUL a byte with this will *fail*: $ echo $'\x00' This will *also* fail: $ printf $'\x00' The first example I gave doesn't fail because the argument is not a NUL byte (empty), it is a backslash followed by a zero. printf sees this argument and understands you want it to expand that into a NUL byte, then emits it on its output STREAM. On 25 Nov 2009, at 14:35, Antonio Macchi wrote: > it sounds strange, beacuse > > $ find . -print0 | while read -d $'\x00'; do touch "$REPLY"; done > > works fine. > > > but if I try to "output" $'\x00', I can't. > >
Re: printf "%q" and $'...'
As for NUL out outputting anything in your result, the cause is C-strings. Arguments are C-strings and those are delimited by NUL bytes. Therefore, the NUL byte that you're putting in it is actually marking the end of the string. So the argument ends BEFORE your NUL byte. So it's empty. As or \x0a, that's a newline. And command substitution trims trailing newlines. So a string "[newline]" gets trimmed to "". On 25 Nov 2009, at 08:19, Antonio Macchi wrote: > Hi, I'm using older bash 3.2.39, so please forgiveme if in your newer bash > this issue does not arise. > > > 0x00 and 0x0a has no output in printf "%q" > > $ for i in {0..9} a; do printf "%q\n" "`echo -en \"\x0$i\"`"; done > '' > $'\001' > $'\002' > $'\003' > $'\004' > $'\005' > $'\006' > $'\a' > $'\b' > $'\t' > '' > > > > $'\x00' outputs nothing > > hd <(echo $'\x00') > 0a|.| > 0001 > > > > $'\x01' outputs twice. > > $ hd <(echo -n $'\x01') > 01 01 |..| > 0002 > > > > thanks for your great job > have a nice day > > (God Save Linux!) > > bye > >