Re: bash 5.0 nested array subscript arithmetic expansion error

[Dan Douglas]

On Wed, Oct 17, 2018 at 9:05 AM Chet Ramey  wrote:
> You know this has already been done, right?

I do now! Still trying to get caught up with the changelog.

Re: bash 5.0 nested array subscript arithmetic expansion error

[Dan Douglas]

On Mon, Aug 27, 2018 at 8:12 PM Chet Ramey  wrote:
>
> On 8/27/18 12:25 PM, Grisha Levit wrote:
> > This used to work:
> >
> > bash-4.4$ a=0
> > bash-4.4$ echo $(( a[a[0]] ))
> > 0
> > bash-4.4$ echo ${a[a[a[0]]]}
> > 0

Just curious, did you decide what to do with this?

> This is part of changes in bash-5.0 to avoid expanding array subscripts
> more than one time. After expanding the subscript once, you don't want
> to expand (or check) nested open and close brackets again -- how do you
> get literal brackets into an associative array key, for instance?

By treating associative arrays and ordinary arrays differently.
Associative arrays should be the easier one because any
unescaped/unquoted bracket that isn't part of an expansion is part of
the string.

Integrating arithmetic into the overall Bison/Flex-defined syntax
would be an idealist answer, I guess. Not an easy fix.

Or, have the arithmetic processor stop whenever it encounters var[,
Hand the remainder of the current command back to the shell parser or
length into the expression where the expansion begins to figure out
its extent. Return the length to the end of the expansion plus the
expanded string, seek ahead, resume.

Or, re-use the code already used in contexts processing expansions on
arbitrary strings, such as the argument to case..esac. Arithmetic
doesn't contain tokens that start expansions so there's no need for
the shell to understand it. Just find the unescaped $'s (and
backticks, ugh), expand them, and tell the math processor their
locations and value. Ignoring ] on expanded strings of course.

If it were a temporary bodge I'd say add a shopt option to disable
recursive subscript processing in the event someone depends upon weird
strings in associative arrays. It often isn't a requirement. Too bad
adding this would probably mean supporting it forever.

> I'll take
> a look and see if there's an easy way to allow this syntax, which -- you
> have to admit -- is fairly obscure.

I can live with broken associative arrays a bit longer if it means
time to come up with a better solution.

Re: Array not defined when being set inline

[Dan Douglas]

On 10/10/2017 07:00 AM, shawn wilson wrote:
> I guess that's the right way to describe what I'm seeing:
> 
> [swilson@localhost ~]$ unset f; f=(aaa bbb ccc) declare -p f
> declare -x f="(aaa bbb ccc)"
> [swilson@localhost ~]$ unset f; f=("aaa" "bbb" "ccc") declare -p f
> declare -x f="(aaa bbb ccc)"
> [swilson@localhost ~]$ unset f; f=(aaa bbb ccc)
> [swilson@localhost ~]$ declare -p f
> declare -a f='([0]="aaa" [1]="bbb" [2]="ccc")'
> 
> Is this known? What exactly is going on / what does this mean?
> 

Bash parses the array assignment as though it were valid while reading
words, assignments, and redirects of the command. After processing
expansions, the saved assignment text is re-interpreted as a string
assignment, presumably because at this point bash no longer cares about
the quoting of metacharacters so they're just treated as literal.

Array assignments aren't valid preceding commands of course because
the environment fundamentally stores strings / key-value pairs of
binary blobs.

This should probably be an error, but this is what bash has always done,
and this hasn't really caused any problems other than occasionally
tricking people into thinking arrays are somehow exportable.



signature.asc
Description: OpenPGP digital signature

Re: command_not_found_handle documentation omission

[Dan Douglas]

On 10/08/2017 03:11 PM, Eduardo A. Bustamante López wrote:
> I guess that instead of changing the semantics of
> command_not_found_handle, a new special trap could be added that
> executes in the context of the shell performing the command lookup.

Possible, but magic traps can be ugly. I often end up just calling
a function from one, otherwise there's no locals or positional
params, though you do get to see the params of the calling context as
compensation. A trap would need a new way to pass in the command name
and args.

> Although I'm not sure how valuable it would be (the added complexity).
> Are there any serious uses of the command_not_found_handle aside from
> suggestions during interactive use?

Mostly non-serious uses in bash libraries that implement wacky
features. command_not_found handle is the only "special" function in bash
at the moment. It's reminiscent of discipline functions, which would be
a sensible place to put this. Huge feature to implement though unless
maybe the dynamic variable code can be substantially reused.



signature.asc
Description: OpenPGP digital signature

Re: command_not_found_handle documentation omission

[Dan Douglas]

On 10/08/2017 10:41 AM, Dan Douglas wrote:
> On 10/08/2017 09:47 AM, Chet Ramey wrote:
>> It was originally intended to take the place of the error message that
>> bash prints when it can't find a program to execute. That message was
>> printed by the subshell forked to execute the command, so the message could
>> be redirected (nearly ll shells do it that way). If you're going to run a
>> command, you run it in the same context as the error message.
> 
> Bash does PATH resolution prior to forking any subshell and no fork
> happens when a command isn't found, right? Indeed I would expect error
> messages to be printed by the process that tried resolving the command,
> which I thought is usually not a subshell.
> 
> ~ # strace -fe trace=process bash -c 'foo'
> execve("/bin/bash", ["bash", "-c", "foo"], 0x7ffd42249af0 /* 61 vars */) 
> = 0
> arch_prctl(ARCH_SET_FS, 0x7f6bbde9fe00) = 0
> bash: foo: command not found
> exit_group(127) = ?
> +++ exited with 127 +++
> 
> Any redirections applied to the command ought to still apply to the
> handler with or without the subshell, and a handler can always save
> and restore FDs if it had to redirect output with exec. If the handler
> wants to guarantee output to the tty it pretty much has to use /dev/tty,
> again with or without the subshell.
> 
> Anyway I know this isn't new and there's probably some code out there
> that depends on the implicit subshell by now.
> 

Thinking out loud some more... it does make sense that a user in an
interactive session expects commands to not alter their shell environment,
and a badly written command_not_found_handle could do that, possibly
without the user's knowledge on systems that put a handler in a global
bashrc (likely the most common scenario).

On the other hand a user that actually defines their own handler could
have a million reasons to want to propagate some effect to the interactive
process, e.g. defining an alias or altering PS1. Same for non-interactive
scripts.



signature.asc
Description: OpenPGP digital signature

Re: command_not_found_handle documentation omission

[Dan Douglas]

On 10/08/2017 09:47 AM, Chet Ramey wrote:
> It was originally intended to take the place of the error message that
> bash prints when it can't find a program to execute. That message was
> printed by the subshell forked to execute the command, so the message could
> be redirected (nearly ll shells do it that way). If you're going to run a
> command, you run it in the same context as the error message.

Bash does PATH resolution prior to forking any subshell and no fork
happens when a command isn't found, right? Indeed I would expect error
messages to be printed by the process that tried resolving the command,
which I thought is usually not a subshell.

~ # strace -fe trace=process bash -c 'foo'
execve("/bin/bash", ["bash", "-c", "foo"], 0x7ffd42249af0 /* 61 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7f6bbde9fe00) = 0
bash: foo: command not found
exit_group(127) = ?
+++ exited with 127 +++

Any redirections applied to the command ought to still apply to the
handler with or without the subshell, and a handler can always save
and restore FDs if it had to redirect output with exec. If the handler
wants to guarantee output to the tty it pretty much has to use /dev/tty,
again with or without the subshell.

Anyway I know this isn't new and there's probably some code out there
that depends on the implicit subshell by now.



signature.asc
Description: OpenPGP digital signature

Re: command_not_found_handle documentation omission

[Dan Douglas]

On 10/07/2017 02:53 PM, Martijn Dekker wrote:
> The bash manual and info pages state:
> 
> | If the search is unsuccessful, the shell searches for a
> | defined shell function named 'command_not_found_handle'.  If that
> | function exists, it is invoked with the original command and the
> | original command's arguments as its arguments, and the function's
> | exit status becomes the exit status of the shell.
> 
> This fails to mention that command_not_found_handle() is run in the
> subshell forked to 'exec' the command, so even an explicit 'exit' will
> not exit anything but that subshell. It also means a command handler
> can't do anything that influences the main shell, except send it a
> signal with 'kill'.

Yeah I wish it didn't do that. If I wanted a subshell I'd add one myself.



signature.asc
Description: OpenPGP digital signature

Re: [BUG] Bash segfaults on an infinitely recursive funcion (resend)

[Dan Douglas]

On 10/05/2017 02:29 PM, Dan Douglas wrote:
> ...

Another band-aid might be to build bash with -fsplit-stack. Hardly
worth mentioning as it doesn't fix anything - you just run out of memory
instead of overflowing a fixed-size stack, should someone actually want
that for some reason.



signature.asc
Description: OpenPGP digital signature

Re: [BUG] Bash segfaults on an infinitely recursive funcion (resend)

[Dan Douglas]

On 09/25/2017 01:38 PM, Eric Blake wrote:
> On 09/24/2017 12:53 PM, Shlomi Fish wrote:
> 
>>
>> I see. Well, the general wisdom is that a program should not ever segfault, 
>> but
>> instead gracefully handle the error and exit.
> 
> This is possible by installing a SIGSEGV handler that is able to
> gracefully exit the program when stack overflow is detected (although
> such a handler is EXTREMELY limited in what it is able to safely do); in
> fact, the GNU libsigsegv library helps in this task, and is used by some
> other applications (such as GNU m4 and GNU awk) that also can cause
> infinite recursion on poor user input. However, Chet is not obligated to
> use it (even though the idea has been mentioned on the list before).
> 
>> Perhaps implement a maximal
>> recursion depth like zsh does.
> 
> Bash does, in the form of FUNCNEST, but you have to opt into it, as
> otherwise it would be an arbitrary limit, and arbitrary limits go
> against the GNU coding standards.
> 
> By the way, it is in general IMPOSSIBLE to write bash so that it can
> handle ALL possible bad user scripts and still remain responsive to
> further input.  Note that in my description of handling SIGSEGV above
> that I mention that it is only safe to gracefully turn what would
> otherwise be the default core dump into a useful error message - but
> bash STILL has to exit at that point, because you cannot guarantee what
> other resources (including malloc locks) might still be on the stack,
> where a longjmp back out to the main parsing loop may cause future
> deadlock if you do anything unsafe.  If you think you can make bash
> gracefully handle ALL possible bad inputs WITHOUT exiting or going into
> an infloop itself, then you are claiming that you have solved the
> Halting Problem, which any good computer scientist already knows has
> been proven to be undecidable.
> 

If a shell (that's interpreted) crashes due to overflowing its process's
actual call stack it can only be because the shell's "call_function"
function (or its callees) call call_function, and call_function is not
itself tail-recursive so the C compiler can't eliminate it. It should
be perfectly possible to implement that without any recursion so the
shell's stack representation (presumably on the heap) can grow without
affecting the real stack for EVERY call to any trivial shell function. I
don't know what kind of major surgery would be required on bash to fix
that. libsigsegv would only be a band-aid.



signature.asc
Description: OpenPGP digital signature

Re: /bin/sh should set SHELL to /bin/sh

[Dan Douglas]

On 07/14/2017 03:13 PM, Greg Wooledge wrote:
> On Sat, Jul 15, 2017 at 02:59:41AM +0700, Robert Elz wrote:
>> IMO, if SHELL gets unset (it is usually initialised by login, or its
>> equivalent), it should simply stay unset, and not be set to anything,
>> until some user (or script) decides to set it again.
> 
> wooledg:~$ unset SHELL
> wooledg:~$ bash -c 'echo "$SHELL"'
> /bin/bash
> wooledg:~$ ksh -c 'echo "$SHELL"'
> /bin/sh
> wooledg:~$ zsh -c 'echo "$SHELL"'
> 
> wooledg:~$ tcsh -c 'echo "$SHELL"'
> SHELL: Undefined variable.
> 
> Looks like there's not much agreement here.
> 

Good thing is bash doesn't export SHELL if it wasn't already exported,
and most shells seem to preserve the value of SHELL exported by some
parent. I see mksh sets -x if SHELL was previously unset... I suppose
that might cause a problem in some scenario (but probably not).



signature.asc
Description: OpenPGP digital signature

Re: "unset var" pops var off variable stack instead of unsetting it

[Dan Douglas]

On 03/18/2017 12:19 PM, Chet Ramey wrote:
> On 3/17/17 6:35 PM, Dan Douglas wrote:
> 
>> The problem is the non-obvious nature of unset's interaction with scope,
>> (and the lack of documentation). Not much can be done about the former,
>> as it is with so many things.
> 
> How would you suggest improving the documentation? I can see highlighting
> the fact that unset applied to a local variable at the same scope
> preserves the local attribute. What else?
> 

The effect of unset on a local was what I had in mind, but really the
manual says very little about scope. All it says right now is:

"Variables local to the function may be declared with the local builtin
command.  Ordinarily, variables and their values are shared between the
function and its caller."

Which doesn't exactly describe dynamic scope even for those that know
what that means.

Also not documented is how a variable declared with declare/typeset is
distinct from an unset variable.

Re: "unset var" pops var off variable stack instead of unsetting it

[Dan Douglas]

On 03/17/2017 09:16 PM, Dan Douglas wrote:
> Why
> would a subshell just make the call stack go away?

I guess slight correction, it's unset itself, because:

> In fact, mksh prints "global" even without the subshell, despite it 
> using dynamic scope for either function definition syntax.

Another "not-sure-if-bug-or-feature". It is a way to guarantee reaching
the global scope, which is impossible in bash, short of calling unset
${#FUNCNAME[@]} times.

If feature, I'm not instantly in love with it.

Re: "unset var" pops var off variable stack instead of unsetting it

[Dan Douglas]

On 03/17/2017 07:21 PM, Stephane Chazelas wrote:
> I don't expect the need to have to add "local var" in
> 
> (
>unset -v var
>echo "${var-OK}"
> )

True. I would pretty much never use a subshell command group when I know
that locals are available though. And if I know locals are available then
(except dash) I know arrays are available, in which case I'd almost never
use field splitting. This is only like the millionth screwy gotcha with
IFS. Everybody knows IFS is broken beyond repair :o)

Then again, you could easily write a similar bug with any other special
variable that has a side-effect.

> would be obvious to many people beside you though.
> very
> People writing function libraries meant to be used by several
> POSIX-like shells need to change their code to:
> 
> split() (
>   [ -z "$BASH_VERSION" ] || local IFS # WA for bash bug^Wmisfeature
>   unset -v IFS
>   set -f 
>   split+glob $1
> )
> 
> if they want them to be reliable in bash.

Even if the inconsistent effect of unset isn't obvious, it should be
obvious that a subshell isn't equivalent to setting a local, because it
doesn't just make dynamic scope go away.

I'm far more surprised by the behavior of mksh and dash, in which it's
the subshell rather than the unset builtin that's inconsistent. Why
would a subshell just make the call stack go away? That makes no sense,
and a subshell isn't supposed to do that. Dash and mksh don't even agree
with one another on how that works:

(cmd) ~ $ mksh /dev/fd/3 3<<\EOF
function f { typeset x=f; g; }
function g { ( unset x; echo "${x-unset}"; ) }
x=global; f
EOF

global

(ins) ~ $ dash /dev/fd/3 3<<\EOF
alias typeset=local function=
function f() { typeset x=f; g; }
function g() { ( unset x; echo "${x-unset}"; ) }
x=global; f
EOF

unset

In fact, mksh prints "global" even without the subshell, despite it
using dynamic scope for either function definition syntax.

At least bash's output in this case (empty) can be fully explained as
a combination of quirks with unset and hidden locals (neither being
documented), plus dynamic scope being what it is.

We're pretty much arguing over which is the less counter-intuitive
inconsistency here. If mksh's subshells worked consistently as in bash,
you'd have written the same bug as bash in your example. And it would
be even easier to do so without the unset quirk since this could happen
within a single function call too:

(cmd) ~ $ mksh /dev/fd/3 3<<\EOF
function f {
  typeset x=f
  ( unset x; echo "${x-unset}"; )
}

x=global; f
EOF

global

This could really bite if x and f are defined in separate files so the
initial state of x isn't necessarily known.

> So what should the documentation be? With my "eval" case in
> mind, it's hard to explain without getting down to how stacking
> variables work. Maybe something like:
> 
> [...]

All that touches on several issues in addition to scope, such as
the various states that a variable can be in, and the exact nature
of references to variables like 'a[@]'. That's some of the least-well
documented stuff, but some of that should also probably be left subject
to change due to the great inconsistency across shells and other issues
just within bash.  (Ugh also have to mention the stupid 'a[0]' with
associative arrays - that's one where "consistency" is itself a bug).

> It might be worth pointing out that "unset -v", contrary to the
> default behaviour, won't unset functions so it's a good idea to
> use "unset -v" instead of "unset" if one can't guarantee that
> the variable was set beforehand (like the common case of using
> unset to remove a variable which was potentially imported from
> the environment).

Yeah. I believe POSIX mentions that as well.

Re: "unset var" pops var off variable stack instead of unsetting it

[Dan Douglas]

The need to localize IFS is pretty obvious to me - of course that's
given prior knowledge of how it works.

The problem is the non-obvious nature of unset's interaction with scope,
(and the lack of documentation). Not much can be done about the former,
as it is with so many things.

Re: command_not_found_handle and sourced script

[Dan Douglas]

On Mon, Dec 26, 2016 at 12:42 PM, Dominique Ramaekers
 wrote:
> As I understand it, the command_not_found_handle is not triggered on an
> unknown command in a shell script, run normally.

Where'd you here that? That's easy to test.

 $ bash <<<'command_not_found_handle () { echo "$FUNCNAME"; }; blah'
command_not_found_handle

Doesn't seem to matter.

> To my opinion, a sourced script (once debuged) shouldn't contain unknown
> commands. So the command_not_found_handle is not necessary.

It may. A shell script may be run in any unknown environment which may
not have the available commands in PATH or elsewhere. Originally
command_not_found_handle was added for features like ubuntu's default
package suggestions when run interactively, but it has been used for
other purposes.

Nonterminating alias

[Dan Douglas]

A simpler one this time. Bash 4.4 only.

 $ bash -c $'alias @="eval {\n}"; eval @'
bash: xrealloc: cannot allocate 18446744071562068464 bytes

I would guess this is part of the way keywords are supposed to be
re-interpolated after alias expansion since 4.4. Maybe not even be a
bug depending on how I'm supposed to look at this.

Re: Command or process substitution resets alias name tracking during alias expansion?

[Dan Douglas]

On Wed, Nov 23, 2016 at 1:25 PM, Chet Ramey <chet.ra...@case.edu> wrote:
> On 11/22/16 5:46 PM, Dan Douglas wrote:
>> Hi. Here's a change between bash 4.1 and 4.2 that persists in 4.4.
>> (Added the counter to make it stop).
>
> Thanks for the incredibly obscure report.  This will be fixed in the next
> push to the devel branch.

There's an incredibly obscure ksh flavour too :-)

 $ ksh -c $'alias @=\'echo test >&2; )$( @ \'\n$( @ )' 2>&1 | head -n3
test
test
test

Ok I'm done. (unless maybe backticks...)

Command or process substitution resets alias name tracking during alias expansion?

[Dan Douglas]

Hi. Here's a change between bash 4.1 and 4.2 that persists in 4.4.
(Added the counter to make it stop).

ormaajtest@smorgbox $ ( bash-4.2 ) <<\EOF
shopt -s expand_aliases; n=0
alias @='((n >= 5)) && alias @="unalias @; echo"
printf "$((n++)) " $()
@'
@
EOF

0 1 2 3 4 5

ormaajtest@smorgbox $ ( bash-4.1 ) <<\EOF
shopt -s expand_aliases; n=0
alias @='((n >= 5)) && alias @="unalias @; echo"
printf "$((n++)) " $()
@'
@
EOF

The empty command expansion triggers this. I don't think it's an alias
quirk I've known about though. (Too bad because it's faster than eval
and I don't know of another way to make this happen).

If an alias that's read lexically outside of any other command in the
"global context" generates two "complete commands" on separate lines,
why is the second command considered within the scope of the current
alias expansion even though it technically hasn't been "read" yet?
Isn't it extra work for bash to remember exactly what code was
generated by which aliases just to keep track of which words to
ignore? I would think the first command would be evaluated before the
second command is read even if it's the product of an alias.

Re: 4.4 change in behavior from 4.3: how to catch unset when using ${#length}

[Dan Douglas]

On Mon, Oct 24, 2016 at 8:25 AM, Chet Ramey  wrote:
> On 10/21/16 5:41 PM, L. A. Walsh wrote:
>> On 4.3 and earlier, at least on arrays, one could have
>> the illusion of this working w/o complaint -- and returning
>> 0 when the array was 0-len or unset, or the array length,
>> otherwise:
>>
>>
>>  echo ${#array[@]:-0}
>>
>> But I note it only seemed to work in arrays, and in 4.4 gets a
>> syntax error:
>>
>>  echo ${#array[@]:-0}   bash: ${#array[@]:-0}: bad substitution
>
> Because it is a syntax error, and if it were not it would be ambiguous.
> The ${param:-word} word expansion takes a parameter, not another word
> expansion, as the object to be expanded.

On a possibly related note, would you consider adjusting +, :+, -, :-,
as in "${var[@]+word}" to align with the meaning of [[ -v var[@] ]] as
discussed in https://lists.gnu.org/archive/html/bug-bash/2014-11/msg00099.html
? I've always felt the best use of that expansion would be to test for
a defined array (or any set array element other than arr[0]).
${var[@]+word}, ${var[0]+word}, and ${var+word} are currently
redundant to my knowledge. The latter two I can understand, but var[@]
seems inconsistent.

My best interpretation of the current behaviour is that it copies that
of $@, which doesn't really make sense IMO because arrays may be
sparse. All the effects of ${@+word} make sense but don't translate
well directly to bash arrays. OTOH [[ -v var[@] ]] since bash 4.3
makes a lot of sense to me and I think they would translate well to
the corresponding parameter expansions.

Re: for Chet Ramney in Born Againe

[Dan Douglas]

On Mon, Oct 24, 2016 at 8:33 AM, Корень Зла  wrote:
> Can u re:coded BASH for Power Shell features to fully intagrated with Linux 
> enviroments

Directly supporting features in a compatible way would not be easy.
Powershell is a completely different language, and bash isn't yet
implemented on the CLR. I'm working on the latter, but my compiler
won't be ready for a while, and the focus is on bash compatibility,
not powershell.

If you're only interested in interoperating with powershell from bash
without changing bash directly, you could write a bash plugin to have
it host the CLR with libmono
http://www.mono-project.com/docs/advanced/embedding/. That isn't
usually very hard, though I haven't actually tried it with a bash
loadable.

Hope I understood your question.

Re: [minor] Space after last element on assoc print

[Dan Douglas]

By the way, that space at the end has been pointed out a number of
times lately. I think Chet clarified at some point that that's just
the way the serializer prints it - so it's nothing. Apparently a lot
of people think it's meaningful.

Re: [minor] Space after last element on assoc print

[Dan Douglas]

On Wed, Oct 19, 2016 at 11:47 PM, Quentin L'Hours
 wrote:
> Hi,
>
> Bash Version: 4.4
> Release Status: release
>
> Description:
>
> Useless space after last item of a declare -p on an assoc array (indexed
> arrays don't print it, and neither does ksh typeset on assoc arrays).
> It doesn't seem to have any consequence though.
>
> Repeat-By:
>
> $ declare -A capital[fr]=Paris
> $ declare -p capital
> declare -A capital=([fr]="Paris" )

You cannot assign an attribute to an individual element of any array. The
behaviour for indexed arrays is described in the manual:

"declare -a name[subscript] is also accepted; the subscript is ignored."

In the case of a previously declared associative array in the current
scope, one might argue that bash should throw an error instead. I think
what you're seeing is bash taking a "reasonable default" instead of
throwing an error in response to a nonsensical assignment.

Note that ksh93 is different in some respects because it allows "nested
variables".

Re: Is this the expected behaviour for nameref in Bash 4.4 now?

[Dan Douglas]

Yes that was an intentional change to require valid identifiers. I can't
say it will always be that way or that there won't at some point be a
workaround. You can stiill use `${!param}' for now to refer to positional
parameters as you always could, but as usual that isn't useful if you
want to assign by reference.

Re: Assigning to BASHPID fails silently

[Dan Douglas]

On Thu, Oct 20, 2016 at 2:35 PM, Chet Ramey  wrote:
> On 10/20/16 11:32 AM, Martijn Dekker wrote:
>
>> So, in some contexts this bug causes a premature exit of the shell, in
>> others it causes a premature exit of a loop. This bug hunt could get
>> interesting.
>
> No, it's trivial.  It's an assignment failure.  The fix is to remove
> the readonly attribute.

Makes sense to me. I noticed the problem on the same day this was
reported while trying to compare two environments while eliminating all
the non-constant dynamic variables from the comparison, that I could
only override BASHPID by calling bash with:

env BASHPID= -- bash -c ...

Seems removing readonly is the thing to do to make it work as
described in variables.c without having to call bash through env.

Re: Magnitude of Order "For Loop" performance deltas based on syntax change

[Dan Douglas]

On Mon, Sep 26, 2016 at 3:32 PM, Chet Ramey  wrote:
> So you want offset N to be the nth element in the array instead of the >
element with index N? Huh.

Maybe, not always. Both would be nice. The offset isn't the element with
the index N. It's the next set element whose index is >= that of the
selected offset. There's no simple way of knowing how many set elements
come before or after offset - possibly zero. In order to insert something
after the first element you have to find the index of the first element.

> Well, you probably want another data structure.

Yes please. Everybody wants more data structures. (I know... patches
accepted.)

Re: Magnitude of Order "For Loop" performance deltas based on syntax change

[Dan Douglas]

Would an array of pointers to structs of key-value pairs be better
here? It should be faster in the common cases even though it may mean
some wasted space and reallocs depending on how you decide to grow the
array. A linear search through an array for an index should be faster
than linked-list traversal. https://youtu.be/YQs6IC-vgmo (why every
std::vector implementation uses arrays, really it's true of analogues
in most non-functional langs).

Also bash itself makes it hard to use sparse arrays efficiently regardless
of the implementation. In the case of lists, one usually wants to address
elements by ordinal position, but both the index in `arr[index]` and the
offset in `${arr[@]:offset:length}` don't allow it, which means random
insertion requires a linear search despite being linked-lists. That also
makes the "length" inconsistent with everything else that looks at the
value of the index, though at least length does what I really wish
offset did.

On top of that, ${!arr[@]:offset:len} doesn't even work. None of the
parameter expansions work with ${!arr[@]}, so to calculate an ordinal
index, a second array containing the indexes of the first is needed. Plus,
pre-existing items are overwritten on assignment, so insertion means
having to save and restore all the indexes and values at least in a
region being inserted to, which means more index calculation.

Re: Location of configuration files

[Dan Douglas]

On Thu, Aug 18, 2016 at 5:34 AM, Joonas Saarinen  wrote:
> Hi,
>
> By default Bash uses these files in my home directory:
>
> .bashrc
> .bash_logout
> .bash_history
> .profile
>
> However, wouldn't it be more streamlined to place these files under
> directory '~/.config/bash'?

+1 for this. I've also been wanting the option to have bash look in
$XDG_CONFIG_HOME for the user-level bashrc and bash_logout. It's not
much of a compatibility breaker unless you have multiple bash versions
installed and actually use more than one of them interactively (or
have stupid programs like scp that do so), in which case a symlink
should work. Bash, openssh, and gnupg are the major things I use that
still don't utilize XDG_*.

.profile should probably stay where it is under ~.

Re: "until continue" causes bash to exit.

[Dan Douglas]

On 07/19/2016 09:45 PM, Chet Ramey wrote:
> On 7/19/16 7:15 PM, Dan Douglas wrote:
>> Hi, I could reproduce this in all bash versions.
>>
>> bash -c 'set -x; until continue; do :; done; echo test1'; echo test2
>> + continue
>> test2
>>
>> I'm not actually sure whether this is supposed to work at all. Almost
>> all shells do something strange when given continue in the first
>> compound-list. E.g. ksh93 does like bash and exits, and zsh breaks
>> out of the loop.
> 
> It should probably print `test1'

That's what I figured. Playing around with this test the current bash
continues an outer loop if levels=1 while the "normal" ones (dash/mksh)
do so at levels=2. I suppose technically a script that before would
have continued the n+1th nested loop will now be off by one. (Not too
likely such a script exists or they would have noticed this.)

 $ { bash /dev/fd/9; echo; } 9<<\EOF
set -x; n=-1 levels=1
let 1 2>/dev/null || let() { IFS=, command eval test '$(($*))' -ne 0; }

until
  echo 1
  let n+=1 && break
do
  until
echo 2;
continue "$levels"
  do :
  done
  echo 3
done
EOF

+ n=-1
+ levels=1
+ let 1
+ echo 1
1
+ let n+=1
+ echo 2
2
+ continue 1
+ echo 1
1
+ let n+=1
+ break



signature.asc
Description: OpenPGP digital signature

"until continue" causes bash to exit.

[Dan Douglas]

Hi, I could reproduce this in all bash versions.

bash -c 'set -x; until continue; do :; done; echo test1'; echo test2
+ continue
test2

I'm not actually sure whether this is supposed to work at all. Almost
all shells do something strange when given continue in the first
compound-list. E.g. ksh93 does like bash and exits, and zsh breaks
out of the loop.



signature.asc
Description: OpenPGP digital signature

Re: Global variable modification by nameref chain

[Dan Douglas]

On Sun, Jun 12, 2016 at 8:33 PM, Chet Ramey  wrote:
> 3. Honor the assignment and delete the nameref variable, creating a new
>one, like bash-4.3:
>
> $ ../bash-4.3-patched/bash ./x1
> declare -n a="b"
> declare -n b="a[1]"
> declare -a a='([1]="foo")'
> declare -n b="a[1]"

I kind of like this option because it's consistent with `typeset -n`
meaning "operate directly on the ref variable", and declare together
with assignment meaning "redefine the variable" in most cases. I'm not
so sure a warning is needed since replacing the variable with a new
definition would be the usual thing to do with any other attributes,
and the identifier literals used within one scope should be under
control.

Re: Global variable modification by nameref chain

[Dan Douglas]

On Thu, Jun 9, 2016 at 4:34 AM, Dan Douglas <orm...@gmail.com> wrote:
> How about just doing it similar to the way mksh resolves arithmetic
> variable loops? As each variable is visited, add it to a list (or hash
> set) and check whether that node was already visited. That should work
> without having to consider scope issues.
>
> $ mksh -c 'a=b b=c c=d d=1; echo $((a))'
> 1
> $ mksh -c 'a=b b=c c=d d=a; echo $((a))'
> mksh: a: expression recurses on parameter 'a'
>
> This nicely says which variable was to blame. I've noticed bash devel
> right now can give messages about "invalid identifiers" which are
> technically valid in most contexts so it's tough to track down
> problems in complicated reference chains.

Sorry one more follow-up. I should note bash has allowed deep
recursion in variable loops for ages using `${!var}` indirection, even
for arrays, and so long as you were careful about unbounded recursion
that never caused a problem before. This would be a
backwards-compatibility breaker if it applied generally. I don't think
namerefs need to be fantastically different from the old indirection
semantics so long as we're ignoring scope.

The only difference between namerefs and old-style indirection is that
an individual array element can't be a nameref and a nameref can't be
an array, so you need to prevent something like this which bash
already does...

 $ bash -c 'typeset -n a=b b; b=a[1]; a=foo; typeset -p a b' # bash 4.3
declare -a a='([1]="foo")'
declare -n b="a[1]"
 $ ./bash -c 'typeset -n a=b b; b=a[1]; typeset -p a b; a=foo' # 4.4
declare -n a="b"
declare -n b="a[1]"
./bash: line 0: `a[1]': not a valid identifier

(That's the confusing error mentioned previously btw)

Re: Global variable modification by nameref chain

[Dan Douglas]

How about just doing it similar to the way mksh resolves arithmetic
variable loops? As each variable is visited, add it to a list (or hash
set) and check whether that node was already visited. That should work
without having to consider scope issues.

$ mksh -c 'a=b b=c c=d d=1; echo $((a))'
1
$ mksh -c 'a=b b=c c=d d=a; echo $((a))'
mksh: a: expression recurses on parameter 'a'

This nicely says which variable was to blame. I've noticed bash devel
right now can give messages about "invalid identifiers" which are
technically valid in most contexts so it's tough to track down
problems in complicated reference chains.

Re: Global variable modification by nameref chain

[Dan Douglas]

On Sat, Jun 4, 2016 at 5:48 PM, Grisha Levit  wrote:
> On May 23, 2016 1:42 PM, "Chet Ramey"  wrote:
>> > Should the assignment work?  I'm considering changing the
>> > assignments to
>> > work more like the references.
>> >
>> > I think it would be useful for the assignment to work, as that allows
>> > functions to take variable names as arguments without worrying about
>> > name
>> > collisions.
>>
>> I don't like the fact that variable binding simply ignores nameref loops
>> and treats them the same as the variable not being found.  That's the
>> difference here.
>
> What if (in a function scope) a nameref self-reference were unambiguously
> treated as referring to a variable in the next higher scope, other that
> declare [+-] on an existing ref?

I've thought about this as a workaround but resolving loops that way
will probably lead to problems. Needless to say I prefer doing it the
way I previously described where ref=ref and ref=$1 are distinct.
That's separate from the subject of nameref loops between locals of
the same scope of course.

You can already see some of the eventual problems in ksh if you try to
define a nameref within a compound that points to a local and pass the
compound by reference or copy it between functions using typeset
-c/-m. It's a huge mess of bugs.

Re: a == a[0] convention

[Dan Douglas]

On Thu, Jun 2, 2016 at 12:18 PM, Grisha Levit  wrote:
> I think all of the stuff here is fixed in the latest devel snapshot

Ah ok I tried this again. Yes this looks better now, thanks.

> there is definitely weirdness if you run the assignments as a typeset
> command.  I hadn't even tried the ref4 case, which gives the weirdest
> results when used with typeset/declare:
> https://groups.google.com/forum/#!topic/gnu.bash.bug/Z-lf3KEnZ34

That's what reminded me of this, but your example was definitely a bug
(regression since 4.3 at least). The ref4[0] one is just an odd
special case but I think it's probably easier to treat it as an error.

Re: RFC: Enabling or disabling tracing shell functions

[Dan Douglas]

On Sun, Jun 5, 2016 at 8:48 PM, Chet Ramey  wrote:
> "Traced functions inherit the DEBUG  and  RETURN  traps  from  the  calling
> shell."

Why did RETURN originally get sucked into set -T? Was it supposed to
be primarily for debugging? Some functions actually use it for
internal purposes and enabling -T to debug causes (probably unwanted)
side-effects in such cases since there's no way to control their
inheritance separately.

If the goal is to inspect values when a function returns that can be
accomplished with just a DEBUG trap:
trap 'if [[ $LASTFUNC != ${FUNCNAME[0]} ]]; then ...;
LASTFUNC=${FUNCNAME[0]}; fi' DEBUG

Re: RFC: Enabling or disabling tracing shell functions

[Dan Douglas]

Not sure exactly how zsh does it but I know not having the option for
both global and local tracing can be annoying.

The two big ways of handling xtrace I mostly see are either bash's
global `set -x` or ksh93's per-function tracing, and it can be
annoying to be missing either one. There are tricks for getting the
effect of global xtrace and/or DEBUG traps in ksh, but its lack of a
BASH_XTRACEFD equivalent means it's sometimes impossible to hide the
side-effects of those tricks, so you get messy output.

With regards to `typeset -ft`, I don't like it. DEBUG and RETURN have
totally different usages. Inheriting RETURN very rarely makes sense,
but I almost always want DEBUG inherited because of course it's
primarily used for debugging.

Re: Mulit-line aliases and PROMPT_COMMAND

[Dan Douglas]

On Thu, Jun 2, 2016 at 7:18 PM, Grisha Levit  wrote:
>
> On Thu, Jun 2, 2016 at 6:57 PM, Chet Ramey  wrote:
>>
>> > Since bash 4.3 multi-line aliases interact very strangely
>>
>> And you're the first person to report them.  I guess there aren't a lot of
>> multi-line aliases out there.
>
>
> I wonder if more crazy use cases will come out of the woodwork when RHEL and
> SUSE finally get 4.3..

I think when they're used in an actual script rather than for
interactive convenience multiline aliases are some of the more useful
ones. The 4.4 changes will make aliases even more interesting.

a == a[0] convention

[Dan Douglas]

This is possibly relevant to some of Grisha's observations. First,
declaring or assigning to a variable with a subscript. I think I would
prefer these to be errors for various reasons. Admittedly there's an
argument for making one or both of these non-errors for declarations
without assignment for consistency with the current documentation, and
maybe even allowing the current behavior (no errors ever). I think
that leads to confusion though.

 $ bash -c 'typeset -n ref[0]=foo' # no error
 $ bash -c 'typeset -n ref[1]=foo' # no error

The other variations on this theme are if there are references to
"arrays" with or without a zero subscript. Bash example:

 $ bash -xc '
typeset -n ref1=a1 ref2=a2 ref3=a3[0] ref4=a4[0]
ref1=foo ref2[0]=foo ref3=foo ref4[0]=foo
echo "${ref1} ${ref2[0]} ${ref3} ${ref4[0]}"
typeset -p {ref,a}{1,2,3,4}'

+ typeset -n ref1=a1 ref2=a2 'ref3=a3[0]' 'ref4=a4[0]'
+ ref1=foo
+ ref2[0]=foo
+ ref3=foo
+ ref4[0]=foo
+ echo 'foo foo foo foo'
foo foo foo foo
+ typeset -p ref1 ref2 ref3 ref4 a1 a2 a3 a4
declare -n ref1="a1"
declare -a ref2=([0]="foo")
declare -n ref3="a3[0]"
declare -a ref4=([0]="foo")
declare -- a1="foo"
/home/ormaaj/doc/programs/bash-build/bash: line 4: typeset: a2: not found
declare -a a3=([0]="foo")
/home/ormaaj/doc/programs/bash-build/bash: line 4: typeset: a4: not found

I think the mksh result for that code is most like what bash is
probably going for. It mostly treats `ref[0]` as `ref` both when
assigning and dereferencing.

 $ mksh -xc '
typeset -n ref1=a1 ref2=a2 ref3=a3[0] ref4=a4[0]
ref1=foo ref2[0]=foo ref3=foo ref4[0]=foo
echo "${ref1} ${ref2[0]} ${ref3} ${ref4[0]}"
typeset -p {ref,a}{1,2,3,4}'
+ typeset -n 'ref1=a1' 'ref2=a2' 'ref3=a3[0]' 'ref4=a4[0]'
+ ref1=foo ref2[0]=foo ref3=foo ref4[0]=foo
+ echo 'foo foo foo foo'
foo foo foo foo
+ typeset -p ref1 ref2 ref3 ref4 a1 a2 a3 a4
typeset -n ref1=a1
typeset -n ref2=a2
typeset -n ref3='a3[0]'
typeset -n ref4='a4[0]'
typeset a1=foo
set -A a2
typeset a2[0]=foo
set -A a3
typeset a3[0]=foo
set -A a4
typeset a4[0]=foo

ksh93 also does that except ref4 which understandably assigns to a4[0][0].

Re: {ref}<&- does not work on array reference subscripts

[Dan Douglas]

You also noticed it runs in a subshell generally. Do you expect
something different here? I think <&- should be consistent.

On Wed, Jun 1, 2016 at 5:15 PM, Grisha Levit  wrote:
> declare -n ref=var[0]
> {ref} {ref}<&-  # fails

Re: redirections preceded by a word with builtins inconsistent

[Dan Douglas]

On Tue, May 24, 2016 at 10:23 AM, Chet Ramey  wrote:
> It's not that Posix `allows' a subshell, it requires a subshell
> environment:

Hm sorry I thought it was "may". This is kind of easy to confuse with
all the other unspecified things about side-effects from assignments
affecting subsequent assignments and commands when there's no command
name.

Re: coproc and existing variables

[Dan Douglas]

I don't even see why we need a magic variable for this. ksh makes you
manually store $! and bash also allows this.

As an alternative, create a special BASH_COPROC_PIDS associative array
to map coproc names to pids. ${foo}_suffix=bar is never as good as an
associative array.

Re: redirections preceded by a word with builtins inconsistent

[Dan Douglas]

On Sun, May 22, 2016 at 1:06 PM, Chet Ramey  wrote:
> On 5/21/16 5:16 PM, adonis papaderos wrote:
>
>> Bash Version: 4.3
>> Patch Level: 42
>> Release Status: release
>>
>> Description:
>> When using redirections preceded by a word on builtins
>> 'i.e. : {varname}<&1'
>> the redirection persists to the current shell.
>
> This is by design, and has been discussed previously.  One such discussion
> starts at http://lists.gnu.org/archive/html/bug-bash/2016-01/msg00156.html.
> Maybe it should only do it when preceded by `exec', but it never has.
>
>> Later using {varname}<&- does not close fd.
>
> You don't precede it with an `exec', so it doesn't affect the current
> shell.  So-called `null' commands with redirections are executed in
> subshells.  This is where you need the `exec'.

I know POSIX allows for a subshell but don't know what the point is. I
was confused by this when I first hit it. The subshell isn't obvious.

 $ bash -c 'echo "$BASHPID"
{x[$((pid=BASHPID))$(echo "2: $pid" >&2),0]}<<<"$((pid=BASHPID))$(echo
"1: $pid" >&2)"'
3907
1: 3908
2: 3908
 $ bash -c 'echo "$BASHPID"
{x[$((pid=BASHPID))$(echo "2: $pid" >&2),0]}<<<"$((pid=BASHPID))$(echo
"1: $pid" >&2)"'
3911
1: 3911
2: 3911

ksh being the only other shell that supports auto-fd assignment for
null commands doesn't create one.

 $ bash -c 'typeset -li y=2
{x[y*=2]}<&$((y*=2,0)); typeset -p x y'
bash: line 1: typeset: x: not found
declare -il y="2"
 $ bash -c 'typeset -li y=2
: {x[y*=2]}<&$((y*=2,0)); typeset -p x y'
declare -a x='([8]="10")'
declare -il y="8"
 $ ksh -c 'typeset -li y=2
{x[y*=2]}<&$((y*=2,0)); typeset -p x y'
typeset -a -i x=([8]=11)
typeset -l -i y=8
 $ ksh -c 'typeset -li y=2
: {x[y*=2]}<&$((y*=2,0)); typeset -p x y'
typeset -a -i x=([8]=11)
typeset -l -i y=8

ksh and zsh are hard to test since they do a horrible job parsing the
{fd} in all but trivial cases.

It would be way simpler to only allow redirects at the end of a
command and require each redirect to be a separate word. That
eliminates a ton of ambiguities for {fd}>x, &>x and >

Maybe add a `set -k` work-alike for legacy scripts and people that insist on
`echo >&2 i like surprise redirec{t}

Re: namref dicussion continued: [PATCH] export -r reference

[Dan Douglas]

On Sun, May 8, 2016 at 6:03 PM, Chet Ramey <chet.ra...@case.edu> wrote:
> On 5/5/16 6:02 PM, Dan Douglas wrote:
>> On Thu, May 5, 2016 at 2:46 AM, Piotr Grzybowski <narsil...@gmail.com> wrote:
>>>  ok, so Dan wants this patch.
>>
>> Yes I think your bash patch is probably an improvement at least in
>> this particular case.
>>
>>> Maybe you can comment on wether the patches are valid.
>>
>> The posix export and readonly don't produce locally scoped variables
>> at all, so it's expected that e.g. `typeset -x` and `export` will
>> differ.
>
> This is a different topic entirely, since neither of the examples
> we're considering take place within a function.  Scoping isn't an
> issue.

Sure I agree that scope isn't the issue. That's was the main point of
my message. I was trying to explain is that ksh does allow -x in
combination with pre-existing references in some contexts, but the
only way to demonstrate that is by calling a function.

Re: Global variable modification by nameref chain

[Dan Douglas]

On Fri, May 6, 2016 at 8:28 AM, Grisha Levit  wrote:
> The issue I was trying to raise is that assignment modifies the global
> variable but expansion uses the local value.
> If the assignment to the global variable is intentional, then shouldn't
> expansion use the global variable's value as well?

Nope. There's no workaround other than to unset every variable with a
conflicting name that lies between you and the global. Even if you
want to try that, there's no way to know how many such variables there
are and which one is currently visible.

declare -g is still sometimes useful though. I use it to set bash
special variables and configurations in bashrc to better ensure
they're being put into the actual global scope and should apply
everywhere. `declare -gn` is a little unusual since normally you
wouldn't use -n with other options, but I can see the use for `-gn`.

Re: namref dicussion continued: [PATCH] export -r reference

[Dan Douglas]

Oh I see you talked about some of this already in the "chained
nameref" thread. I haven't read that one yet... I wonder why `export
-r` requires a different solution because it's pretty much the same
underlying problem.

Re: namref dicussion continued: [PATCH] export -r reference

[Dan Douglas]

On Thu, May 5, 2016 at 2:46 AM, Piotr Grzybowski  wrote:
>  ok, so Dan wants this patch.

Yes I think your bash patch is probably an improvement at least in
this particular case.

> Maybe you can comment on wether the patches are valid.

The posix export and readonly don't produce locally scoped variables
at all, so it's expected that e.g. `typeset -x` and `export` will
differ.

This is yet another problem that's better solved by immutable C++-like
references. At least, the problem goes away when variables are passed
explicitly.

The exact mechanism isn't really documented. When ksh determines that
a nameref assignment is to a positional parameter during evaluation,
you get a completely different type of reference. That is, if `$1`
expands to `x`, then `typeset -n ref=$1` and `typeset -n ref=x` aren't
equivalent like they are in bash.  In the former case, any use of
`ref` except for `typeset +n` and `unset -n` applies to the exact
object that `$1` referred to at the time the reference was created,
even if it's another nameref in which case that reference is followed.
The latter gives you a bash-like dynamic reference that points to
somewhere within either the current scope or the global scope (due to
static scoping) unless it happens to point to another reference
parameter of course.

When you changed that `typeset` line to `export -r _x`, things become
less clear. Ksh sees that _x is a reference parameter and because of
the rules it does the only thing that makes sense which is apply the
export attribute to whatever was passed to the function for both
`export` or `typeset -x`, though there may be some other differences
between those.

Bash on the other hand never does what you'd want for either `export`
or `typeset -x`. For typeset, it tears down the reference and
redefines it as a new local with the export attribute. It does that
indiscriminately for any usage of typeset, which means you can't
correctly modify an attribute by reference in bash. If you actually
wanted that result, the correct way would be to first use `typeset +n
_x` followed by `typeset -x _x`. If you choose `export`, the correct
behavior is even less clear thanks to dynamic scope.

Your patch is a partial solution that makes `export` modify the
attribute in the way you would expect in the absence of the reference
parameter feature. Ignoring namerefs, the only other way for bash to
modify the attribute of the currently visible dynamically scoped local
without declaring a new local is to use export, which is why bash
behaves as it did. This solution doesn't solve the same problem for
attributes other than export and readonly. Bash can't use e.g.
`typeset -u` by reference. It also doesn't solve the many other
problems caused by this like name conflicts.

Re: param expansion with single-character special vars in the environment

[Dan Douglas]

...Also remember it isn't feasible to actually validate a "name" in a
script because a name can contain a subscript with a command
substitution that effectively requires parsing the full language.
(there are some tricks like with `set -nv` and reading its output to
shanghai the shell parser into doing your bidding, but that's not very
practical.). Before bash had namerefs, it could ignore aspects of
"invalid" names, like trailing characters after an array subscript,
which makes some valid ksh names at least get partially interpreted in
bash, like `a[foo].bar` or `a[foo][bar]`.

Re: param expansion with single-character special vars in the environment

[Dan Douglas]

On Wed, May 4, 2016 at 2:37 PM, Piotr Grzybowski  wrote:
>
> On 4 May 2016, at 17:51, Chet Ramey wrote:
>
>> The issue I'm thinking about currently is whether or not to allow nameref
>> variables to have numeric (integer) values.  bash-4.3 didn't allow those
>> values to be expanded, but allowed them to be assigned.  It doesn't seem
>> harmful to change the error to assignment time.
>
> I vote for error at assignment, whats the point of allowing pointless 
> assignment?

Since almost all bash errors are runtime errors that you might never
hit until something you didn't expect gets passed the wrong value,
errors (especially fatal errors) that might sound like a good idea at
first end up being totally pathological. You end up having to manually
test and validate all inputs to all functions whose callers aren't
strictly controlled, or add redirects to hide warning messages for
non-fatal errors. There are even cases where it's useful to allow or
at least ignore invalid names. If this were a language where I could
get a compiler error caught through static analysis then it would be a
different story.

Re: namref dicussion continued: [PATCH] export -r reference

[Dan Douglas]

Yeah I was just looking for this old script last night and just found it :)

https://gist.github.com/ormaaj/04923e11e8bdc27688ad

If you scroll down to the output for "test 3" where "h" gets called
and passes a local "x" to a function that creates a reference to it
and exports the reference you can see that bash calls "x" unset in
both the first and second scope. As I recall we were discussing the
way exported locals interact between scopes at the time, not namerefs,
but I added the case for namerefs since I wasn't sure how this should
work. I should probably run it again with the current devel branch.

Even though bash doesn't yet support references for parameters like
ksh93, neither does mksh, which also shows differences from bash.

Re: param expansion with single-character special vars in the environment

[Dan Douglas]

On Mon, May 2, 2016 at 1:59 PM, Grisha Levit  wrote:
>
> On Mon, May 2, 2016 at 2:48 PM, Chet Ramey  wrote:
>>
>> and this one throws away the nameref attribute:
>>
>> typeset -n foo ; typeset -i foo ; foo=7*6 ; typeset -p foo
>
>
> I think it's important to note that the nameref attribute is thrown away at
> the `typeset -i foo' step, which makes some sense since `typeset -n -i' does
> not seem to be valid in ksh.

ksh93 stores integer variables as actual integers internally so
there's no opportunity for storing the expression, and the nameref
attribute just wouldn't make sense.

Bash stores a string that gets coerced as necessary when performing
arithmetic. That's why ksh is so much faster than bash when you
declare integers, and an integer will wrap around while a string will
get converted back and fourth via pretty much the same rules bash
uses.

 ~ $ ksh -c 'integer n=$(getconf LONG_MAX); typeset m=$n; ((m++,n++));
typeset -p n m'
typeset -l -i n=-9223372036854775808
m=9.22337203685477581e+18

The only effect of -i in bash is to modify the way += assignments work.

Array expansions in scalar assignments

[Dan Douglas]

Hi, it looks like these expansions still aren't quite being delimited
by IFS consistently. Probably strangest of these is a4 where `$*`
doesn't agree with `${a[*]}`, and a3 vs. a4 where quoting modifies the
result. I would think bash should look like the ksh output with all
@'s space-separated and all *'s comma-separated.

This was essentially the same bug, though I didn't test typeset vs
plain assignments or quoting variations back then:
https://lists.gnu.org/archive/html/bug-bash/2013-01/msg00167.html

+smorgbox bash-build # cat ~ormaaj/doc/projects/bash/testcases/peflattening2
shopt -s extglob
{ for sh in /bin/bash?(-*) ~ormaaj/doc/programs/bash-build/bash ksh
mksh; do "$sh" /dev/fd/3; done; } 3<<\EOF
function f {
   typeset -a a
   a=("$@")
   typeset IFS=,
   typeset a1="${a[@]} ${a[*]} $@ $*"
   typeset a2=${a[@]}\ ${a[*]}\ $@\ $* a3 a4
   a3="${a[@]} ${a[*]} $@ $*"
   a4=${a[@]}\ ${a[*]}\ $@\ $*
   unset -v IFS
   printf '%s\n' "${BASH_VERSION:-${KSH_VERSION}}:" "a1=$a1" "a2=$a2"
"a3=$a3" "a4=$a4"
}

echo
f a b c
# vim: ft=sh et
EOF

+smorgbox bash-build # bash ~ormaaj/doc/projects/bash/testcases/peflattening2

4.3.42(1)-release:
a1=a,b,c a,b,c a,b,c a,b,c
a2=a,b,c a,b,c a,b,c a,b,c
a3=a b c a,b,c a b c a,b,c
a4=a b c a,b,c a b c a b c

4.4.0(1)-rc2:
a1=a,b,c a,b,c a,b,c a,b,c
a2=a,b,c a,b,c a,b,c a,b,c
a3=a b c a,b,c a b c a,b,c
a4=a b c a,b,c a b c a b c

Version ABIJM 93v- 2014-12-24:
a1=a b c a,b,c a b c a,b,c
a2=a b c a,b,c a b c a,b,c
a3=a b c a,b,c a b c a,b,c
a4=a b c a,b,c a b c a,b,c

@(#)MIRBSD KSH R52 2016/04/09:
a1=a b c a,b,c a b c a,b,c
a2=a b c a,b,c a b c a,b,c
a3=a b c a,b,c a b c a,b,c
a4=a b c a,b,c a b c a,b,c

Re: [patch] /* XXX - possibly run Coproc->name through word expansion? */

[Dan Douglas]

FWIW, something like this works currently. This pattern is useful in a
bunch of situations where the shell wants to assign to a fixed
variable name. (getopts being another).

 ~ $ bash /dev/fd/9 9<<\EOF
function mkProcs {
   typeset -n COPROC ref=$1
   set -- "${ref[@]}"
   for COPROC; do
   coproc {
   typeset status=$((RANDOM%255))
   echo "${!COPROC} status: $status"
   return "$status"
   }
   COPROC[2]=$COPROC_PID
   done
   for COPROC; do
   echo "${!COPROC}: $(cat <&"${COPROC[0]}"; wait "${COPROC[2]}") $?"
   done

}

typeset -a coprocs=({a..f}) "${coprocs[@]}"
mkProcs coprocs
EOF

/dev/fd/9: line 4: warning: execute_coproc: coproc [7768:COPROC] still exists
/dev/fd/9: line 4: warning: execute_coproc: coproc [7769:COPROC] still exists
/dev/fd/9: line 4: warning: execute_coproc: coproc [7770:COPROC] still exists
/dev/fd/9: line 4: warning: execute_coproc: coproc [7771:COPROC] still exists
/dev/fd/9: line 4: warning: execute_coproc: coproc [7772:COPROC] still exists
a: a status: 216 216
b: b status: 90 90
c: c status: 196 196
d: d status: 87 87
e: e status: 191 191
/dev/fd/9: line 13: "${COPROC[0]}": Bad file descriptor
f:  154

... Not sure why that last iteration always fails. I might want to
experiment with reducing the limitation that makes this only work
within a function on for loops over positional parameters. I know
that's how it is in ksh but I've never found that it's necessary.

Re: Security Vulnerability Reporting

[Dan Douglas]

On Fri, Feb 26, 2016 at 10:02 AM, Eric Blake <ebl...@redhat.com> wrote:
> Very few bugs in bash are security vulnerabilities (shellshock being the
> obvious exception).  Yes, bash has bugs, but in most cases, what people
> think are security bugs in bash are actually poorly-written shell
> functions that crash for the user, but which can't exploit bash to
> escalate the user's privileges.

All true. To be a genuine issue it usually has to be something that
causes a security problem in programs that utilize bash independent of
the script being run, or which exploits some common aspect of any script
that couldn't have been foreseen. The script is usually to blame.

-- 
Dan Douglas

Re: Security Vulnerability Reporting

[Dan Douglas]

On Fri, Feb 26, 2016 at 8:54 AM, Travis Garrell
<travis.garr...@oracle.com> wrote:
> Is there a set process in place for reporting security vulnerabilities
> against bash? If so, what might that process be?

Mail the maintainer. See:
https://tiswww.case.edu/php/chet/bash/bashtop.html#Bugs
Encrypt with: https://tiswww.case.edu/php/chet/gpgkey.asc

-- 
Dan Douglas

Re: [PATCH] admit 'typeset' is here to stay

[Dan Douglas]

On Sat, Feb 20, 2016 at 5:28 PM, Martijn Dekker  wrote:
> Am I missing something? I thought they did exactly the same thing in
> bash. If I'm not wrong about that, then as far as bash is concerned,
> they are in fact synonyms and functionally equivalent.

Yes declare and typeset are identical in bash.

> 'typeset' and 'declare' in bash are both dynamically scoped, aren't
> they? Where is the difference between them?

typeset has more historical baggage. If you're sticking to bash then use
whichever you prefer. though the only reason you'd typically use typeset
is if you're not planning on doing that, in which case you have to care
about the differences.

> 'typeset' in AT ksh93 is statically scoped, but ksh93 is the only
> shell to have statically scoped local variables -- all the other shells
> with 'typeset' (pdksh/mksh, bash, zsh, yash) have it as in ksh88,
> dynamically scoped -- and it doesn't look like this is ever going to
> change, or it would have changed sometime in the 23 years since ksh93
> came out.

It may not change, but dynamic scope can be a handicap in all of
those. Regardless, "local" is reasonably portable to bash and zsh. It's a
default alias to typeset in mksh. Ksh93 v- includes a dynamically-scoped
local (the details of which are somewhat unclear but hopefully that
situation will improve). That leaves old shells like ksh88 that had
dynamically scoped typeset without local. If ksh88 and portability are
that important then you probably don't want to use typeset at all (or
alias it to local and be careful with how it's used, or use typeset if
you know what you're doing).

Scope isn't the only problem. typeset is tied to all of the nonstandard
shell data types. zsh's contiguous integer array type for example that
clobbered `typeset -a` by giving it the same name as ksh/bash sparse
arrays. Even the "emulate" modes don't fix that blunder. Problems like
that are probably a hopeless lost cause. typeset didn't come from bash,
and at least some aspects of it differ wildly between every shell that
has it. (I haven't even gotten into the interaction with "function"
vs "name()" and all the weird scope issues with traps and set options
in {m,}ksh)

>> For the most common requirement: localizing or applying an attribute to
>> a variable within a single function; "local" and the posix-compatible
>> declaration builtins are usually what people want. "local" is supported
>> by every current implementation that also has "typeset" and "declare",
>
> That would be bash and zsh only, so that's not very meaningful.
>
> Perhaps you meant "every current implementation that also has
> 'typeset'", which would be ksh93, pdksh/mksh, bash, zsh, and yash.

yes.

> The latter two have dynamically scoped 'typeset', so on those, you can
> simply alias 'local' to 'typeset' to get the same functionality with the
> familiar keyword.

bingo.

> But on ksh93, getting the equivalent of a dynamically scoped 'local' is
> impossible. It has statically scoped local variables only. See Part III,
> Q28 at http://kornshell.com/doc/faq.html

It's not impossible, there are many workarounds even without the bash
compatibility mode. I have lots of scripts that run in both ksh93 and
bash and make heavy use of indirection and locals. Only complex scripts
are very sensitive to the scope issues to begin with.

Bash and mksh are much better nowadays since all three have namerefs
with compatible syntax. The only big missing feature is references
on positional parameters. For simple scripts that's not necessarily
a problem.

Re: [PATCH] admit 'typeset' is here to stay

[Dan Douglas]

On Sat, Feb 20, 2016 at 2:55 PM, Dan Douglas <orm...@gmail.com> wrote:
> http://thread.gmane.org/gmane.comp.standards.posix.austin.general/8371/focus=8377

I meant to post Chet's reply to my question:
http://article.gmane.org/gmane.comp.standards.posix.austin.general/8482

But that whole thread has some good info.

Re: [PATCH] admit 'typeset' is here to stay

[Dan Douglas]

On Thu, Jan 28, 2016 at 4:36 PM, Martijn Dekker  wrote:
> In 'help typeset', the 'typeset' builtin is called obsolete and has been
> so since at least bash 2.05b (2002) or possibly earlier. Perhaps it's
> time to just call it a synonym, as indeed the texinfo documentation does.
>
> diff -ur bash-4.4-beta.orig/builtins/declare.def
> bash-4.4-beta/builtins/declare.def
> --- bash-4.4-beta.orig/builtins/declare.def 2015-08-24
> 19:02:28.0 +0100
> +++ bash-4.4-beta/builtins/declare.def  2016-01-28 22:32:16.0 +
> @@ -65,7 +65,7 @@
>  $SHORT_DOC typeset [-aAfFgilnrtux] [-p] name[=value] ...
>  Set variable values and attributes.
>
> -Obsolete.  See `help declare'.
> +Synonym of `declare'.  See `help declare'.
>  $END
>
>  #include 
>

"typeset" vs. "declare" isn't always a straightforward decision
depending on one's requirements. Bash's "typeset" has always
been there for portability despite being a synonym for "declare",
so I think just calling it a synonym would be counterproductive.
http://thread.gmane.org/gmane.comp.standards.posix.austin.general/8371/focus=8377

It's tricky because the big problems surrounding "typeset" lie in the
details of variable scope and how each option is supported. There are
some good reasons for choosing "typeset" if you're willing to work around
all the subtleties but I think that's a bit beyond a concise description
for the help output.

For the most common requirement: localizing or applying an attribute to
a variable within a single function; "local" and the posix-compatible
declaration builtins are usually what people want. "local" is supported
by every current implementation that also has "typeset" and "declare",
plus a few others like dash (to a limited degree).  Those that are mostly
concerned with bash compatibility usually choose "declare" when not using
"local".

Re: capturing in ${VAR//(*(\/)(+([^\/]))?(\/)/_${BASH_REMATCH[1]}_}

[Dan Douglas]

On Mon, Feb 8, 2016 at 9:20 AM, Stephane Chazelas
 wrote:
> 2016-02-08 09:00:09 -0500, Chet Ramey:
>> On 2/8/16 2:47 AM, Linda Walsh wrote:
>> > When you are doing a var expansion using the
>> > replacement format ${VAR//./.}, is there some way to
>> > put parens around some part of the expression and reference
>> > them as in the [[V~re]] RE-matches?
>>
>> No.  Shell patterns do not have backreferences.
> [...]
>
> Note that the feature is available in other shells and quite
> handy there. It could be worth adding to bash
>
> $ zsh -o extendedglob -c 'a=1234; echo 
> ${a//(#b)(?)(?)/${match[2]}${match[1]}}'
> 2143
> (#b) to activate back-references stored in $match array.
>
> $ zsh -o extendedglob -c 'a=1234; echo ${a//(#m)?/<$MATCH>}'
> <1><2><3><4>
> (#m) to record the matched portion in $MATCH.
>
> Though I suspect for bash you would prefer the ksh93 syntax:
>
> $ ksh93 -c 'a=1234; echo ${a//@(?)@(?)/\2\1}'
> 2143

Technically that's "grouping", but yeah it's a useful feature. ksh
does backrefs in plain shell patterns also.

$ ksh -c 'a=11223344; echo "${a//@(@(?)\2)@(@(?)\4)/\3\1}"'
22114433

Re: bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Dan Douglas]

Sorry, spoofed identity (thanks gmail for picking a random sender).
I'll be rid of gmail as soon as I get a little free time.

Re: bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Dan Douglas]

\On Tue, Feb 16, 2016 at 1:30 AM, Linda Walsh <b...@tlinx.org> wrote:
>
>
> Dan Douglas wrote:
>>
>> Ah so `arr+=([a]=x [b]=y)` will no longer be the same as `arr+=([a]+=x
>> [b]+=y)`? I never liked that for associative arrays because the only
>> workaround was to do multiple arr[key]= assignments to update or add
>> more than one element at a time.
>>
>> My thinking was that this is due to bash's unique auto-incrementing of
>> indexed array keys. Since in ksh you're not allowed to specify an
>> index for one element of a compound assignment without specifying it
>> for all elements, bash has some additional things to consider.
>>
>>  ~ $ bash-4.2 -c 'typeset -ia a=({0..5}); a+=([0]+=1 {2..4}); typeset -p
>> a'
>> declare -ai a='([0]="1" [1]="3" [2]="5" [3]="7" [4]="4" [5]="5")'
>>
>> Bash 4.4:
>>  ~ $ ./doc/programs/bash-build/bash -c 'typeset -ia a=({0..5});
>> a+=([0]+=1 {2..4}); typeset -p a'
>> declare -ai a=([0]="1" [1]="2" [2]="3" [3]="4" [4]="4" [5]="5")
>>
>> I almost think it makes sense to treat ordered and unordered
>> collections differently.
>
> Why?
>>
>>  With an unordered collection an outer +=
>> should obviously mean "add or update for each assignment".
>
> For an array, you mean? like
> array a=( 1 2 3)
> array a+=( 4 5 6)
> then you get array a=(5 7 9). Or are you saying
> for an ordered array you'd have to use indexes, like
> array a=(1 2 3)
> array a+=([0]=4 [1]=7 [2]=10), then that would do your
> -- but wouldn't that be doing a vector operation of
> sorts?

I mean exactly the example I posted. In bash if you explicitly specify
an index for an assignment at any point within an auto-incrementing
compound assignment, bash will jump to that position and continue
incrementing. Bash is the only shell that has that property.

There's nothing wrong with that feature but the question of what to do
when an auto-incrementing assignment encounters an element with a
previous value isn't as obvious because the semantics differ. For an
ordered collection the outer += translates to "append to the list"
while with an unordered collection it roughly means to "union" the two
sets of keys.

You could argue that for consistency bash should always bulldoze over
previous values even if auto-incrementing keys. That makes sense from
a certain perspective and is consistent. From another perspective you
might expect the outer += to mean append for each sub-assignment to an
auto-incrementing array when not appending to the end of the array.
That especially makes sense if it's an integer array. I can understand
the logic either way.

I wrote about all of these issues years ago (I'm pretty sure also on
this list) so people probably know about them and possibly believed
the old behavior was by design like I did.

http://wiki.bash-hackers.org/syntax/arrays#bugs_and_portability_considerations.

Re: bug adding K,V pairs to existing hash with HASH+=([K]=V)

[Dan Douglas]

Ah so `arr+=([a]=x [b]=y)` will no longer be the same as `arr+=([a]+=x
[b]+=y)`? I never liked that for associative arrays because the only
workaround was to do multiple arr[key]= assignments to update or add
more than one element at a time.

My thinking was that this is due to bash's unique auto-incrementing of
indexed array keys. Since in ksh you're not allowed to specify an
index for one element of a compound assignment without specifying it
for all elements, bash has some additional things to consider.

 ~ $ bash-4.2 -c 'typeset -ia a=({0..5}); a+=([0]+=1 {2..4}); typeset -p a'
declare -ai a='([0]="1" [1]="3" [2]="5" [3]="7" [4]="4" [5]="5")'

Bash 4.4:
 ~ $ ./doc/programs/bash-build/bash -c 'typeset -ia a=({0..5});
a+=([0]+=1 {2..4}); typeset -p a'
declare -ai a=([0]="1" [1]="2" [2]="3" [3]="4" [4]="4" [5]="5")

I almost think it makes sense to treat ordered and unordered
collections differently. With an unordered collection an outer +=
should obviously mean "add or update for each assignment". For an
ordered collection when appending beginning at an index lower than the
max index I'm not so sure.

-- 
Dan Douglas

Re: `foo=1 declare -r foo' prints an error message but its exit status is zero

[Dan Douglas]

On Wednesday, September 9, 2015 2:17:30 PM CDT ziyunfei wrote:
> $ foo=1 declare -r foo
> bash: foo: readonly variable
> $ echo $?
> 0
> $ echo $foo
> 1
> 
> Is this a bug? 
> 
> $ bar=1 readonly bar; # throw an error in Bash 4.2, fixed in Bash 4.3

It's a bit more illustrative when you use `readonly' instead of `declare'. In
the former case bash only prints the error in non-POSIX mode because it
modifies the special builtin to unset its value when the command finishes.
`declare' will print an error in either mode. If there's any bug here it's that
bash fails to bail out and executes the echo command anyway.

 $ ( for sh in bash sh ksh mksh zsh dash bb posh; do printf '%-5s x=%5s 
status=%s\n' "${sh}:" "$("$sh" -c "\${ZSH_VERSION+false}||emulate sh; trap 
'echo \${x+unset}' EXIT; $(

signature.asc
Description: This is a digitally signed message part.

Re: Integer Overflow in braces

[Dan Douglas]

On Monday, August 17, 2015 04:15:50 PM Eric Blake wrote:
 On 08/17/2015 09:58 AM, Pasha K wrote:
  Hey Greg,
  
  I wasn't particularly trying to actually generate that large amount of
  strings in memory, I wa purposely trying to overflow the integer variable
  nelemhoping to get Code Execution. This could potentially be a security
  risk as shell shock was just more of a denial of service rather than
  straight up code execution. However, just because I wasn't able to gain
  control of the registers doesn't mean someone else with more skill can't.
 
 This is not a security risk.
 
 Shell shock was a security hole because the shell could be coerced into
 executing user-supplied code WITHOUT a way for a script to intervene.
 
 Any poorly-written shell script can do stupid things, including crashing
 bash because it overflows the heap by trying to allocate memory for such
 a stupidly large expansion.  But unless the problem can be triggered
 without a script (the way shell shock executed user code before even
 starting to parse a script), then you can't exploit the problem to gain
 any more access to the system than you already have by being able to run
 a script in the first place.
 
 Fix your script to not do stupid things, like trying an insanely-large
 brace expansion, or trying an 'eval' (or similar) on untrusted user
 input. But don't call it a bash security hole that bash allows you to
 write stupid scripts.
 
 

IMHO the issue of whether the integer is allowed to overflow is separate from 
the question of whether the resulting expansion is too big. Code that does 
an `eval blah{0..$n}` is reasonably common and not necessarily stupid. 
-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.

Re: Parameter Expansion: Case Modification: ${var~} not documented

[Dan Douglas]

On Tuesday, August 18, 2015 9:54:55 AM CDT Isaac Good wrote:
 Would you mind sharing the rational behind having it undocumented?

Since I like guessing: the syntax for parameter expansion operators is 
currently non-extensible, so the namespace of terse operators is in limited 
supply. New syntax should be extensible to suit future needs while keeping the 
language minimal. This is new syntax that adds one function that will be 
rarely used. I can think of better ways to use that operator.

The operators in use currently are already a disaster. We *really* could use a 
solution for the circumfix operators `!var[@]` and `!var[*]` that collide with 
the prefix `!` operator, and for reasons unknown don't interoperate with any 
of the other expansions such as array slicing / subscripting. I wouldn't want 
to add new (pointless) syntax before the fundamental problems are addressed.

-- 
Dan Douglas

Re: quoted compound array assignment deprecated

[Dan Douglas]

Sorry I meant to reply to that thread but ran out of time. I think Stephane's 
eventual proposal was pretty close to what I had in mind but expressed badly. 
I'm not sure why it was eventually decided to deprecate the current system 
entirely but I'm not opposed to the idea - especially if it provides no 
functionality for which there aren't easy workarounds.

The only thing I'm actively abusing this for at the moment in scripts I 
actually use is as a way of encoding 2D arrays. It's very much a read-only 
datastructure too.

~ $ ( key1=foo key2=bar; declare -A a=([foo]='([bar]=baz)') b=${a[$key1]}
typeset -p a b; echo ${b[$key2]} )
declare -A a='([foo]=([bar]=baz) )'
declare -A b='([bar]=baz )'
baz

Any change will likely break this property but I think wrapping it in eval 
gives the same result.

-- 
Dan Douglas

Re: Integer Overflow in braces

[Dan Douglas]

On Tuesday, August 18, 2015 09:04:33 AM Greg Wooledge wrote:
 On Tue, Aug 18, 2015 at 07:54:48AM -0500, Dan Douglas wrote:
  IMHO the issue of whether the integer is allowed to overflow is separate 
from 
  the question of whether the resulting expansion is too big. Code that 
does 
  an `eval blah{0..$n}` is reasonably common and not necessarily stupid. 
 
 Yes, that's fine.  But I don't actually understand what kind of overflow
 Pasha K was actually trying to test for.  He/she mentioned nelem, which
 only appears in two places in the bash source code: once in indexed
 arrays, and once in associative arrays.  But there were no arrays in
 the script being executed.
 
 {0..} should produce an error because it runs out of
 memory.  So I would expect to see a malloc failure, or something similar.
 If Pasha is saying that an integer overflow occurs before the malloc
 failure, then that may or may not be interesting to Chet.  If it crashes
 bash, then it's not interesting to me, because the inevitable malloc
 failure would have crashed it if the overflow didn't.  It only becomes
 interesting to me if the integer overflow causes some weird behavior to
 happen BEFORE bash crashes.
 

Actually I think I spoke too soon. There's already some considerable logic in 
braces.c to check for overflow (e.g. around braces.c:390 shortly after 
declaration of the int). Looks like there were some changes in this code last 
year to beef it up a bit. (see commit 
67440bc5959a639359bf1dd7d655915bf6e9e7f1). I suspect this is probably fixed in 
devel.
-- 
Dan Douglas

Re: Parameter Expansion: Case Modification: ${var~} not documented

[Dan Douglas]

On Tuesday, August 18, 2015 08:50:51 AM Dan Douglas wrote:
 I'm pretty sure that's intentional. The corresponding `declare -c` has never 
 been documented either.
 

Hrm, it doesn't correspond actually. declare -c just capitalizes the first 
letter of the string.

Another thing about the ${var~} expansions is I wonder why it isn't just built 
in to the substitution expansion. The `~` is obviously inspired by the vim 
movement to toggle caps. Given `foobarbaz`, vim can also do `:s/foo\zs\(bar\)
\zebaz/\U\1/` and yield `fooBARbaz`. This is much more powerful, though it 
requires bash to start supporting backrefs in substitutions.

There's also this ksh feature I've never found a use for:

$ ksh -c 'x=foobarbaz; typeset -M toupper x; echo $x'
FOOBARBAZ

I don't know, the only purpose is to replace `typeset -l/-u` and allow for 
other towctrans operations.

-- 
Dan Douglas

Re: unset does not act as expected on namerefs

[Dan Douglas]

On Tue, Jun 2, 2015 at 7:31 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 There's declare -i, but no sane person USES that, so we can ignore it.

While in bash `-i` likely either has no effect or slows things down
very slightly, in ksh93 and probably zsh it's a huge performance win
because it prevents integers and floats from being automatically
coerced to and from a string on every operation. It also guarantees
(after the attribute is set) that the value will never be anything
other than the right type unless the attribute is explicitly removed.
It's perfectly sane to use it for values that are always guaranteed
correct by construction or by validation. (Use the `integer' and
`float' aliases in ksh, which are preset to use the largest currently
available type).

So long as you always remember to initialize the variable you can't
really go wrong with it.

Re: Request to a new feature on read

[Dan Douglas]

On Thu, Apr 16, 2015 at 9:50 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 I don't see why such features should be compiled into bash's read builtin.
 I'd have no problem with adding better splitting/joining/parsing features
 in a more general context, probably operating on a string variable, but
 certainly not operating on a file descriptor.

I don't think they should be part of `read` either. Some way to extend
the BASH_REMATCH mechanism would be better.

 Doesn't the underlying C library only guarantee you a single character of
 lookahead when reading?  (Or maybe a single byte.  I'm way out of date.
 My knowledge of C comes from the days when char = byte.)  You can't do
 all this fancy perl-RE-style lookahead stuff on a stream with only a
 single byte/char of lookahead.

Hm, maybe you're referring to ungetc? IIRC one byte is the only
guarantee when dealing with pipes. I don't really care about having it
pattern match while reading a stream. To make that work well would
probably involve mmap (and even then, only on regular files).

Probably the most portable way to support fancier regex is to call
into std::regex. Any system with a modern C++ compiler should support
ECMAScript regex, which is close to a superset of ERE.

Re: Request to a new feature on read

[Dan Douglas]

On Thu, Apr 16, 2015 at 9:32 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Thu, Apr 16, 2015 at 09:29:56AM -0500, Dan Douglas wrote:
 I find myself in need of something along the lines of Python's
 `re.split` and `re.findall` all the time. E.g. splitting an ip into an
 array of octets.

 IFS=. read -ra octets  $ip

Sure, but validation is then separate if needed. There are plenty of
applications where you want either a multi-character or non-static
delimiter, possibly with pattern matching on the data at the same
time.

Re: Request to a new feature on read

[Dan Douglas]

On Thu, Apr 16, 2015 at 8:55 AM, Eduardo A. Bustamante López
dual...@gmail.com wrote:
 Any reason to justify this instead of using a simple loop?

I find myself in need of something along the lines of Python's
`re.split` and `re.findall` all the time. E.g. splitting an ip into an
array of octets.

On Thu, Apr 16, 2015 at 5:49 AM, Valentin Bajrami
valentin.bajr...@gmail.com wrote:
 Hi,

 According to ''help read'' we can specify  -N[chars]  to trigger return
 automatically.  Is it possible to approach read differently?

 For example:  $re is some regular expression

FWIW, ksh has two redirect operators that can be used together with
`read` to get something like this. They're somewhat difficult to use
IMO:

#pattern Seeks forward to the beginning of the next line
containing pattern.

##patternThe same as # except that the portion of the
file that is skipped is copied to standard output.

-- 
Dan Douglas

Re: [Help-bash] make function local

[Dan Douglas]

 On Thu, Apr 9, 2015 at 5:40 PM, Peng Yu pengyu...@gmail.com wrote:
  However, the real problem is that global variable make larger programs
  difficult to maintain. Therefore, the problem becomes severe for
  people using bash for making bash libraries. Any ad hoc use of bash
  can always get away from this problem.

 The solution is namespaces. ksh93 supports namespace blocks (including nested
 namespaces), though it's still quite buggy.

 Note C doesn't have namespaces either and gets by despite being used as
 a library language to a much greater extent than bash. It's worked around with
 naming conventions. There are many prefix conventions even within the
 C standard library and POSIX for various headers - atomic_*, mtx_*, pthread_*,
 sem_*, etc.

 On Fri, Apr 10, 2015 at 8:54 AM, Peng Yu pengyu...@gmail.com wrote:
  You don't know functional programming, do you? That example is
  functional program (not full fledged though) but not OO.
 
  It seems that you don't know its importance, either? With functional
  programming, you basically can make OO on top of it, but not the other
  way around. R has examples on how to make OO using functional
  programing.

 Functional programming is a style, not a language feature. Features that
 facilitate it are common in OO langs.

 C# implements first-class functions with closures by translating them in the IL
 to inner classes that maintain the lexical state. I'm pretty sure Java 8 does
 essentially the same with its lambdas. People have been using the inner class
 trick for ages prior to that.

  function as first class citizen is just one of the feature that makes
  javascript so successful now days.

 The type of function you're talking about isn't merely a local function.
 JavaScript doesn't exactly have local functions, and probably isn't the best
 example because of its numerous ways of defining functions. It also doesn't
 have a separate namespace for functions. No matter how you define them, you're
 likely using sugar that in the end results in a function object that's
 accessed through an identifier that's in the same namespace as any other
 variable (which might be local).

 Bash needs several major features before it could even think about supporting
 function objects.

 First it needs yet another type of function (probably with yet another syntax)
 which supports returning a value, and declaration as an expression so it can
 be passed around and treated as a value.

 At the same time it needs to add some sort of type system so that data other
 than strings (objects or functions) can be bound to variables and formal
 parameters.

 ksh93 has the beginnings of an object system which supports user-defined types,
 but functions are still insufficient for doing anything functional. Some of
 its basic ideas are nice but I have pretty much given up on understanding David
 Korn's design. It's obviously half-baked in its current state and hard to use
 for getting real work done. It's better than nothing I guess.

 Anyway, you can probably do something resembling FP with some combination of
 `typeset -T` wrappers to contain your functions, and `typeset -M` and `-C` to
 move and copy objects around by reference. It's not pretty.

 --
 Dan Douglas

Fwd: [shellcheck] Bash arithmetic expansion diagnostics (#255)

[Dan Douglas]

Hi, This was noticed in a comment to a github issue. Just forwarding it along.

$ for ((; 1(1) ;)); do break; done
 -bash: syntax error near unexpected token `newline'

$ echo $BASH_VERSION
4.3.33(1)-release

https://github.com/koalaman/shellcheck/issues/255#issuecomment-88452555

-- Forwarded message --
From: Dan Douglas orm...@gmail.com
Date: Wed, Apr 1, 2015 at 12:15 PM
Subject: Re: [shellcheck] Bash arithmetic expansion diagnostics (#255)
To: koalaman/shellcheck
reply+0003e7c2a607770e1564420080ff167a99c7ac0c55c501de92cf00011133a16592a169ce02dfd...@reply.github.com


On Wed, Apr 1, 2015 at 7:07 AM, brother notificati...@github.com wrote:

 I walked straight into one of these. It looks like something might have 
 changed in bash on top of everything else because it was not new code that 
 exploded in my face, but I have indeed a new bash version on this machine 
 since last time I tried it.

 brother ~$ cat /tmp/test.sh
 #!/bin/bash

 slots=14

 unset slotaddr
 for ((i=0x82 ; i((0x82+(slots*2))) ; i=i+2 )); do
 slotaddr+=($i)
 done

 echo ${slotaddr[@]}
 brother ~$ shellcheck /tmp/test.sh
 brother ~$ /tmp/test.sh
 /tmp/test.sh: command substitution: line 9: syntax error near unexpected 
 token `slots*2'
 /tmp/test.sh: command substitution: line 9: `(0x82+(slots*2))) ; i=i+2 '
 /tmp/test.sh: line 6: syntax error: arithmetic expression required
 /tmp/test.sh: line 6: syntax error: `((i=0x82 ; i((0x82+(slots*2))) ; i=i+2 
 ))'

 on bash 4.3.30(1)-release


That's a bug. There were some changes to the way C-for was parsed
during 4.2. Or it's possibly related to when process substitution was
removed from certain arithmetic contexts during that time.

Re: Fwd: [shellcheck] Bash arithmetic expansion diagnostics (#255)

[Dan Douglas]

On Wed, Apr 1, 2015 at 1:01 PM, Eduardo A. Bustamante López
dual...@gmail.com wrote:
 Already fixed:

 dualbus@yaqui:~/local/src/gnu/bash$ for ((; 1(1) ;)); do break; done
 dualbus@yaqui:~/local/src/gnu/bash$
 GNU bash, version 4.4.0(1)-devel (x86_64-unknown-linux-gnu)

 On Wed, Apr 01, 2015 at 12:53:16PM -0500, Dan Douglas wrote:
 Hi, This was noticed in a comment to a github issue. Just forwarding it 
 along.

 $ for ((; 1(1) ;)); do break; done
  -bash: syntax error near unexpected token `newline'

 $ echo $BASH_VERSION
 4.3.33(1)-release

 https://github.com/koalaman/shellcheck/issues/255#issuecomment-88452555

Oh awesome, nevermind then! :)

Re: definite way to determine the running shell

[Dan Douglas]

I just test for version strings. e.g. BASH_VERSION. It isn't ideal or
robust but gets the job done depending on your requirements. There is
no generally great solution.

There are scripts that make better guesses but there aren't any that
are meant for use in a library or that do fine-grained feature
detection. Something like that is on my todo list for some day.

https://www.mirbsd.org/cvs.cgi/contrib/code/Snippets/getshver?rev=HEAD
http://www.in-ulm.de/~mascheck/various/whatshell/whatshell.sh.html
http://stchaz.free.fr/which_interpreter

-- 
Dan Douglas

Re: definite way to determine the running shell

[Dan Douglas]

On Fri, Mar 27, 2015 at 2:37 PM, Christoph Anton Mitterer
cales...@scientia.net wrote:
 On Fri, 2015-03-27 at 01:44 -0600, Eduardo A. Bustamante López wrote:
 Why would the user set BASH_VERSION in the environment? That sounds like 
 asking
 for trouble!
 Murphy's law! :D

 - try to unset BASHOPTS (did it work? then it's not bash)
 - Now, store the value of BASHOPTS in a temp var
 - Set a shopt option that's not set already in BASHOPTS (or unset one from
 BASHOPTS)
 - Check if the value of BASHOPTS changed.
 that sounds like a nice idea.

This might not be a great idea. Even if it works now it's not
future-proof. Some shells have implemented BASH-prefixed variables for
the sake of compatibility. I understand why they did it and don't
think it was a bad idea, it just means you can't rely on their
presence to mean you're using bash.

ksh93's bash mode and mksh both have examples of this

 $ exec -a bash ksh -c 'typeset -p' | grep BASH
BASH=bash
BASH_EXECUTION_STRING='typeset -p'
typeset -r -a BASH_VERSINFO=(4 2 0 0 ksh93 i386-unknown-linux)
BASH_VERSION='4.2.0(0)-ksh93'
typeset -n BASH_REMATCH='.sh.match[0]'
typeset -n BASH_SOURCE=file
typeset -n BASH_SUBSHELL=subshell

 $ mksh -c 'typeset -p' | grep BASH
typeset -i -U BASHPID=1207

-- 
Dan Douglas

Re: Feature: disallow |

[Dan Douglas]

On Tue, Mar 17, 2015 at 11:40 AM, Ed Avis e...@waniasset.com wrote:
 In bash, the empty command can be redirected to a file:

 % out

 It will create or overwrite 'out' as the empty file.  That may be useful.
 But somehow you can also say

 % whoami | out

This isn't surprising or deserving of a special case treatment. It's a
valid permutation that I would expect to work as required.

This isn't even close to the most likely way one might screw up a
redirect. Forgetting a quote or escape in a non-obvious place such as
within an assignment or argument to a command is far more likely to
bite you. Examples:

var=fooout
let x5\?y:z

If you're worried about making a mistake, enable the noclobber option
in your bashrc.

-- 
Dan Douglas

Re: Experiment to provide a mirror of bash with detailed git history

[Dan Douglas]

On Mon, Mar 16, 2015 at 3:54 PM, Eduardo A. Bustamante López
dual...@gmail.com wrote:
 I know that some people are interested in a more detailed commit history in
 bash's git repository. After all, it's easier for all of us to just learn to
 use a tool, and use that for everything.

How are you planning on creating a more detailed commit history than
the history that's already there? You're going through the changelog
and guessing about how code changes map to certain changelog entries?
I'm not sure how well that can work.

I've thought about ways to improve the situation but I'm not sure
about how to improve commit detail when almost all the changes come in
periodic code drops. You would have to do a lot of guesswork.

Starting with a better workflow now is certainly possible. There
should ideally be a stable branch for each stable release (4.0, 4.1,
...) forked from master from the correct points. Development of patch
releases should of course be done in their own branches and merged
into stable branches. New features would ideally have per-feature
branches that get merged to master rather than a single devel branch
that tends to be way ahead of master.

Maybe you have better ideas. It's probably best to do whatever you
plan on your own branch if possible.

-- 
Dan Douglas

Re: how to search for commands

[Dan Douglas]

On Mon, Feb 23, 2015 at 10:50 PM,  garegi...@gmail.com wrote:
 How do you search for commands? In powershell you have the get-command 
 cmdlet. Is there anything equivalent in unix?

Depends on the type of command. For shell builtins, bash has `help':

$ help '*ad'
Shell commands matching keyword `*ad'

read: read [-ers] [-a array] [-d delim] [-i text] [-n nchars] [-N
nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
Read a line from the standard input and split it into fields.
...

To search for commands found in PATH (or functions or aliases) use
`type'. See `help type' for how to use it.

Searching for commands by package is OS-specific. e.g. in Gentoo
`equery f -f cmd pkg' will show commands belonging to a package.
Cygwin's equivalent is `cygcheck -l'. Pretty much every distro has
something similar.

-- 
Dan Douglas

Re: array subscripts act differently for integers(ie. let)

[Dan Douglas]

On Thu, Feb 19, 2015 at 7:47 PM, Chet Ramey chet.ra...@case.edu wrote:
 On 2/18/15 4:14 PM, emanuelczi...@cryptolab.net wrote:
 tl;dr: thanks! no reply needed;

 Thanks guys. I had a hard time accepting that this is how it's supposed to
 work.
 I accepted it now. :)

 Yeah.  The expression between (( and )) is word expanded, since (( ... ))
 is supposed to be exactly equivalent to let 

 The real issue is that assignment statements in arithmetic expressions
 that contain array references are also word expanded, almost as if they
 were executed in an assignment statement context.  This is how bash has
 always behaved, though; backwards compatibility is a concern.

The way bash works right now allows safely handling arbitrary
associative array keys. This is a pretty good illustration:

$ bash -c 'key=\]; typeset -A a=([$key]=x [y]=y); unset -v
a[\${key}]; typeset -p a'
declare -A a='([y]=y )'
$ ksh -c 'key=\]; typeset -A a=([$key]=x [y]=y); unset -v
a[\${key}]; typeset -p a'
typeset -A a=([']']=x [y]=y)

Since ksh doesn't do parameter expansion while resolving variables
quite consistently it's impossible to handle this case. (I'll have to
ask about it on their list one of these days).

IMO bash is exactly correct. I don't know how this could be fixed in
a way that would satisfy people without changing something very
fundamental. If you disable the initial expansion to `(())', then
`(($x))' wouldn't work (because arithmetic evaluation doesn't itself
evaluate parameter expansions). However if you disabled evaluating
parameter expansions during variable resolution (for array indexes)
then you would be stuck with exactly the above ksh problem.

It's unfortunate people don't understand this, but when you think
about it it can't really work any other way.

-- 
Dan Douglas

Re: array subscripts act differently for integers(ie. let)

[Dan Douglas]

it occurs when an associative array with an empty subscript is modified.

$ ( typeset -A a; x='a[$()]++'; ((x)); )
-bash: a: bad array subscript
-bash: a[$()]: bad array subscript
Segmentation fault (core dumped)

I could live without the error on an empty key to begin with. It can
be hard to protect against errors. Although bash doesn't throw
exceptions on referencing any undefined index like many languages, you
still have to guard against an empty key whenever its value isn't
known.

${key+${a[$key]}}, or else ${a[${key}_]} followed by
${a[${key%_}]} on each usage. Or at minimum [[ $key ]] on every
iteration of a loop over ${!a[@]}. It's almost uglier than `set -u'
workarounds...

-- 
Dan Douglas

Re: declare a=$b if $a previously set as array

[Dan Douglas]

So I'm still getting caught up on this thread, but hasn't this issue been done
to death in previous threads? Back when I was examining bash's strange declare
-a, you were the very first person I found to notice its quasi-keyword behavior
10 years ago
(https://lists.gnu.org/archive/html/bug-bash/2004-09/msg00110.html). Perhaps
you didn't realize what this was doing at the time?

To me the biggest problem is what happens when you explicitly request a scalar
assignment. (I even specified a[1] to ensure it's not an a vs. a[0] issue.)

bash -c 'typeset -a a; b=(foo bar); typeset a[1]=$b; typeset -p a; 
printf %s  ${a[@]}; echo'
declare -a a='([0]=foo [1]=bar)'
foo bar

This doesn't fit with my understanding of how it should work. Otherwise I think
the way declare's arguments are interpreted based upon their form and current
set of attributes is pretty well understood, albeit undocumented.

I agree that ksh sometimes makes more sense. I'd add that although ksh's
typeset will never evaluate its non-literal parts of assignment arguments as
assignment syntax after the initial expansion, in ksh it isn't possible to
modify the _type_ of a variable after declaration to begin with, because ksh
has a confusing distinction between types and attributes. Every time you
use a _different_ declaration command, you're wiping all attributes and
effectively unsetting then declaring a whole new variable, while attributes
remain mutable.

 # In this case we're modifying attributes
$ ksh -c 'typeset -a x=(foo); print ${@x}; typeset -r x=; print 
${@x}'
typeset -a
typeset -r -a

 # In this case we're modifying an attribute, then declaring a new 
variable
 # (and then implicitly modifying its attribute, though ${@a} fails to
 # report it correctly).

$ ksh -c 'typeset -a a=(foo); print ${@a}; typeset -T A=(typeset x); 
A a=(x=foo); print ${@a}; a+=(x=bar); print ${@a}; typeset -p a'
typeset -a A
A
A -a a=((x=foo) (x=bar))

If it were redesigned today there would surely be no attributes, only types.

-- 
Dan Douglas

Re: declare a=$b if $a previously set as array

[Dan Douglas]

Sorry I did a bad job copy/pasting then editing that last example.

ksh -c 'typeset -a x=(foo); print ${@x}; typeset -T A=(typeset y); A 
x=(y=foo); print ${@x}; x+=(y=bar); print ${@x}; typeset -p x'
typeset -a
A
A
A -a x=((y=foo) (y=bar))

-- 
Dan Douglas

Re: Bash bug - in read -e -r var

[Dan Douglas]

On Sunday, December 14, 2014 04:27:51 PM Chet Ramey wrote:
 On 12/13/14 12:06 AM, Daniel A. Gauthier wrote:
  
  If you do a read -e -r var in bash, and use tab completion to fill-in
  a filename, it fills in My\ File\ Name.txt and that's exactly
  what's read into the variable.  The fix is simple, in tab completion,
  the text placed into the buffer should be in the same format that the
  read is expecting, so if -r is in effect for the read operation, the
  the filename should be added to the buffer unquoted.
 
 This seems like an awfully specific case (-r in effect, one or fewer
 variables specified) to change readline for.  I question whether it's of
 general interest.
 
 
I'm generally interested in what read with (or without) -r combined with -e 
even means. I understand that read's now basically useless behavior without -r 
was originally there to make using it interactively without a line editor 
slightly easier.

-- 
Dan Douglas

Re: Bash bug - in read -e -r var

[Dan Douglas]

On Monday, December 15, 2014 10:47:29 AM Chet Ramey wrote:
 On 12/15/14, 7:11 AM, Dan Douglas wrote:

  I'm generally interested in what read with (or without) -r combined with -e
  even means.

 I'm not sure what you're driving at.  The -e option says how to read the
 line; the -r option affects how the characters are processed after being
 read.


Ah ok I thought -e basically negated the effect of -r for some reason but I
guess not (like you can still use \ to escape line continuations without -r it
seems). Long ago I assumed read str was equivalent to read -r $'str'
based upon what the manual says.

BTW I like how show-mode-in-prompt On together with read -ep prompt shows
the indicator. That's a nice touch.
--
Dan Douglas

Re: declare a=$b if $a previously set as array

[Dan Douglas]

On Wednesday, December 10, 2014 10:43:45 AM Stephane Chazelas wrote:
 David Korn mentions that local scoping a la ksh was originally rejected by
 POSIX because ksh88 originally implemented dynamic scoping (like most
 shells now do, but most other languages don't). ksh changed to lexical
 scoping in ksh93 (probably partly to address that POSIX /requirement/) and
 now everyone else wants dynamic scoping because that's how most shells have
 implemented it in the interval.

A local builtin and dynamic scope is in the current beta of ksh.

I don't care very much for how it works though. It's only enabled as a part of
bash mode, which actively disables a bunch of ksh features and doesn't allow
you to pick and choose. Bash mode is all or nothing at least as of the last
snapshot, with no tunables like zsh's setopt.

 $ ksh -c 'function f { typeset x=in f; g; typeset -p x; }; function 
g { x=in g; }; x=global; f; typeset -p x'
x='in f'
x='in g'

 $ ( exec -a bash ksh -c 'function f { typeset x=in f; g; typeset -p 
x; }; function g { x=in g; }; x=global; f; typeset -p x' )
x='in g'
x=global

I would have much preferred if David Korn followed his original proposal of
adding dynamic scope for only POSIX-style function definitions across the
board. At least porting bash scripts will be slightly easier in the very
specific case that ksh is invoked correctly.

-- 
Dan Douglas

Re: declare a=$b if $a previously set as array

[Dan Douglas]

On Sunday, December 14, 2014 02:39:29 PM Chet Ramey wrote:
 And we get to the fundamental issue.  Is it appropriate to require
 arguments to declaration commands to be valid assignment statements when
 the parser sees them, instead of when the builtin sees them, at least
 when using the compound assignment syntax.
 
 I'm ok with tightening the rules and saying that it is, but it's not
 backwards compatible and changes the behavior of things like
 
 declare -a foo='(1 2 3)'
 
 which I imagine plenty of people have used and has been a supported way
 to do compound assignment for many years.

That would be one way but I think this can be solved without going quite
so far. How do you feel about these rules?

1. If a word that is an argument to declare is parsed as a valid
   assignment, then perform the assignment immediately as it appears
   lexically. If such a word is parsed as a simple assignment (with or
   without an index) then bash treats it as a simple scalar assignment 
to
   the variable or array element as any ordinary assignment would. If 
word
   is parsed as an assignment with array initializer list then bash 
treats
   it as such.

2. Any words that do not parse as valid assignments lexically during 
step
   1 undergo the usual set of expansions for simple commands and the
   resulting words are saved.

3. After evaluation of all words from steps 1-2, those that have 
undergone
   expansion per 2 are passed as arguments to declare. Declare then 
parses
   each of these arguments, testing for assignments that became valid as
   a result of expansion. During this step, for assignments to variables
   that have an array attribute set, or for all assignments if declare 
has
   been given the -a or -A option, declare will treat assignments that 
have
   initializer lists as array assignments accordingly. Otherwise, 
declare
   will treat all assignments as scalar assignments to a variable or 
array
   element. Words that still fail to parse as valid assignments are an
   error.

I think this covers all the bases and is very close to the current behavior
with one exception. All of the problems Stephane and I have brought up deal
with the situation where a word that parses lexically as a scalar assignment
isn't treated as such, so that is the only required change. I don't think
that change will break many scripts.

Some observations:

 * This solves the problem without breaking backwards compatibility and allows
   declare -p output to continue to function.

 * These rules will make the following become compatible with current ksh
   versions:

 $ ksh -c 'function f { typeset x[1]=(foo bar); typeset -p x; }; f'
 typeset -a x=([1]='(foo bar)')

 $ bash -c 'function f { typeset x[1]=(foo bar); typeset -p x; }; f'
 declare -a x='([0]=foo [1]=bar)'

 * Treating words that parse as scalar assignments as scalar assignments is
   consistent with bash's usual treatment of a[idx]=(foo). If bash sees
   declare foo[1]=(foo), it should print:

   bash: foo[1]: cannot assign list to array member.

 * Note the fine print in step 3. It deals with this case in bash's current
   behavior and doesn't need to change (-a vs no -a):

$ bash -c 'x=[4]=foo [6]=bar; declare -a y=($x); declare -p y'
declare -a y='([4]=foo [6]=bar)'

$ bash -c 'x=[4]=foo [6]=bar; declare y=($x); declare -p y'
declare -- y=([4]=foo [6]=bar)

-- 
Dan Douglas

Re: [RFC] Logically composable variant of errexit

[Dan Douglas]

I would still propose that a simple and powerful way to extend Bash with
exception handling would be to extend the ERR trap by passing it some metadata
about the type and location of the exception incurred so that it can be handled
by user code. This proposal allows us to largely avoid having to answer the
question of when and where does it make sense for this to get triggered? for
the user.

It might require a new type of trap that just fires on anything that returns
false or fails so that the user handler can choose what to do about it in
various contexts.

A more ambitious proposal might be to try cloning zsh's try/catch. It would be
familiar to programmers and of course having a pre-existing implementation is a
plus. try/catch is probably more intrusive than my first proposal due to the
additional keywords and control flow changes.

-- 
Dan Douglas

Re: bash uses tmp files for inter-process communication instead of pipes?

[Dan Douglas]

On Monday, October 06, 2014 02:00:47 PM Linda Walsh wrote:
 
 Greg Wooledge wrote:
  On Mon, Oct 06, 2014 at 12:38:21PM -0700, Linda Walsh wrote:
  According to Chet , only way to do a multi-var assignment in bash is
 
  read a b c d  $(echo ${array[@]})

It's best to avoid turning your nice data structure into a stream only to read 
(and mangle) it back again.

If you must, create a function.

function unpack {
${1+:} return 1
typeset -n _arrRef=$1 _ref
typeset -a _keys=(${!_arrRef[@]})
typeset -i n=0
shift

for _ref; do
${_keys[n]+:} return
_ref=${_arrRef[${_keys[n++]}]}
done
}

#  $ a=(1 2 3) e=hi; unpack a b c d e; typeset -p {a..e}
# declare -a a='([0]=1 [1]=2 [2]=3)'
# declare -- b=1
# declare -- c=2
# declare -- d=3  

  
# declare -- e=hi

 Um... it used a socket.. to transfer it, then it uses a tmp file on top
 of that?!  :

That's the wrong process. AF_NETLINK is the interface used by iproute2 for IPC 
with the kernel. Bash doesn't use netlink sockets.

  So why would someone use a tmp file to do an assignment.
  
  It has nothing to do with assignments.  Temp files are how here documents
  (and their cousin here strings) are implemented.  A here document is a
  redirection of standard input.  A redirection *from a file*
 
   Nope:
   This type of redirection instructs the shell to  read  input  from  the
 current source until a line containing only delimiter (with no 
trailing
 blanks) is seen.  All of the lines read up to that point are then  
used
 as the standard input for a command.
 
 The current source -- can be anything that input can be
 redirected from -- including memory.
 
 
  in fact,
  albeit a file that is created on the fly for you by the shell.
 
 Gratuitous expectation of slow resources...  Non-conservation
 of resources, not for the user, but for itself.
 
 Note:
  a=$(foo)
  echo $a
 one
 two
 three
 ---
 no tmp files used, but it does a file read on foo.
 
 b=$a
 -- the above uses no tmp files.
 
 b=$(echo $a)
 ---
 THAT uses no tmp files.
 
 But
 b=$a
 
 uses a tmp file.
 
 That's ridiculously lame.

That isn't working code the way you think (except in mksh). In most shells its 
just a null command that sets b to empty.

Just perform the assignment directly. There's no need for any special 
expansion or redirect.

  So why would a temp file be used?
  
  Historical reasons, and because many processes will expect standard input
  to have random access capabilities (at least the ability to rewind via
  lseek).  Since bash has no idea what program is going to be reading the
  here document, why take chances?
 
   How could you use a seek in the middle of a HERE doc
 read?
 
 Not only that, it's just wrong./dev/stdin most often comes from
 a tty which is not random access.

There is no requirement for how heredocs are implemented. Temp files are 
perfectly reasonable and most shells use them. The shell reads the heredoc, 
but it is the command that would be doing any seeking, or possibly the shell 
itself via nonstandard features like the #(()) redirect.

This can be an optimization. Consider:

for file in file1 ... filen; do
ed -s $file /dev/fd/0
done \EOF
ed script...
wq
EOF

In the above case, bash doesn't actually have to re-create the input 
repeatedly, it just creates a new FD, effectively rewinding the input.

-- 
Dan Douglas

Re: REGRESSION: shellshock patch rejects valid function names

[Dan Douglas]

Just a few points to add.

On Monday, September 29, 2014 04:29:52 PM Stephane Chazelas wrote:
 2014-09-29 09:04:00 -0600, Eric Blake:
 [...]
   The function is named fname; the application shall ensure that it is a
   name (see XBD Name) and that it is not the name of a special built-in 
utility.
   
   
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05

This doesn't normally matter because POSIX requires special builtins to take 
precedence over functions during command search, so even if you have such a 
function defined it is impossible to call. Bash doesn't use the correct search 
order however.

Mksh has the reverse bug. It allows defining the function (wrongly) but then 
calls the special builtin anyway (correctly).

Another bug is in ksh93 whose `builtin` allows disabling special builtins 
(which according to the manual, shouldn't work).

$ ksh -c 'builtin -d set; function set { echo test; }; set'
test

Bash's enable correctly disallows that.

 I agree the requirement is on the application, and I can't see
 why POSIX should force a shell to reject a function whose name
 doesn't contain a valid identifier.
 ...

Another thing you can do in bash is bypass its command name check by using a 
null zeroth word.

$ { function } { echo test; }; () }; }
test

Ordinarily } would be uncallable, but apparently since bash only checks the 
command name of the first word, calling with e.g. `() }` or `$() }` works.

-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.

Re: REGRESSION: shellshock patch rejects valid function names

[Dan Douglas]

On Tuesday, September 30, 2014 12:11:15 AM Andreas Schwab wrote:
 Dan Douglas orm...@gmail.com writes:
 
  Another thing you can do in bash is bypass its command name check by using 
  a 
  null zeroth word.
 
  $ { function } { echo test; }; () }; }
  test
 
  Ordinarily } would be uncallable,
 
 Only literal } is special, you can use \} instead, or store it in a
 variable.

Yeah I forgot there are a few ways. `0 }` is yet another.

-- 
Dan Douglas

Re: REGRESSION: shellshock patch rejects valid function names

[Dan Douglas]

On Friday, September 26, 2014 02:17:03 PM Eric Blake wrote:
 Ugg. Anyone advising people to write bash functions while using the
 obsolete keyword 'function' ought to be questioned about their advice in
 general.

It's mainly the combination of `function` and `()` that make no sense in any 
context.

I usually prefer the function keyword especially for bash functions, but it's 
entirely for portability reasons that not a lot of people care about. (this 
opinion could also change if the ksh dynamic scope option becomes less buggy 
before the next version, or dash were to make some improvements to make me 
care about its locals, among other things.)

-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.

Re: Issues with exported functions

[Dan Douglas]

On Thursday, September 25, 2014 09:03:03 AM Chet Ramey wrote:
 On 9/25/14, 4:52 AM, Gabriel Corona wrote:
  Hello,
  
  As the interface is not specified, would it make sense to:
  
   * add a prefix (use BASH_FUNCTION_foo instead of foo for exported
 function foo);
  
   * still expand the variable if it matches the 'exported function'
 pattern.
 
 Yes, that's one of the approaches under consideration.  It raises the
 bar for abuse by requiring that an attacker be able to create environment
 variables with arbitrary names as well as values.  It is not,
 unfortunately, backwards compatible.
 

Have you considered the FPATH mechanism? Exploiting it requires being able to 
create files and set FPATH accordingly. I've had some success with the 
function loader code in examples/functions/autoload.*. I believe it serves 
mostly the same purpose as exported functions.

-- 
Dan Douglas

Re: Issues with exported functions

[Dan Douglas]

On Thursday, September 25, 2014 10:29:16 AM Chet Ramey wrote:
 On 9/25/14, 10:25 AM, Dan Douglas wrote:
 
  Have you considered the FPATH mechanism? Exploiting it requires being able 
to 
  create files and set FPATH accordingly. I've had some success with the 
  function loader code in examples/functions/autoload.*. I believe it serves 
  mostly the same purpose as exported functions.
 
 I have thought about it, but it is even less backwards compatible than
 the other suggestions.

True.

I guess one key point is that it's lazy and only gets checked if a command 
doesn't exist. A user could implement function import entirely by means of 
command_not_found_handle - searching only for a function with the name given 
in $1, which should be inherently more secure than looking at every variable 
at startup.

-- 
Dan Douglas

Re: 'declare' does not honor '-e' in command substituted assignments - a bug ?

[Dan Douglas]

On Saturday, August 09, 2014 04:34:11 PM Jason Vas Dias wrote:
 Good day bash list -
 
 I don't understand why this emits any output  :
  $ ( set -e;  declare  v=$(false); echo 'Should not get here'; )
  Should not get here
  $
 
 While this does not:
  $ ( set -e;   v=$(false); echo 'Should not get here'; )
  $

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01

If there is no command name, but the command contained a command
substitution, the command shall complete with the exit status of the 
last
command substitution performed. Otherwise, the command shall complete 
with
a zero exit status.

The declaration commands return the status of the command itself, which masks
the status of assignments. To get the status, either perform the declaration
and assignment separately, or use a temporary variable to store the status. For
set -e the former is the only option.

-- 
Dan Douglas

Re: Tilde expansion during command search

[Dan Douglas]

On Friday, August 01, 2014 06:20:28 AM Mike Frysinger wrote:
 On Wed 23 Jul 2014 08:51:19 Dan Douglas wrote:
  On Wednesday, July 23, 2014 09:28:02 AM you wrote:
   On 7/23/14, 8:22 AM, Dan Douglas wrote:
Hi, from this discussion:

https://github.com/koalaman/shellcheck/issues/195#issuecomment-49678200

I can't find any reference that says substituting a literal tilde in
PATH
should occur during command search.
   
   Bash has always done this, even back to the pre-version 1 days, and I
   don't
   see any reason to change it now.
  
  The only concerns I can think of are inconsistency with programs that use
  execvp(), or possibly double-expansion in the event of a user name or any
  path containing ~.
 
 how so ?  execvp doesn't do tilde expansion.  only the shell does.

That's the problem. A program using execvp should be able to expect the same 
result as that of the shell for a given PATH value. Because of this, it can't, 
unless exec has the same implementation.

 and it does it once at assignment.

The example from my first post demonstrates that Bash substitutes a literal 
tilde in PATH during the actual path search before executing a command, in 
addition to during assignments and word expansions.

-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.

Re: Tilde expansion during command search

[Dan Douglas]

On Thursday, July 24, 2014 02:34:13 PM lolilolicon wrote:
 On Wed, Jul 23, 2014 at 11:00 PM, lolilolicon loliloli...@gmail.com wrote:
  On Wed, Jul 23, 2014 at 10:43 PM, Eric Blake ebl...@redhat.com wrote:
  But we are talking about the case where ~ IS quoted during
  the assignment, and only bash then re-expands it during path lookup.
 
  That's my point. Let me try quoting this again,
 
  Note that the tildes are expanded during the assignment to PATH, _not
  when PATH is accessed during command search._
 
  in which I was trying to imply that, Bash *does* expand the tildes _when
  PATH is accessed during command search._
 
 I'm sorry if it felt like I was taking that sentence out of context.
 Well, I was; but I think it's fair to assume that whoever wrote that
 sentence would agree that tilde expansion should not be performed when
 PATH is accessed during command search.

The section you quoted isn't present at all in issue 7. The current version of 
the spec makes tilde expansion within assignments mandatory, including 
multiple tilde expansions when there are colons. The part that describes how 
that works moved to XRAT section 2.6.1. However the part that refers to what 
happens during command search appears to have been removed.

I agree that if the quoted sentence were present it would probably imply that 
this behavior is explicitly disallowed.
-- 
Dan Douglas

Tilde expansion during command search

[Dan Douglas]

Hi, from this discussion:

https://github.com/koalaman/shellcheck/issues/195#issuecomment-49678200

I can't find any reference that says substituting a literal tilde in PATH 
should occur during command search.

$ ( cd ~
mkdir tmpdir
echo $'#!/bin/sh\necho Hello from ${BASH_SOURCE}!' ~/tmpdir/testscript
chmod u+x ~/tmpdir/testscript
PATH=\~/tmpdir
typeset -p PATH
testscript )

declare -x PATH=~/tmpdir
Hello from /home/devtest/tmpdir/testscript!

$ echo $BASH_VERSION
4.3.18(1)-release

-- 
Dan Douglas

Re: Tilde expansion during command search

[Dan Douglas]

On Wednesday, July 23, 2014 09:28:02 AM you wrote:
 On 7/23/14, 8:22 AM, Dan Douglas wrote:
  Hi, from this discussion:
  
  https://github.com/koalaman/shellcheck/issues/195#issuecomment-49678200
  
  I can't find any reference that says substituting a literal tilde in PATH 
  should occur during command search.
 
 Bash has always done this, even back to the pre-version 1 days, and I don't
 see any reason to change it now.
 

The only concerns I can think of are inconsistency with programs that use 
execvp(), or possibly double-expansion in the event of a user name or any path 
containing ~.

You're probably right in that it's not super critical if it hasn't caused 
problems so far.

-- 
Dan Douglas

Re: Tilde expansion during command search

[Dan Douglas]

On Wednesday, July 23, 2014 07:58:26 AM Eric Blake wrote:
 On 07/23/2014 07:51 AM, Dan Douglas wrote:
  On Wednesday, July 23, 2014 09:28:02 AM you wrote:
  On 7/23/14, 8:22 AM, Dan Douglas wrote:
  Hi, from this discussion:
 
  https://github.com/koalaman/shellcheck/issues/195#issuecomment-49678200
 
  I can't find any reference that says substituting a literal tilde in PATH 
  should occur during command search.
 
  Bash has always done this, even back to the pre-version 1 days, and I don't
  see any reason to change it now.
 
  
  The only concerns I can think of are inconsistency with programs that use 
  execvp(), or possibly double-expansion in the event of a user name or any 
  path 
  containing ~.
  
  You're probably right in that it's not super critical if it hasn't caused 
  problems so far.
 
 Might be worth asking the POSIX folks if it is allowed by POSIX.  What
 do other shells do?
 

I tested it in the link from my first post. AFAICT Bash is the only one.

#!/bin/bash
printf '%s\n' '#!/bin/sh' 'echo test' ~/tmpdir/testscript
chmod u+x ~/tmpdir/testscript
s=(bash ksh mksh posh zsh dash bb jsh)

# Testing for parameter expansion.
for sh in ${s[@]}; do
printf '%-4s %s\n' ${sh}: $($sh -c 'PATH=\${HOME}/tmpdir; testscript' 
21)
done
echo

# Testing for tilde expansion.
for sh in ${s[@]}; do
printf '%-4s %s\n' ${sh}: $($sh -c 'PATH=\~/tmpdir; testscript' 21)
done

bash: bash: testscript: command not found
ksh: ksh: testscript: not found
mksh: mksh: testscript: not found
posh: posh: testscript: not found
zsh: zsh:1: command not found: testscript
dash: dash: 1: testscript: not found
bb:  bb: testscript: not found
jsh: jsh: testscript: not found

bash: test
ksh: ksh: testscript: not found
mksh: mksh: testscript: not found
posh: posh: testscript: not found
zsh: zsh:1: command not found: testscript
dash: dash: 1: testscript: not found
bb:  bb: testscript: not found
jsh: jsh: testscript: not found

-- 
Dan Douglas

signature.asc
Description: This is a digitally signed message part.