Re: Changing the way bash expands associative array subscripts

[Koichi Murase]

2021年3月30日(火) 0:03 Chet Ramey :
> On 3/15/21 9:41 PM, Koichi Murase wrote:
> > Can you also take care of the behavior of `unset -v 'a[@]''?
>
> Well, you have to pick one behavior or another. The `@' subscript always
> expands to all members of the array. The current behavior is consistent
> with that, and documented.

When `shopt -s assoc_expand_once' is enabled, I think it is better
that `unset -v 'a[@]'' will remove just an element associated with the
key `@' but not remove the entire associative array `a' although it
might be inconsistent to the other places such as `${a[@]}'. More
specifically, I'm thinking of the use case that a user wants to remove
an entry for an arbitrary $key using

  unset -v "assoc[$key]"

If `unset -v 'a[@]'' would continue to remove the entire array even
after the change of this time, when key=@, the above command will
cause unexpected behavior for the user: The entire associative array
will be removed. This can be another serious pitfall of Bash
associative arrays which users would easily miss. If this pitfall
would remain, I think the current behavior is better.

But, maybe we can introduce a special syntactic treatment of `unset'
as done for `declare/typeset/local/export/readonly'. Just as ${a[@]}
and ${a[$key]} are treated differently, maybe we can treat

  unset -v a[@]

and

  unset -v a[$key]

differently. Also, the syntax can recognize `name[key]' as a special
pattern not to be confused with a glob pattern, which is another
typical pitfall of `unset' array elements. Now I think this type of
fix is the cleanest way. The treatment of `test -v a[@]' and `test -v
a[$key]' may also be changed the same as `unset'. Although `test' and
`[' are defined in POSIX, the option `-v' is a Bash extension so the
syntactic treatment might be changed.

> > Maybe this is unrelated to the extra expansions of array subscripts,
> > but is there any reason that Bash disallows empty associative keys?
>
> That's a separate change I haven't considered as part of this.

Yes. If there isn't a strong reason for disallowing the empty
associative-array keys, may I create a new thread of bug-bash for the
feature request of supporting empty keys?

--
Koichi

Re: select syntax violates the POLA

[Greywolf]

On 4/5/2021 14:21, Chet Ramey wrote:

This presented an opportunity to marginally simplify the grammar, so it's a
win all around.

The change will be in the next devel branch push.


You have my sincere gratitude.  I did not expect so much to come of it.


Chet


--*greywolf;

Re: select syntax violates the POLA

[Greywolf]

On 4/5/2021 13:52, Chet Ramey wrote:

On 4/1/21 5:40 PM, Greywolf wrote:


Or do you mean my coding style (which has been valid for over 25 years)?



Hey, if you want to go there, `select' has been in  bash with its current
syntax for longer than that. ;-)

(A month over 27 years, if you're curious.)



HAH!  And I only just found about it very recently while perusing the
man page!  Go figure!

In any case, I apologise if, at any point, I came across as hostile
(never my intent), and I thank you for taking the time to parley.
I've learned a few things.

--*greywolf;

Re: Changing the way bash expands associative array subscripts

[konsolebox]

On Mon, Apr 5, 2021 at 9:44 PM Chet Ramey  wrote:
>
> On 4/1/21 3:02 PM, konsolebox wrote:
>
> >> To do what, exactly?
> >
> > To keep this working in both behaviors.
> >
> > declare -A a
> > key='$(echo foo)'
> > a[$key]=1
> > unset 'a["$key"]'
> > declare -p a # declare -A a=()
>
> I suppose. You'll be able to set compatibility levels to make this work
> without resorting to wrappers.

Is it not possible to just have an added option to unset to make it
use old behavior without needing to use compat*?  Compatibility mode
affects general behavior and this isn't always something I'll need to
keep my scripts compatible.  In the manual I read "It's intended to be
a temporary solution" implying the script at one point should upgrade
its minimum version requirement and that's not what I'd want.


-- 
konsolebox

Re: select syntax violates the POLA

[Chet Ramey]

On 4/1/21 5:54 PM, Greywolf wrote:


On 4/1/2021 8:57, Chet Ramey wrote:


It's more like `select' doesn't inherit some of the shell's special cases.


    select dir in ${d[@]}; do {
    break;
    } done;

...but select breaks on the "} done;" syntax 


Yes, you need a list terminator so that `done' is recognized as a reserved
word here. `;' is sufficient. Select doesn't allow the `done' unless it's
in a command position. Some of the other compound commands have special
cases, mostly inherited from the Bourne shell, to allow it.


I took a look at bash-1.14, where the `select' command first appeared. At
the time, it was identical to the `for' command -- which makes sense, since
the syntaxes are essentially identical (and internally, they are
represented the same way). They both used the `list' grammar production.

When I updated the grammar to use the `compound_list' production in
bash-2.0, to be closer to how POSIX specified the compound commands in the
POSIX grammar, I left the select command alone. It's been the same way
since late 1994, though bash-2.0 wasn't released until 1996.

I took this opportunity to examine the grammar again, and there's no good
reason `select' and `for' should be different -- they started out the same,
and they should both use compound_list now. So they will.

This presented an opportunity to marginally simplify the grammar, so it's a
win all around.

The change will be in the next devel branch push.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: select syntax violates the POLA

[Chet Ramey]

On 4/1/21 9:23 PM, Dale R. Worley wrote:

Greg Wooledge  writes:

It's amazing how many people manage to post their code with NO comments
or explanations of what it's supposed to do, what assumptions are being
made about the inputs, etc.  This leaves us to guess.


It seems to be a modern style.  When I was learning to program, poorly
commented code was considered a failing.  But recently, I have had
managers object that I put too many comments in.


If it was hard to write, it should be hard to understand!

(That's snark, for those who are wondering.)

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: select syntax violates the POLA

[Chet Ramey]

On 4/1/21 5:40 PM, Greywolf wrote:


Or do you mean my coding style (which has been valid for over 25 years)?



Hey, if you want to go there, `select' has been in  bash with its current
syntax for longer than that. ;-)

(A month over 27 years, if you're curious.)

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Empty array referenced by indirection reports unbound variable

[konsolebox]

set -u
array=()
__ref=array[@]
: "${array[@]}" # Reports nothing
: "${!__ref}" # Unbound variable

Using bash version 5.1.4. I know this can be avoided by using namerefs
instead but indirection is more portable when no assignment is needed
and this sometimes allows my portable functions to have less versions.

The detection of unbound variables seem to have become more reasonable
since 5.0 and I hope it gets improved further.  I don't use set -u in
older bash and it's only for development mode.

-- 
konsolebox

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[konsolebox]

On Tue, Apr 6, 2021 at 2:46 AM Chet Ramey  wrote:
> There's no guarantee that file descriptors opened with socket(2) show up in
> /dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so it's up
> to the OS on those systems whether they do or not.
>
> It doesn't generate that error on macOS, for instance, which has /dev/fd.

Thanks. I'll take note of it for future reference.


-- 
konsolebox

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[konsolebox]

On Tue, Apr 6, 2021 at 3:38 AM Oğuz  wrote:
>> Yeah. If you have /dev/fd though you have /dev/fd/1 (and even /dev/stdout), 
>> so OP could have done `echo a | tee /dev/fd/1 >/dev/tcp/host/port' instead
>
>
> That should have been `echo a | tee /dev/fd/3 3>&1 >/dev/tcp/host/port'

Nice strategies.


-- 
konsolebox

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[Oğuz]

5 Nisan 2021 Pazartesi tarihinde Oğuz  yazdı:

> 5 Nisan 2021 Pazartesi tarihinde Chet Ramey  yazdı:
>
>> On 4/5/21 3:14 PM, Oğuz wrote:
>>
>>> 5 Nisan 2021 Pazartesi tarihinde Chet Ramey >> > yazdı:
>>>
>>> There's no guarantee that file descriptors opened with socket(2)
>>> show up in
>>> /dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so
>>> it's up
>>> to the OS on those systems whether they do or not.
>>>
>>> It doesn't generate that error on macOS, for instance, which has
>>> /dev/fd.
>>>
>>>
>>> I can reproduce the error, but it is coming from `tee', which tries to
>>> open `/dev/fd/??' as a regular file whereas it is a socket, and gets an
>>> ENXIO error.
>>>
>>
>> That's what I mean. What happens on a system that implements /dev/fd and N
>> in /dev/fd/N is a valid descriptor? There are varying answers.
>>
>> The simplest /dev/fd drivers just dup the file descriptor passed as an
>> argument. There are progressively more complex implementations. It's
>> not guaranteed they treat descriptors returned by socket(2) the same as
>> those returned by open(2), even to whether they show up in /dev/fd at all.
>>
>> If you don't have /dev/fd, all bets are off, of course, but then I'd guess
>> the error would be ENOENT.
>
>
> Yeah. If you have /dev/fd though you have /dev/fd/1 (and even
> /dev/stdout), so OP could have done `echo a | tee /dev/fd/1
> >/dev/tcp/host/port' instead
>

That should have been `echo a | tee /dev/fd/3 3>&1 >/dev/tcp/host/port'


>
>
>>
>> --
>> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>>  ``Ars longa, vita brevis'' - Hippocrates
>> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>>
>
>
> --
> Oğuz
>
>

-- 
Oğuz

Re: select syntax violates the POLA

[Greywolf]

On 4/5/2021 6:27, Chet Ramey wrote:

On 4/5/21 5:06 AM, Greywolf wrote:


but if you want to go down that track, it's kind of interesting to note
that {} on a single line demands a ; before the }, while () prohibits
it. 


This isn't true:

$ ./bash -c '(echo a b c;)'
a b c

but it is the case that } is a reserved word, while ) is an operator.



D'OH!  You're right!

I was thinking

$ foo &;

Can't put a semicolon after a null statement.

My mistake.  Carry on...

--*greywolf;

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[Oğuz]

5 Nisan 2021 Pazartesi tarihinde Chet Ramey  yazdı:

> On 4/5/21 3:14 PM, Oğuz wrote:
>
>> 5 Nisan 2021 Pazartesi tarihinde Chet Ramey > chet.ra...@case.edu>> yazdı:
>>
>> There's no guarantee that file descriptors opened with socket(2) show
>> up in
>> /dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so
>> it's up
>> to the OS on those systems whether they do or not.
>>
>> It doesn't generate that error on macOS, for instance, which has
>> /dev/fd.
>>
>>
>> I can reproduce the error, but it is coming from `tee', which tries to
>> open `/dev/fd/??' as a regular file whereas it is a socket, and gets an
>> ENXIO error.
>>
>
> That's what I mean. What happens on a system that implements /dev/fd and N
> in /dev/fd/N is a valid descriptor? There are varying answers.
>
> The simplest /dev/fd drivers just dup the file descriptor passed as an
> argument. There are progressively more complex implementations. It's
> not guaranteed they treat descriptors returned by socket(2) the same as
> those returned by open(2), even to whether they show up in /dev/fd at all.
>
> If you don't have /dev/fd, all bets are off, of course, but then I'd guess
> the error would be ENOENT.


Yeah. If you have /dev/fd though you have /dev/fd/1 (and even /dev/stdout),
so OP could have done `echo a | tee /dev/fd/1 >/dev/tcp/host/port' instead


>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>


-- 
Oğuz

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[Chet Ramey]

On 4/5/21 3:14 PM, Oğuz wrote:
5 Nisan 2021 Pazartesi tarihinde Chet Ramey > yazdı:


There's no guarantee that file descriptors opened with socket(2) show up in
/dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so it's up
to the OS on those systems whether they do or not.

It doesn't generate that error on macOS, for instance, which has /dev/fd.


I can reproduce the error, but it is coming from `tee', which tries to open 
`/dev/fd/??' as a regular file whereas it is a socket, and gets an ENXIO error.


That's what I mean. What happens on a system that implements /dev/fd and N
in /dev/fd/N is a valid descriptor? There are varying answers.

The simplest /dev/fd drivers just dup the file descriptor passed as an
argument. There are progressively more complex implementations. It's
not guaranteed they treat descriptors returned by socket(2) the same as
those returned by open(2), even to whether they show up in /dev/fd at all.

If you don't have /dev/fd, all bets are off, of course, but then I'd guess
the error would be ENOENT.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[Oğuz]

5 Nisan 2021 Pazartesi tarihinde Chet Ramey  yazdı:
>
> There's no guarantee that file descriptors opened with socket(2) show up in
> /dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so it's up
> to the OS on those systems whether they do or not.
>
> It doesn't generate that error on macOS, for instance, which has /dev/fd.
>

I can reproduce the error, but it is coming from `tee', which tries to open
`/dev/fd/??' as a regular file whereas it is a socket, and gets an ENXIO
error.


-- 
Oğuz

Re: No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[Chet Ramey]

On 4/5/21 1:31 PM, konsolebox wrote:

Not sure if this is a bug or a limitation but this works:

# echo a | ( exec {fd}>/path/to/regular/file; tee /dev/fd/"$fd" )

While this produces a "No such device or address" error:

# echo a | ( exec {fd}>/dev/tcp/host/port; tee /dev/fd/"$fd" )


There's no guarantee that file descriptors opened with socket(2) show up in
/dev/fd. Bash doesn't emulate /dev/fd if the system provides it, so it's up
to the OS on those systems whether they do or not.

It doesn't generate that error on macOS, for instance, which has /dev/fd.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: SIGCHLD traps shouldn't recurse

[L A Walsh]

On 2021/04/03 00:41, Oğuz wrote:

 but I
don't think it's useful at all because the number of pending traps keeps
piling up, and there is no way to reset that number. If there is no real
use case for recursive SIGCHLD traps (which I can't think of any), I think
this should change; no SIGCHLD trap should be queued while a SIGCHLD trap
is already in progress.
  
  
   So what happens if a child dies while you are servicing the previous

child?

From the perl ipc page:

sub REAPER {
  my $child;
  while (($waitedpid = waitpid(-1, WNOHANG)) > 0) {
  logmsg "reaped $waitedpid" . ($? ? " with exit $?" : "");
  }
  $SIG{CHLD} = \&REAPER; }
---
   The while loop grabs finished task stats before returning.


With this change:

diff --git a/trap.c b/trap.c
index dd1c9a56..5ce6ab4f 100644
--- a/trap.c
+++ b/trap.c
@@ -643,6 +643,8 @@ void
 queue_sigchld_trap (nchild)
  int nchild;
 {
+  if (sigmodes[SIGCHLD] & SIG_INPROGRESS)
+return;
   if (nchild > 0)
 {
   catch_flag = 1;

bash behaves this way:

$ trap uname chld
$ uname -sr
Linux 5.4.0-66-generic
Linux
$ uname -sr
Linux 5.4.0-66-generic
Linux
$

and I think this is what average user would expect. Whether there's a
better fix is beyond me though.
  


   Looks like your uname is called twice, whereas some langs (perl)
tried to auto-cleanup such things.

No such device or address when opening /dev/fd/ on opened /dev/tcp socket

[konsolebox]

Not sure if this is a bug or a limitation but this works:

# echo a | ( exec {fd}>/path/to/regular/file; tee /dev/fd/"$fd" )

While this produces a "No such device or address" error:

# echo a | ( exec {fd}>/dev/tcp/host/port; tee /dev/fd/"$fd" )

Simpler forms like this does the same:

# echo a | tee /dev/fd/4 4>/dev/tcp/host/port
# echo a | ( tee /dev/fd/4 ) 4>/dev/tcp/host/port

The workaround is to use process substitution and cat:

# echo a | tee >( cat > /dev/tcp/host/port )

Tested in 5.1.4.

-- 
konsolebox

Re: Changing the way bash expands associative array subscripts

[Chet Ramey]

On 3/30/21 7:05 PM, konsolebox wrote:


As I've observed, in single expansion mode, unset will fail to unset a
value when the key has a closing bracket in it. Perhaps unset should
check the last character first when looking for the closing bracket.
Tested in 5.1.4.


There is some support for this as part of assoc_expand_once, but it's
incomplete in bash-5.1.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: zsh style associative array assignment bug

[Oğuz]

5 Nisan 2021 Pazartesi tarihinde Jesse Hathaway 
yazdı:

> This equivalence, though it is reasonable, creates a special case for the
> last
> value in an array which imposes a greater cognitive burden on the
> programmer.
> The programmer needs to be aware that omitting the last value will create
> an
> empty value rather than emitting an error.
>

Averting the possible effects of programmer's incompetence shouldn't really
be the programming language's responsibility.


-- 
Oğuz

Re: zsh style associative array assignment bug

[Jesse Hathaway]

On Mon, Mar 29, 2021 at 4:18 PM Chet Ramey  wrote:
> If you look at
>
> a=( k1 v1 k2 v2 k3 v3)
>
> as more or less syntactic sugar for
>
> a=( [k1]=v1 [k2]=v2 [k3]=v3 )
>
> it's reasonable that
>
> a=( k1 v1 k2 )
>
> is equivalent to
>
> a=( [k1]=v1 [k2]= ). And that's what bash does.

This equivalence, though it is reasonable, creates a special case for the last
value in an array which imposes a greater cognitive burden on the programmer.
The programmer needs to be aware that omitting the last value will create an
empty value rather than emitting an error. Given that you need quotes to create
an empty value for any other element but the last:

  $ a=( k1 "" k2 v2 k3)
  $ declare -p a
  declare -A a=([k1]="" [k2]="v2" [k3]="" )

I would prefer the consistency and readability of always requiring an even
number of arguments:

  $ a=( k1 "" k2 v2 k3 "")
  $ declare -p a
  declare -A a=([k1]="" [k2]="v2" [k3]="" )

In general I find that fewer surprises in a language lead to easier to grok
code.

Re: select syntax violates the POLA

[Chet Ramey]

On 4/5/21 5:06 AM, Greywolf wrote:


the same level of
egregious brokenness as select.


Good god, take a breath.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Changing the way bash expands associative array subscripts

[Chet Ramey]

On 4/1/21 3:02 PM, konsolebox wrote:


To do what, exactly?


To keep this working in both behaviors.

declare -A a
key='$(echo foo)'
a[$key]=1
unset 'a["$key"]'
declare -p a # declare -A a=()


I suppose. You'll be able to set compatibility levels to make this work
without resorting to wrappers.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: select syntax violates the POLA

[Chet Ramey]

On 4/5/21 5:06 AM, Greywolf wrote:


but if you want to go down that track, it's kind of interesting to note
that {} on a single line demands a ; before the }, while () prohibits
it. 


This isn't true:

$ ./bash -c '(echo a b c;)'
a b c

but it is the case that } is a reserved word, while ) is an operator.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: select syntax violates the POLA

[Greywolf]

On 4/2/2021 8:21, konsolebox wrote:
That's not a rule but a special compromise.  [[ ]] and (( )) are a 
form of reserved words themselves just like () and {} as they can be 
used multi-line but they aren't allowed to be adjacent to else et al 
without a semicolon.  [[ ]], (( )), {}, and () are practically just 
commands with first-class parsing that consistently have to end with

a semicolon if followed by another reserved word or command.



I'd call it less a compromise than a mechanism; of all listed,
{} represents a block of code in the current shell, () represents
a block of code in a subshell...

but if you want to go down that track, it's kind of interesting to note
that {} on a single line demands a ; before the }, while () prohibits
it. This also seems inconsistent to me, but not at the same level of
egregious brokenness as select.

--*greywolf;

Re: select syntax violates the POLA

[Greywolf]

On 4/1/2021 8:57, Chet Ramey wrote:

Yes, you need a list terminator so that `done' is recognized as a reserved
word here. `;' is sufficient. Select doesn't allow the `done' unless it's
in a command position. Some of the other compound commands have special
cases, mostly inherited from the Bourne shell, to allow it.


I don't know that I'd call them "special cases" if sh has pretty much 
condoned it from the get-go; 'select' is the outlier, from my POV.


--*greywolf;

Re: select syntax violates the POLA

[Greywolf]

On 4/4/2021 23:13, konsolebox wrote:

On Sat, Apr 3, 2021 at 1:09 AM Robert Elz  wrote:

   |  [[ ]] and (( )) are a form of reserved words themselves

Those are bash specials, and I am fairly sure that (( and )) will be
operators, not reserved words (they cannot really be the latter, as ( and
) are operators) and I suspect that [[ and ]] might be as well, but there
I'm not sure.   operators and reserved words are quite different things.
Operators (unquoted) are recognised as themselves wherever they appear.


Stop suspecting and read the source code.  Look at parse.y starting at
line 2150.  They are labeled as "Reserved words".




[[ and ]] behave like reserved words requiring whitespace between
themselves and their operands.  (( and )) do not carry that restriction;
within (( )), neither do any of */%^&|+- (or their respective OP=),
nor == || && !.  This effectively classifies (( )) as operators.

But we've drifted considerably afield.

--*greywolf;