Re: Arithmetic pow results incorrect in arithmetic expansion.

[Robert Elz]

Date:Sat, 9 Jan 2021 14:45:49 +0900
From:Hyunho Cho 
Message-ID:  


  | I didn't know that all other shells work the same.

Not "all" other shells, most don't implement exponentiation at all,
since it isn't a standard C operator.

Just the ones that happen to implement it.

kre

Re: Arithmetic pow results incorrect in arithmetic expansion.

[Hyunho Cho]

I didn't know that all other shells work the same.
Thanks for the clarification.




2021년 1월 9일 (토) 오후 2:29, Lawrence Velázquez 님이 작성:

> > On Jan 8, 2021, at 11:19 PM, Hyunho Cho  wrote:
> >
> > Machine: x86_64
> > OS: linux-gnu
> > Compiler: gcc
> > Compilation CFLAGS: -g -O2
> > -fdebug-prefix-map=/build/bash-a6qmCk/bash-5.0=.
> > -fstack-protector-strong -Wformat -Werror=format-security -Wall
> > -Wno-parentheses -Wno-format-security
> > uname output: Linux EliteBook 5.4.0-42-generic #46-Ubuntu SMP Fri Jul
> > 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> > Machine Type: x86_64-pc-linux-gnu
> >
> > Bash Version: 5.0
> > Patch Level: 17
> > Release Status: release
> >
> > 
> >
> > i have tested below in gnome calculator, Qalculate, gawk, perl
> > and all results in -4 but bash is 4
> >
> > $ awk 'BEGIN { print -2 ^ 2 }'
> > -4
> >
> > $ perl -E 'say -2 ** 2'
> > -4
> >
> > $ echo $(( -2 ** 2 )) # only bash results in 4
> > 4
>
> The bash results disagree with awk and perl (and, admittedly, common
> mathematical convention), but the man page clearly states that unary
> minus has higher precedence than exponentiation, so this behavior
> is intentional.
>
> The operators and their precedence, associativity, and values are
> the same as in the C language. The following list of operators is
> grouped into levels of equal-precedence operators. The levels are
> listed in order of decreasing precedence.
>
> id++ id--   variable post-increment and post-decrement
> - + unary minus and plus
> ++id --id   variable pre-increment and pre-decrement
> ! ~ logical and bitwise negation
> **  exponentiation
> [remaining operators follow]
>
> Additionally, this precedence is consistent with other shells.
>
> % bash --version | head -n 1
> GNU bash, version 5.0.17(1)-release (x86_64-apple-darwin18.7.0)
> % bash -c 'printf %s\\n "$((-2 ** 2))"'
> 4
>
> % ksh --version
>   version sh (AT Research) 93u+ 2012-08-01
> % ksh -c 'printf %s\\n "$((-2 ** 2))"'
> 4
>
> % zsh --version
> zsh 5.8 (x86_64-apple-darwin18.7.0)
> % zsh -fc 'printf %s\\n "$((-2 ** 2))"'
> 4
> % zsh -f -o C_PRECEDENCES -c 'printf %s\\n "$((-2 ** 2))"'
> 4
>
> The zshmisc(1) man page even goes out of its way to address this.
>
> Note the precedence of exponentiation in both cases is below that of
> unary operators, hence `-3**2' evaluates as `9', not `-9'. Use
> parentheses where necessary: `-(3**2)'. This is for compatibility
> with other shells.
>
> I don't know why things shook out this way, but it sure seems like
> the ship has sailed.
>
> vq
>

Re: Arithmetic pow results incorrect in arithmetic expansion.

[Lawrence Velázquez]

> On Jan 8, 2021, at 11:19 PM, Hyunho Cho  wrote:
> 
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> Compilation CFLAGS: -g -O2
> -fdebug-prefix-map=/build/bash-a6qmCk/bash-5.0=.
> -fstack-protector-strong -Wformat -Werror=format-security -Wall
> -Wno-parentheses -Wno-format-security
> uname output: Linux EliteBook 5.4.0-42-generic #46-Ubuntu SMP Fri Jul
> 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> Machine Type: x86_64-pc-linux-gnu
> 
> Bash Version: 5.0
> Patch Level: 17
> Release Status: release
> 
> 
> 
> i have tested below in gnome calculator, Qalculate, gawk, perl
> and all results in -4 but bash is 4
> 
> $ awk 'BEGIN { print -2 ^ 2 }'
> -4
> 
> $ perl -E 'say -2 ** 2'
> -4
> 
> $ echo $(( -2 ** 2 )) # only bash results in 4
> 4

The bash results disagree with awk and perl (and, admittedly, common
mathematical convention), but the man page clearly states that unary
minus has higher precedence than exponentiation, so this behavior
is intentional.

The operators and their precedence, associativity, and values are
the same as in the C language. The following list of operators is
grouped into levels of equal-precedence operators. The levels are
listed in order of decreasing precedence.

id++ id--   variable post-increment and post-decrement
- + unary minus and plus
++id --id   variable pre-increment and pre-decrement
! ~ logical and bitwise negation
**  exponentiation
[remaining operators follow]

Additionally, this precedence is consistent with other shells.

% bash --version | head -n 1
GNU bash, version 5.0.17(1)-release (x86_64-apple-darwin18.7.0)
% bash -c 'printf %s\\n "$((-2 ** 2))"' 
4

% ksh --version
  version sh (AT Research) 93u+ 2012-08-01
% ksh -c 'printf %s\\n "$((-2 ** 2))"'
4

% zsh --version
zsh 5.8 (x86_64-apple-darwin18.7.0)
% zsh -fc 'printf %s\\n "$((-2 ** 2))"'
4
% zsh -f -o C_PRECEDENCES -c 'printf %s\\n "$((-2 ** 2))"'
4

The zshmisc(1) man page even goes out of its way to address this.

Note the precedence of exponentiation in both cases is below that of
unary operators, hence `-3**2' evaluates as `9', not `-9'. Use
parentheses where necessary: `-(3**2)'. This is for compatibility
with other shells.

I don't know why things shook out this way, but it sure seems like
the ship has sailed.

vq

Re: Arithmetic pow results incorrect in arithmetic expansion.

[Oğuz]

9 Ocak 2021 Cumartesi tarihinde Hyunho Cho  yazdı:
>
> i have tested below in gnome calculator, Qalculate, gawk, perl
> and all results in -4 but bash is 4
>
> $ awk 'BEGIN { print -2 ^ 2 }'
> -4
>
> $ perl -E 'say -2 ** 2'
> -4
>
> $ echo $(( -2 ** 2 )) # only bash results in 4
> 4
>

`bc' does that too. Here's another trivial side effect of implementing
unary minus using binary minus:

$ echo $((16#-10))
-10
$ echo $((-16#10))
-16


-- 
Oğuz

Arithmetic pow results incorrect in arithmetic expansion.

[Hyunho Cho]

Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -g -O2
-fdebug-prefix-map=/build/bash-a6qmCk/bash-5.0=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall
-Wno-parentheses -Wno-format-security
uname output: Linux EliteBook 5.4.0-42-generic #46-Ubuntu SMP Fri Jul
10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.0
Patch Level: 17
Release Status: release



i have tested below in gnome calculator, Qalculate, gawk, perl
and all results in -4 but bash is 4

$ awk 'BEGIN { print -2 ^ 2 }'
-4

$ perl -E 'say -2 ** 2'
-4

$ echo $(( -2 ** 2 )) # only bash results in 4
4

Re: declare accept + before =

[Léa Gris]

Le 08/01/2021 à 15:26, Chet Ramey écrivait :

On 1/8/21 8:45 AM, Léa Gris wrote:

Just curious why it accepts a + before =


Try it with a variable that already has a value.


OMG I feel stupid now!


--
Léa Gris

Re: Associative array keys are not reusable in (( command

[Ilkka Virta]

On Fri, Jan 8, 2021 at 5:44 PM Chet Ramey  wrote:

> On 1/8/21 10:24 AM, Oğuz wrote:
> > This situation is why bash-5.0 introduced the `assoc_expand_once'
> option.
> > But it allows arbitrary command injection.
>
> If you want to run array keys through word expansions, this is one
> potential result. Command substitution is "arbitrary command injection."
>

Let's say I don't want to run the keys through expansions. How
does assoc_expand_once help then, if it doesn't stop the expansion inside
$key?

With the keys in the previous messages, what _seems_ to work is

  shopt -u assoc_expand_once
  (( assoc[\$k]++ ))

But it only works with assoc_expand_once disabled, and somehow, I'm not
sure if it's safe for more complex keys.

then there's also

x=${assoc[$k]}
assoc[$k]=$(( x + 1 ))

which doesn't seem to be affected by assoc_expand_once but is a bit weary
to use.

Re: Associative array keys are not reusable in (( command

[Chet Ramey]

On 1/8/21 10:24 AM, Oğuz wrote:


8 Ocak 2021 Cuma tarihinde Chet Ramey > yazdı:


On 1/8/21 5:20 AM, Oğuz wrote:

See:

      $ declare -A assoc=($'\n\'' 42)
      $ for key in "${!assoc[@]}"; do (( assoc[$key]++ )); done
      bash: ((: assoc[
      ']++ : bad array subscript (error token is "assoc[
      ']++ ")
      $
      $ (( assoc[${key@Q}]++ ))
      bash: ((: assoc[$'\n\'']++ : bad array subscript (error token is
"assoc[$'\n\'']++ ")


This situation is why bash-5.0 introduced the `assoc_expand_once' option.


But it allows arbitrary command injection.


If you want to run array keys through word expansions, this is one
potential result. Command substitution is "arbitrary command injection."


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Associative array keys are not reusable in (( command

[Oğuz]

8 Ocak 2021 Cuma tarihinde Chet Ramey  yazdı:

> On 1/8/21 5:20 AM, Oğuz wrote:
>
>> See:
>>
>>  $ declare -A assoc=($'\n\'' 42)
>>  $ for key in "${!assoc[@]}"; do (( assoc[$key]++ )); done
>>  bash: ((: assoc[
>>  ']++ : bad array subscript (error token is "assoc[
>>  ']++ ")
>>  $
>>  $ (( assoc[${key@Q}]++ ))
>>  bash: ((: assoc[$'\n\'']++ : bad array subscript (error token is
>> "assoc[$'\n\'']++ ")
>>
>
> This situation is why bash-5.0 introduced the `assoc_expand_once' option.


But it allows arbitrary command injection.

$ declare -A assoc
$ key='x],b[$(uname >&2)'
$ shopt -s assoc_expand_once
$ (( assoc[$key]++ ))
Linux


> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>


-- 
Oğuz

Re: declare -p name=value thinks 'name=value' is variable

[Ilkka Virta]

On Fri, Jan 8, 2021 at 4:06 PM Chet Ramey  wrote:

> No. `declare -p' does not accept assignment statements.
>

The synopsis in the documentation doesn't make it clear, though. It shows
only one case with -p and assignment, while the similar case of export -p
is listed separately from the other forms of export. Well, at least in my
5.0 man page, not on the online manual, though. Also, the documentation
doesn't seem to say the assigned value is ignored with -p, even though it
does say additional options are ignored.

So, suggest changing the synopsis from

  declare [-aAfFgiIlnrtux] [-p] [name[=value] …]

to something like:

  declare [-aAfFgiIlnrtux] [name[=value] …]
  declare [-aAfFgiIlnrtux] -p [name…]

And/or add to:

  "The -p option will display the attributes [...] additional options,
other than -f and -F, are ignored."

something like

  "An assignment cannot be used with the -p option."

Also, the error message should probably be "invalid variable name" like you
get for referring to something like that via ${!var}, instead of "not
found".

Since I mentioned export, export -p foo doesn't seem to do anything, even
if foo is exported, not even give an error. But, contrary to declare,
export -p foo=bar
_does_ assign the value, silently. Confusing.

Re: declare accept + before =

[Chet Ramey]

On 1/8/21 8:45 AM, Léa Gris wrote:

Just curious why it accepts a + before =


Try it with a variable that already has a value.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Associative array keys are not reusable in (( command

[Chet Ramey]

On 1/8/21 5:20 AM, Oğuz wrote:

See:

 $ declare -A assoc=($'\n\'' 42)
 $ for key in "${!assoc[@]}"; do (( assoc[$key]++ )); done
 bash: ((: assoc[
 ']++ : bad array subscript (error token is "assoc[
 ']++ ")
 $
 $ (( assoc[${key@Q}]++ ))
 bash: ((: assoc[$'\n\'']++ : bad array subscript (error token is
"assoc[$'\n\'']++ ")


This situation is why bash-5.0 introduced the `assoc_expand_once' option.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: declare -p name=value thinks 'name=value' is variable

[Chet Ramey]

On 1/8/21 4:05 AM, William Park wrote:

Another issue I came across.

 declare -p name=value

thinks 'name=value' is the variable.  My reading of manpage seems to say
'name' should be used, but not sure.  Is this a bug?


No. `declare -p' does not accept assignment statements.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

declare accept + before =

[Léa Gris]

Just curious why it accepts a + before =

unset a; declare a+=hello; typeset -p a

bash, version 5.0.17(1)-release wrote:
> declare -- a="hello"


--
Léa Gris

Re: declare -p name=value thinks 'name=value' is variable

[Eli Schwartz]

On 1/8/21 4:05 AM, William Park wrote:

Another issue I came across.

 declare -p name=value

thinks 'name=value' is the variable.  My reading of manpage seems to say
'name' should be used, but not sure.  Is this a bug?

Workaround is, of course, use separate lines,
 declare name=value
 declare -p name


There's an obvious inference here, that declare -p only prints, does not 
create/modify, and only accepts NAMEs, therefore the optional [=VALUE] 
available to other modes will not be interpreted the way you seem to 
think it will be, in *this* mode.


--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User



OpenPGP_signature
Description: OpenPGP digital signature

Re: Associative array keys are not reusable in (( command

[Koichi Murase]

2021年1月8日(金) 20:07 Oğuz :
> On Fri, Jan 8, 2021 at 2:14 PM Koichi Murase  wrote:
>> There is no reason to introduce a different expansion rule of
>> `((...))' from that of `$((...))'.
>
> But there already is a different expansion rule. While `((
> assoc['$key']++ ))' works, `: $(( assoc['$key']++ ))' fails with a
> bad subscript error.

Actually, that inconsistency has been fixed in Bash 5.1 to match with
the behavior of `$(( assoc['$key']++ ))', i.e., now in Bash 5.1, `((
assoc['$key']++ ))' doesn't work either.  So, `(( assoc[\$key]++ ))'
is the only working way to access the associative array in the
arithmetic command of the form `(( ... ))'.  Or, maybe you could write
`let 'assoc[$key]++'`.

--
Koichi

Re: Associative array keys are not reusable in (( command

[Oğuz]

On Fri, Jan 8, 2021 at 2:14 PM Koichi Murase  wrote:

> This topic actually pops up occasionally.  FYI, you can also see the
> following discussions:
>
> https://lists.gnu.org/archive/html/bug-bash/2014-06/msg3.html
> https://lists.gnu.org/archive/html/bug-bash/2014-10/msg00154.html
> https://lists.gnu.org/archive/html/bug-bash/2015-02/msg00066.html
>
> 2021年1月8日(金) 18:21 Oğuz :
> > `(( assoc[\$key]++ ))' works as usual, but this is not documented as
> > far as I know,
>
> This is explained in the first link above by quoting two sentences in
> the document, but I agree with you that this is so non-trivial that
> it's worth explicitly documenting IMHO.
>

You are right, the first quote explains it.


> > Considering that the following two works, it doesn't make much sense
> > that `(( assoc[$key]++ ))' doesn't.
>
> I think the POSIX expansion order should be applied to, for example,
> $((assoc[$key]++)).  In this case, the arithmetic evaluation of the
> whole expression should be performed only after $key in
> $((assoc[$key]++)) is expanded.  There is no reason to introduce a
> different expansion rule of `((...))' from that of `$((...))'.  It
> feels a bit strange to me to expand `$key' in `(( assoc[$key]++ ))'
> after determining the arithmetic structure thinking the normal
> expansion order.
>

But there already is a different expansion rule. While `(( assoc['$key']++
))' works, `: $(( assoc['$key']++ ))' fails with a bad subscript error.


> > It would be better if at least quoting the key by means of parameter
> > transformation or `printf %q' worked, because these are the first
> > workarounds that come to mind;
>
> This is maybe a good idea.  From Bash 5.1, the single quotation is
> not a subject to the quote removal of `(( ... ))', so we can use the
> single quotation as a part of the arithmetic expression.  We can just
> introduce the string literal of the form '...' or $'...' in the
> arithmetic evaluator.
>

$'...' would be nice, it would even fix existing scripts that rely on @Q or
%q.


> --
> Koichi
>

Re: Associative array keys are not reusable in (( command

[Koichi Murase]

This topic actually pops up occasionally.  FYI, you can also see the
following discussions:

https://lists.gnu.org/archive/html/bug-bash/2014-06/msg3.html
https://lists.gnu.org/archive/html/bug-bash/2014-10/msg00154.html
https://lists.gnu.org/archive/html/bug-bash/2015-02/msg00066.html

2021年1月8日(金) 18:21 Oğuz :
> `(( assoc[\$key]++ ))' works as usual, but this is not documented as
> far as I know,

This is explained in the first link above by quoting two sentences in
the document, but I agree with you that this is so non-trivial that
it's worth explicitly documenting IMHO.

> Considering that the following two works, it doesn't make much sense
> that `(( assoc[$key]++ ))' doesn't.

I think the POSIX expansion order should be applied to, for example,
$((assoc[$key]++)).  In this case, the arithmetic evaluation of the
whole expression should be performed only after $key in
$((assoc[$key]++)) is expanded.  There is no reason to introduce a
different expansion rule of `((...))' from that of `$((...))'.  It
feels a bit strange to me to expand `$key' in `(( assoc[$key]++ ))'
after determining the arithmetic structure thinking the normal
expansion order.

> It would be better if at least quoting the key by means of parameter
> transformation or `printf %q' worked, because these are the first
> workarounds that come to mind;

This is maybe a good idea.  From Bash 5.1, the single quotation is
not a subject to the quote removal of `(( ... ))', so we can use the
single quotation as a part of the arithmetic expression.  We can just
introduce the string literal of the form '...' or $'...' in the
arithmetic evaluator.

--
Koichi

Associative array keys are not reusable in (( command

[Oğuz]

See:

$ declare -A assoc=($'\n\'' 42)
$ for key in "${!assoc[@]}"; do (( assoc[$key]++ )); done
bash: ((: assoc[
']++ : bad array subscript (error token is "assoc[
']++ ")
$
$ (( assoc[${key@Q}]++ ))
bash: ((: assoc[$'\n\'']++ : bad array subscript (error token is
"assoc[$'\n\'']++ ")

`(( assoc[\$key]++ ))' works as usual, but this is not documented as far as
I know, and not obvious to ordinary user at all. Considering that the
following two works, it doesn't make much sense that `(( assoc[$key]++ ))'
doesn't. It would be better if at least quoting the key by means of
parameter transformation or `printf %q' worked, because these are the first
workarounds that come to mind; it's not written anywhere that the
arithmetic evaluator can expand `$key' on its own.

$ (( assoc[$'\n\'']++ ))
$ declare -p assoc
declare -A assoc=([$'\n\'']="43" )
$
$ (( assoc['
'\']++ ))
$ declare -p assoc
declare -A assoc=([$'\n\'']="44" )

Oğuz

Re: local -p var prints nothing

[Lawrence Velázquez]

> On Jan 8, 2021, at 3:50 AM, William Park  wrote:
> 
> I don't know if it's a bug.  Manpage says "local" builtin takes all the
> options that "declare" takes.  But, "local -p var" doesn't print
> anything, where as "declare -p var" does.
> 
> f1()
> {
>local a=123
>local -p a
> }
> 
> f2()
> {
>local a=123
>declare -p a
> }
> 
> f1# prints nothing
> f2# prints $a

Please specify your version of bash when posting to these lists.
According to the NEWS file, bash 5.1 behaves as you expect:

y. `local' now honors the `-p' option to display all local
variables at the current context.

https://git.savannah.gnu.org/cgit/bash.git/tree/NEWS#n79

vq

declare -p name=value thinks 'name=value' is variable

[William Park]

Another issue I came across.

declare -p name=value

thinks 'name=value' is the variable.  My reading of manpage seems to say
'name' should be used, but not sure.  Is this a bug?  

Workaround is, of course, use separate lines,
declare name=value
declare -p name
-- 
William Park 

local -p var prints nothing

[William Park]

I don't know if it's a bug.  Manpage says "local" builtin takes all the
options that "declare" takes.  But, "local -p var" doesn't print
anything, where as "declare -p var" does.

f1()
{
local a=123
local -p a
}

f2()
{
local a=123
declare -p a
}

f1  # prints nothing
f2  # prints $a

-- 
William Park