Re: array subscripts act differently for integers(ie. let)
On 2/20/15 3:52 PM, Dan Douglas wrote: IMO bash is exactly correct. I don't know how this could be fixed in a way that would satisfy people without changing something very fundamental. If you disable the initial expansion to `(())', then `(($x))' wouldn't work (because arithmetic evaluation doesn't itself evaluate parameter expansions). However if you disabled evaluating parameter expansions during variable resolution (for array indexes) then you would be stuck with exactly the above ksh problem. Yeah, exactly. I could potentially avoid the double-expansion issue by adding flags and other conditional beahvior, but I'm afraid it would be fragile and inconsistent. The backwards compatibility issue would come up either way. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: array subscripts act differently for integers(ie. let)
On Thu, Feb 19, 2015 at 7:47 PM, Chet Ramey chet.ra...@case.edu wrote:
On 2/18/15 4:14 PM, emanuelczi...@cryptolab.net wrote:
tl;dr: thanks! no reply needed;
Thanks guys. I had a hard time accepting that this is how it's supposed to
work.
I accepted it now. :)
Yeah. The expression between (( and )) is word expanded, since (( ... ))
is supposed to be exactly equivalent to let
The real issue is that assignment statements in arithmetic expressions
that contain array references are also word expanded, almost as if they
were executed in an assignment statement context. This is how bash has
always behaved, though; backwards compatibility is a concern.
The way bash works right now allows safely handling arbitrary
associative array keys. This is a pretty good illustration:
$ bash -c 'key=\]; typeset -A a=([$key]=x [y]=y); unset -v
a[\${key}]; typeset -p a'
declare -A a='([y]=y )'
$ ksh -c 'key=\]; typeset -A a=([$key]=x [y]=y); unset -v
a[\${key}]; typeset -p a'
typeset -A a=([']']=x [y]=y)
Since ksh doesn't do parameter expansion while resolving variables
quite consistently it's impossible to handle this case. (I'll have to
ask about it on their list one of these days).
IMO bash is exactly correct. I don't know how this could be fixed in
a way that would satisfy people without changing something very
fundamental. If you disable the initial expansion to `(())', then
`(($x))' wouldn't work (because arithmetic evaluation doesn't itself
evaluate parameter expansions). However if you disabled evaluating
parameter expansions during variable resolution (for array indexes)
then you would be stuck with exactly the above ksh problem.
It's unfortunate people don't understand this, but when you think
about it it can't really work any other way.
--
Dan Douglas
Re: array subscripts act differently for integers(ie. let)
On 2/18/15 7:57 PM, Eduardo A. Bustamante López wrote:
On Wed, Feb 18, 2015 at 10:14:10PM +0100, emanuelczi...@cryptolab.net wrote:
That segfault though:
I confirm that the segmentation fault is in the latest devel version.
Here's the patch I applied, very similar to yours.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
*** ../bash-20150206/variables.c 2015-01-23 20:39:27.0 -0500
--- variables.c 2015-02-19 13:56:12.0 -0500
***
*** 2873,2880
v = bind_variable (lhs, rhs, 0);
! if (v isint)
! VSETATTR (v, att_integer);
!
! VUNSETATTR (v, att_invisible);
return (v);
--- 2873,2882
v = bind_variable (lhs, rhs, 0);
! if (v)
! {
! if (isint)
! VSETATTR (v, att_integer);
! VUNSETATTR (v, att_invisible);
! }
return (v);
Re: array subscripts act differently for integers(ie. let)
On 2/18/15 4:14 PM, emanuelczi...@cryptolab.net wrote: tl;dr: thanks! no reply needed; Thanks guys. I had a hard time accepting that this is how it's supposed to work. I accepted it now. :) Yeah. The expression between (( and )) is word expanded, since (( ... )) is supposed to be exactly equivalent to let The real issue is that assignment statements in arithmetic expressions that contain array references are also word expanded, almost as if they were executed in an assignment statement context. This is how bash has always behaved, though; backwards compatibility is a concern. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: array subscripts act differently for integers(ie. let)
also this variant does the same:
$ (('ar[$idbad2]+=11'))
Because (( and let are essentially equivalent.
I think the more important question isn't why does (( behave this way?,
but rather should (( behave this way?.
It's probably not reasonable to expect the author to know and take into
account that (( arr[$key] )) treats key's data as bash code. Really, the
behaviour of ((' arr[$key] ')) makes more sense as a default behaviour.
The former is broken (as the expectation that $key expands data is not met).
Re: array subscripts act differently for integers(ie. let)
On 2/18/15 3:49 PM, Maarten Billemont wrote:
also this variant does the same:
$ (('ar[$idbad2]+=11'))
Because (( and let are essentially equivalent.
I think the more important question isn't why does (( behave this way?,
but rather should (( behave this way?.
It's probably not reasonable to expect the author to know and take into
account that (( arr[$key] )) treats key's data as bash code.
What does `bash code' mean? It undergoes the usual set of word expansions.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: array subscripts act differently for integers(ie. let)
tl;dr: thanks! no reply needed;
Thanks guys. I had a hard time accepting that this is how it's supposed
to work.
I accepted it now. :)
I just hope nobody is going to try to exploit it or something.
That segfault though:
set -x
declare -A ar
key='`echo -n 1times.txt`'
((++ar[$key])) # cmd in $key gets executed twice too
declare -p ar
$ ./b.bash
./b.bash:5+ declare -A ar
./b.bash:6+ key='`echo -n 1times.txt`'
./b.bash:7+ (( ++ar[`echo -n 1times.txt`] ))
../b.bash:7+ echo -n 1
./b.bash: line 7: ar: bad array subscript
../b.bash:7+ echo -n 1
./b.bash: line 7: ar[`echo -n 1times.txt`]: bad array subscript
Segmentation fault (core dumped)
Disclaimer: I have stumbled upon this non-issue by mistake and I've no
intention of exploiting anything ('cause I got principles, heh;
seriously though)
Ok bye.
PS: this still makes me sad though :')
On 2015-02-18 20:14, Chet Ramey wrote:
On 2/16/15 12:23 PM, emanuelczi...@cryptolab.net wrote:
Oh I see, I had no idea that's how it's meant to work. My apologies.
However this case still doesn't work, but maybe I should use single
quotes
all the time?:
this fails(double quotes):
$ declare -A ar
$ idbad2=[
$ let ar[$idbad2]+=11
bash: let: ar[[]+=11: bad array subscript (error token is ar[[]+=11)
$ declare -p ar
bash: declare: ar: not found
Bash expects the brackets to match. This is true when the parser tries
to
find the closing bracket and when the arithmetic expression evaluator
does -- they use the same criteria. You can make a case that they
should
not have to match, but bash has always behaved this way and you'll have
to
work with it, at least for now.
this works(single quotes):
$ let 'ar[$idbad2]+=11'
$ declare -p ar
declare -A ar='([[]=11 )'
This works because the arithmetic expression evaluator doesn't have to
deal with the extra `[' when trying to find the end of the subscript,
and
it's smart enough to figure out whether it's dealing with an indexed or
associative array and evaluate accordingly.
also this variant does the same:
Because (( and let are essentially equivalent.
Chet
Re: array subscripts act differently for integers(ie. let)
On Wed, Feb 18, 2015 at 10:14:10PM +0100, emanuelczi...@cryptolab.net wrote: That segfault though: I confirm that the segmentation fault is in the latest devel version. dualbus@dualbus ~ % gdb GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-linux-gnu. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. (gdb) file /tmp/bash-devel Reading symbols from /tmp/bash-devel...done. (gdb) r poc Starting program: /tmp/bash-devel poc + declare -A ar + key='`echo -n 1times.txt`' + (( ++ar[`echo -n 1times.txt`] )) ++ echo -n 1 poc: line 4: ar: bad array subscript ++ echo -n 1 poc: line 4: ar[`echo -n 1times.txt`]: bad array subscript Program received signal SIGSEGV, Segmentation fault. bind_int_variable (lhs=0x7b8f08 ar[`echo -n \1\times.txt`], rhs=0x7b32f8 1) at variables.c:2878 warning: Source file is more recent than executable. 2878 (gdb) bt #0 bind_int_variable (lhs=0x7b8f08 ar[`echo -n \1\times.txt`], rhs=0x7b32f8 1) at variables.c:2878 #1 0x0044e901 in expr_bind_variable (lhs=0x7b8f08 ar[`echo -n \1\times.txt`], rhs=0x7b32f8 1) at expr.c:317 #2 0x0044f984 in exp0 () at expr.c:987 #3 0x0044f89f in exp1 () at expr.c:955 #4 0x0044f790 in exppower () at expr.c:910 #5 0x0044f58a in exp2 () at expr.c:835 #6 0x0044f51f in exp3 () at expr.c:809 #7 0x0044f4b0 in expshift () at expr.c:785 #8 0x0044f403 in exp4 () at expr.c:755 #9 0x0044f38c in exp5 () at expr.c:733 #10 0x0044f34a in expband () at expr.c:715 #11 0x0044f30c in expbxor () at expr.c:696 #12 0x0044f2ce in expbor () at expr.c:677 #13 0x0044f23f in expland () at expr.c:650 #14 0x0044f1ac in explor () at expr.c:622 #15 0x0044f075 in expcond () at expr.c:578 #16 0x0044ed3c in expassign () at expr.c:466 #17 0x0044ed07 in expcomma () at expr.c:446 #18 0x0044ec84 in subexpr (expr=0x7b9d88 ++ar[`echo -n \1\times.txt`]) at expr.c:428 #19 0x0044eb1c in evalexp (expr=0x7b9d88 ++ar[`echo -n \1\times.txt`], validp=0x7fffdd6c) at expr.c:393 #20 0x004403c4 in execute_arith_command (arith_command=0x7b9a48) at execute_cmd.c:3561 #21 0x0043bc18 in execute_command_internal (command=0x7b8d88, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x7b8e68) at execute_cmd.c:995 #22 0x0043add0 in execute_command (command=0x7b8d88) at execute_cmd.c:416 #23 0x004263dd in reader_loop () at eval.c:163 #24 0x0042412b in main (argc=2, argv=0x7fffdfe8, env=0x7fffe000) at shell.c:757 (gdb) info locals v = 0x0 isint = 0 isarr = 1 implicitarray = 0 (gdb) l 2873 else 2874INVALIDATE_EXPORTSTR (entry); 2875 2876 if (var_isset (entry)) 2877dispose_command (function_cell (entry)); 2878 2879 if (value) 2880var_setfunc (entry, copy_command (value)); 2881 else 2882var_setfunc (entry, 0); This patch seems to fix it: dualbus@dualbus ~/local/src/bash % git log -p -1|cat commit 5d29a37a60c9daabf85de66dd7df3c459bd0c468 Author: Eduardo A. Bustamante López dual...@gmail.com Date: Wed Feb 18 18:53:04 2015 -0600 Check if v is not NUL diff --git a/variables.c b/variables.c index 2f07ebb..91912cd 100644 --- a/variables.c +++ b/variables.c @@ -2875,6 +2875,7 @@ bind_int_variable (lhs, rhs) if (v isint) VSETATTR (v, att_integer); + if (v) VUNSETATTR (v, att_invisible); return (v);
Re: array subscripts act differently for integers(ie. let)
it occurs when an associative array with an empty subscript is modified.
$ ( typeset -A a; x='a[$()]++'; ((x)); )
-bash: a: bad array subscript
-bash: a[$()]: bad array subscript
Segmentation fault (core dumped)
I could live without the error on an empty key to begin with. It can
be hard to protect against errors. Although bash doesn't throw
exceptions on referencing any undefined index like many languages, you
still have to guard against an empty key whenever its value isn't
known.
${key+${a[$key]}}, or else ${a[${key}_]} followed by
${a[${key%_}]} on each usage. Or at minimum [[ $key ]] on every
iteration of a loop over ${!a[@]}. It's almost uglier than `set -u'
workarounds...
--
Dan Douglas
Re: array subscripts act differently for integers(ie. let)
On 2/18/15 3:49 PM, Maarten Billemont wrote: [...] I think the more important question isn't why does (( behave this way?, but rather should (( behave this way?. It's probably not reasonable to expect the author to know and take into account that (( arr[$key] )) treats key's data as bash code. I guess this is part of the discussion here: * http://lists.gnu.org/archive/html/bug-bash/2014-12/msg00071.html The whole thread is very interesting.
Re: array subscripts act differently for integers(ie. let)
On Mon, Feb 16, 2015 at 03:04:49AM +0100, emanuelczi...@cryptolab.net wrote: Description: when dealing with integer operations(let, ++), bad array subscript errors can happen because subscripts get unquoted (and evaluated even though they are in single quotes); unlike what happens when dealing with strings. Not a bug. That's just how bash works.
Re: array subscripts act differently for integers(ie. let)
On Mon, Feb 16, 2015 at 03:04:49AM +0100, emanuelczi...@cryptolab.net wrote: please see attached bash script to reproduce because that explains it much better than I could in words. The argument you give to let should be quoted. imadev:~$ declare -A ar imadev:~$ idbad1=bad string imadev:~$ let ar[$idbad1]+=11 imadev:~$ declare -p ar declare -A ar='([bad string]=11 )'
Re: array subscripts act differently for integers(ie. let)
Oh I see, I had no idea that's how it's meant to work. My apologies.
However this case still doesn't work, but maybe I should use single
quotes all the time?:
this fails(double quotes):
$ declare -A ar
$ idbad2=[
$ let ar[$idbad2]+=11
bash: let: ar[[]+=11: bad array subscript (error token is ar[[]+=11)
$ declare -p ar
bash: declare: ar: not found
this works(single quotes):
$ let 'ar[$idbad2]+=11'
$ declare -p ar
declare -A ar='([[]=11 )'
also this variant does the same:
fails:
$ ((ar[$idbad2]+=11))
bash: ((: ar[[]+=11: bad array subscript (error token is ar[[]+=11)
works:
$ (('ar[$idbad2]+=11'))
$ declare -p ar
declare -A ar='([[]=22 )'
Thanks.
On 2015-02-16 17:56, Greg Wooledge wrote:
On Mon, Feb 16, 2015 at 03:04:49AM +0100, emanuelczi...@cryptolab.net
wrote:
please see attached bash script to reproduce because that explains
it much better than I could in words.
The argument you give to let should be quoted.
imadev:~$ declare -A ar
imadev:~$ idbad1=bad string
imadev:~$ let ar[$idbad1]+=11
imadev:~$ declare -p ar
declare -A ar='([bad string]=11 )'
array subscripts act differently for integers(ie. let)
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
-DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib
-D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe
-fstack-protector-strong --param=ssp-buffer-size=4
-DDEFAULT_PATH_VALUE='/usr/local/sbin:/usr/local/bin:/usr/bin'
-DSTANDARD_UTILS_PATH='/usr/bin' -DSYS_BASHRC='/etc/bash.bashrc'
-DSYS_BASH_LOGOUT='/etc/bash.bash_logout'
uname output: Linux manji 3.19.0-gbfa76d49 #28 SMP Mon Feb 9 14:29:14
CET 2015 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu
Bash Version: 4.3
Patch Level: 33
Release Status: release
Description:
when dealing with integer operations(let, ++), bad array subscript
errors can happen because subscripts get unquoted (and evaluated even
though they are in single quotes); unlike what happens when dealing with
strings.
Repeat-By:
please see attached bash script to reproduce because that explains it
much better than I could in words.
Thank you.
#!/bin/bash
#tested under: GNU bash, version 4.3.33(1)-release (x86_64-pc-linux-gnu) of Gentoo Linux, package: app-shells/bash-4.3_p33-r1
# also tested on the same version on Linux Manjaro: GNU bash, version 4.3.33(1)-release (x86_64-unknown-linux-gnu) /usr/bin/bash is owned by bash 4.3.033-1, /bin - usr/bin/
set -u
set -x
set +e
declare -A ar=()
idgood=goodstring
idbad1=bad string
idbad2=[
let ar[$idgood]+=1 #works
echo ${ar[$idgood]}
let ar['$idbad1']+=12 #works but it's a workaround
echo ${ar[$idbad1]}
let ar['$idbad2']+=33 #works but it's a workaround
echo ${ar[$idbad2]}
let ar[$idbad1]+=44 #fail: + let 'ar[bad' 'string]+=44'
#./subscripty_integer.bash: line 21: let: ar[bad: bad array subscript (error token is ar[bad)
let ar[$idbad2]+=55 #fail: + let 'ar[[]+=55'
#./subscripty_integer.bash: line 24: let: ar[[]+=55: bad array subscript (error token is ar[[]+=55)
ar[$idbad1]+=12 #works but it's considered as string which makes sense! but just showing that this works for string types, but not for `let` above
echo ${ar[$idbad1]}
ar[$idbad2]+=33 #works but it's string as expected; but set -x doesn't show value of $idbad2 when displaying: + ar[$idbad2]+=33
echo ${ar[$idbad2]}
echo ${ar[[]}
echo before0: ${ar[$idbad1]}
((ar['$idbad1']++)) #works but not as you'd expect(ie. it evaluates $idbad1); set -x shows: + (( ar[$idbad1]++ ))
echo after0: ${ar[$idbad1]}
((ar['$idbad2']++)) #works, workaround
#echo before1: ${ar[$idbad1]}
((ar[$idbad1]++)) #works as expected! set -x shows: + (( ar[bad string]++ ))
#echo after1: ${ar[$idbad1]}
((ar[$idbad2]++)) #fails: ((: ar[[]++: bad array subscript (error token is ar[[]++)
#echo before2: ${ar[$idbad1]}
((ar[$idbad1]++)) #works as expected!
#echo after2: ${ar[$idbad1]}
((ar[$idbad2]++)) #fails: ((: ar[[]++: bad array subscript (error token is ar[[]++)
