Re: declare a=$b if $a previously set as array
On 12/16/14 4:49 PM, Stephane Chazelas wrote: I believe my proposal in Message-ID: 20141214204845.ga5...@chaz.gmail.com http://thread.gmane.org/gmane.comp.shells.bash.bugs/22737/focus=22789 would be the most consistent syntax. But that breaks compatibility especially as it means the output of declare -p on arrays would no longer work. Now, like Dan, I think we can have a middle ground that doesn't break backward compatibility as much but would remove most of the security concerns. Thanks for this. There are really only a couple of changes to the current behavior. 1.2 litteral scalar assignment x=a b y='($(uname))' declare $options a=$x b=~ c='(1 2)' d='($a)' e=$y those are *parsed* as scalar assignments in that ~ is expanded and $x doesn't undergo split+glob, but after that parsing and expansion is done, depending on whether -a/-A was passed or not, those a=a b b=/home/stephane c=(1 2) d=($a) e=($(uname)) arguments are interpreted differently. If -a is passed, that becomes: a=([0]=a b) b=([0]=/home/stephane) c=([0]=1 [1]=2) d=([0]=a [1]=b) e=([0]=Linux) # second round of evaluation but if you didn't # want that you could still do e=($y) This stays for backwards compatibility. I have some code in there (commented out) that changes it based on the shell's compatibility level setting. declare $options a[1]='(1 2)' however sets a[1] to '(1 2)' regardless of whether -a, -A or neither is passed. This is one of the changes. In previous versions, the [1] was treated as a subscript sizing the array and accepted for backwards compatibility (but always ignored). In the current development code, the assignment is performed on a[1] if the variable previously existed as an array, and behaves as in bash-4.3 otherwise. This is a concession to backwards compatibility. So, the main difference with the current behaviour would be that declare a='(1 2 3)' or: declare 'a=(1 2 3)' or declare a='([0]=a [1]=b)' would not be an array (or hash) assignment if a was previously declared as a hash or array. This is the other change. declare -a a='(1 2)' would still be but be deprecated in favour of: declare -a a=(1 2) or declare a=(1 2) I changed the output of `declare -p' to remove the single quotes around the rhs of compound assignment statements to make the recommended syntax a little more apparent. Thanks for all the discussion about this. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/17/14 3:58 AM, konsolebox wrote: On Mon, Dec 15, 2014 at 4:34 AM, Chet Ramey chet.ra...@case.edu wrote: It does implement `emulated behavior of normal assignments'. The question is whether or not it should do that after having had its arguments undergo one round of word expansion. After studying the code I realized that that's actually the only thing we have to consider for changing. It turns out that the solution here is to just not allow expanded arguments to have compound assignments. This may break a little compatibility but I believe it's the only proper solution for it. Yes, if we decide to go that way, this is the place to change it. There are a few more things to do if we want to do more to unify the way that arguments to declare and assignment statements are handled. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/16/14 7:30 AM, Stephane Chazelas wrote: * This solves the problem without breaking backwards compatibility and allows declare -p output to continue to function. [...] Well, if declare -p continues to function which is not what I understand your proposal to do, then: You don't have to assume that declare -p will continue to quote the rhs of the assignment statement when printing array values. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On Mon, Dec 15, 2014 at 1:48 PM, konsolebox konsole...@gmail.com wrote: Last thing is `declare -x $a_complete_assignment` should not be allowed; even quoted assignments like `declare -x $one=$two` simply because they are already a reinterpreted form i.e. they are already evaluated with a pair of single or double-quotes before declare tries to evaluate them and it is were declare's behavior becomes inconsistent and it is were everything starts going wrong. If we do allow it, how would we apply the proper rules to the expanded form of $any? - knowing that $any is already a terrible combination of raw constructs and values were literal values meant to be literal values become subject to misinterpretation as forms of assignment to an array. And what if `$a_complete_assignment` contain `x=$something`, would we expand $something or not? This just makes implementation of declare way more complicated, difficult and unclear. Sorry about this. I realized that we could actually allow this but only in scalar form and with no other form of expansion. So $something would remain as '$something'. In an expanded argument to declare, any character that follows the first equal sign in it would be assigned to the variable (or the first index of the array variable) as is. konsolebox
Re: declare a=$b if $a previously set as array
On Mon, Dec 15, 2014 at 4:34 AM, Chet Ramey chet.ra...@case.edu wrote:
It does implement `emulated behavior of normal assignments'. The question
is whether or not it should do that after having had its arguments undergo
one round of word expansion.
After studying the code I realized that that's actually the only thing
we have to consider for changing. It turns out that the solution here
is to just not allow expanded arguments to have compound assignments.
This may break a little compatibility but I believe it's the only
proper solution for it.
If we are concerned about compatibility, as suggested we can consider
adding another shopt option to allow other people to retain its old
behavior - which is to allow expanded arguments to be recognized as
possible forms of compound assignments.
And I thought that this concept is better proven with some modifications
to the code so I went ahead to create some. It turns out that we only
need to change one significant line to have it done which is this one:
if (value[0] == '(' value[vlen-1] == ')')
I changed it to this:
if (delayed_compounds value[0] == '(' value[vlen-1] == ')')
I named the shopt option delayed_compounds (it can be changed) which
if enabled would allow old behavior. I made it disabled by default but
this could be enabled with an option on compile time
(--enable-delayed-compounds-default).
This is my complete patch that's compatible with 4.3.30:
diff --git a/builtins/declare.def b/builtins/declare.def
index a634e7c..8fdf4ae 100644
--- a/builtins/declare.def
+++ b/builtins/declare.def
@@ -92,6 +92,14 @@ extern int posixly_correct;
static int declare_internal __P((register WORD_LIST *, int));
+#if defined (ARRAY_VARS)
+ #ifdef DELAYED_COMPOUNDS_DEFAULT
+int delayed_compounds = 1;
+ #else
+int delayed_compounds = 0;
+ #endif
+#endif
+
/* Declare or change variable attributes. */
int
declare_builtin (list)
@@ -540,7 +554,7 @@ declare_internal (list, local_var)
int vlen;
vlen = STRLEN (value);
- if (value[0] == '(' value[vlen-1] == ')')
+ if (delayed_compounds value[0] == '(' value[vlen-1] == ')')
compound_array_assign = 1;
else
simple_array_assign = 1;
diff --git a/builtins/shopt.def b/builtins/shopt.def
index 6050a14..abdbaa9 100644
--- a/builtins/shopt.def
+++ b/builtins/shopt.def
@@ -117,6 +117,10 @@ extern char *shell_name;
extern int debugging_mode;
#endif
+#if defined (ARRAY_VARS)
+ extern int delayed_compounds;
+#endif
+
static void shopt_error __P((char *));
static int set_shellopts_after_change __P((char *, int));
@@ -163,8 +167,15 @@ static struct {
{ compat42, shopt_compat41, set_compatibility_level },
#if defined (READLINE)
{ complete_fullquote, complete_fullquote, (shopt_set_func_t *)NULL},
+ #if defined (ARRAY_VARS)
+ { delayed_compounds, delayed_compounds, (shopt_set_func_t *)NULL },
+ #endif
{ direxpand, dircomplete_expand, shopt_set_complete_direxpand },
{ dirspell, dircomplete_spelling, (shopt_set_func_t *)NULL },
+#else
+ #if defined (ARRAY_VARS)
+ { delayed_compounds, delayed_compounds, (shopt_set_func_t *)NULL },
+ #endif
#endif
{ dotglob, glob_dot_filenames, (shopt_set_func_t *)NULL },
{ execfail, no_exit_on_failed_exec, (shopt_set_func_t *)NULL },
diff --git a/config.h.in b/config.h.in
index 08af2ba..720c7f0 100644
--- a/config.h.in
+++ b/config.h.in
@@ -167,6 +167,9 @@
/* Define for case-modifying word expansions */
#undef CASEMOD_EXPANSIONS
+/* Define to make the `delayed_compounds' shopt option enabled by default. */
+#undef DELAYED_COMPOUNDS_DEFAULT
+
/* Define to make the `direxpand' shopt option enabled by default. */
#undef DIRCOMPLETE_EXPAND_DEFAULT
diff --git a/configure b/configure
index 98bf890..175935f 100755
--- a/configure
+++ b/configure
@@ -803,6 +803,7 @@ enable_cond_command
enable_cond_regexp
enable_coprocesses
enable_debugger
+enable_delayed_compounds_default
enable_direxpand_default
enable_directory_stack
enable_disabled_builtins
@@ -1486,6 +1487,8 @@ Optional Features:
--enable-coprocessesenable coprocess support and the coproc reserved
word
--enable-debugger enable support for bash debugger
+ --enable-delayed-compounds-default
+ enable the delayed_compounds shell option by default
--enable-direxpand-default
enable the direxpand shell option by default
--enable-directory-stack
@@ -3085,6 +3088,11 @@ if test ${enable_debugger+set} = set; then :
enableval=$enable_debugger; opt_debugger=$enableval
fi
+# Check whether --enable-delayed-compounds-default was given.
+if test ${enable_delayed_compounds_default+set} = set; then :
+ enableval=$enable_delayed_compounds_default;
opt_delayed_compounds_default=$enableval
+fi
+
# Check whether --enable-direxpand-default was given.
if test ${enable_direxpand_default+set} = set; then :
Re: declare a=$b if $a previously set as array
2014-12-16 01:53:52 -0600, Dan Douglas: [...] That would be one way but I think this can be solved without going quite so far. How do you feel about these rules? 1. If a word that is an argument to declare is parsed as a valid assignment, then perform the assignment immediately as it appears lexically. If such a word is parsed as a simple assignment (with or without an index) then bash treats it as a simple scalar assignment to the variable or array element as any ordinary assignment would. If word is parsed as an assignment with array initializer list then bash treats it as such. Then, like in my proposal, declare a='(1 2 3)' and declare -a a='(1 2 3}' with valid scalar assignments would work like a='(1 2 3)' so like: a[0]='(1 2 3)' Or are you saying that declare -a/-A should be parsed differently from declare without -a/-A? 2. Any words that do not parse as valid assignments lexically during step 1 undergo the usual set of expansions for simple commands and the resulting words are saved. 3. After evaluation of all words from steps 1-2, those that have undergone expansion per 2 are passed as arguments to declare. Declare then parses each of these arguments, testing for assignments that became valid as a result of expansion. During this step, for assignments to variables that have an array attribute set, or for all assignments if declare has been given the -a or -A option, declare will treat assignments that have initializer lists as array assignments accordingly. Otherwise, declare will treat all assignments as scalar assignments to a variable or array element. Words that still fail to parse as valid assignments are an error. If I understand correctly, does that mean: declare 'a=($(uname 2))' (assuming a was previously declared as an array) should do the same as: eval 'a=($(uname 2))' ? I think this covers all the bases and is very close to the current behavior with one exception. All of the problems Stephane and I have brought up deal with the situation where a word that parses lexically as a scalar assignment isn't treated as such, so that is the only required change. I don't think that change will break many scripts. Some observations: * This solves the problem without breaking backwards compatibility and allows declare -p output to continue to function. [...] Well, if declare -p continues to function which is not what I understand your proposal to do, then: a='($(uname2))' bash -c 's=$(declare -p a); a[0]=foo; eval $s' would still run uname. And: a='($(uname2))' bash -c 'b[0]=; declare -l b=$a' as well. -- Stephane
Re: declare a=$b if $a previously set as array
On 12/16/14, 2:53 AM, Dan Douglas wrote: On Sunday, December 14, 2014 02:39:29 PM Chet Ramey wrote: And we get to the fundamental issue. Is it appropriate to require arguments to declaration commands to be valid assignment statements when the parser sees them, instead of when the builtin sees them, at least when using the compound assignment syntax. I'm ok with tightening the rules and saying that it is, but it's not backwards compatible and changes the behavior of things like declare -a foo='(1 2 3)' which I imagine plenty of people have used and has been a supported way to do compound assignment for many years. That would be one way but I think this can be solved without going quite so far. How do you feel about these rules? For context, these are the current semantics that bash attempts to implement. This change is slated for the next release of the standard (issue 8). http://austingroupbugs.net/view.php?id=351 This introduces the concept of a `declaration utility' and specifies their behavior with respect to assignment statements as arguments. In bash, declare, typeset, local, export, and readonly are all declaration commands. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/15/14, 9:41 PM, Linda Walsh wrote:
Though I just ran into a bit of weirdness (in 4.2.45)
(output is commented out and indented):
env -i /bin/bash --norc --noprofile
declare -a ar=(ONE TWO THREE)
declare -p ar
# declare -a ar='([0]=ONE [1]=TWO [2]=THREE)'
add 'l', and note output:
declare -al ar=(${ar[@]})
declare -p ar
# declare -al ar='([0]=ONE [1]=TWO [2]=THREE)' # Note - no
conversion
# ok, now set export:
declare -x ar=(${ar[@]})
declare -p ar
declare -axl ar='([0]=one [1]=two [2]=three)' # now -l
takes effect
Thanks for the report. This is a bug, and will be fixed in the next
release of bash.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
2014-12-15 22:00:54 -0500, Chet Ramey: On 12/14/14 4:44 PM, Stephane Chazelas wrote: There's still a (security) issue with declare -x/-l/-r that may still be used in non-function contexts (not export or readonly), and as result for things like: saved=$(export -p var) ... eval $saved And with declare -g in functions, so probably still worth addressing. What's your proposal? Let's see if we can get to something concrete. Something like what I proposed in one of my previous messages would probably work to make I believe my proposal in Message-ID: 20141214204845.ga5...@chaz.gmail.com http://thread.gmane.org/gmane.comp.shells.bash.bugs/22737/focus=22789 would be the most consistent syntax. But that breaks compatibility especially as it means the output of declare -p on arrays would no longer work. Now, like Dan, I think we can have a middle ground that doesn't break backward compatibility as much but would remove most of the security concerns. I'm not sure I understood Dan's proposal correctly and we may very well mean the same thing. If we make it that: 1- words that form valid assignments are parsed as such, like 1.1 litteral array assignment. x=1 y=2 z='[7]' declare a=([x+y]=c $z=foo) treats it like a=([x+y]=c $z=foo) That is, if a was previously not an array or hash, it promotes it to an array and is like: a=([3]=c [4]=a [5]='[7]=foo') And if a was previously declared as a hash, we get a syntax error. And: declare a=(x) a[1]=(1 2 3) gives a bash: a[1]: cannot assign list to array member error, 1.2 litteral scalar assignment x=a b y='($(uname))' declare $options a=$x b=~ c='(1 2)' d='($a)' e=$y those are *parsed* as scalar assignments in that ~ is expanded and $x doesn't undergo split+glob, but after that parsing and expansion is done, depending on whether -a/-A was passed or not, those a=a b b=/home/stephane c=(1 2) d=($a) e=($(uname)) arguments are interpreted differently. If -a is passed, that becomes: a=([0]=a b) b=([0]=/home/stephane) c=([0]=1 [1]=2) d=([0]=a [1]=b) e=([0]=Linux) # second round of evaluation but if you didn't # want that you could still do e=($y) (all variables promoted to arrays (or cannot convert associative to indexed array error if they were hashes)). If -A is passed, you get a syntax error because of the wrong c=(...) syntax. If none of those is passed, that's scalar assignment regardless of the current types of the variable. So setting $a or $a[0], so is like: a='a b' (or a[0]='a b' if a is an array or hash) b='/home/stephane' (or ...) c='(1 2)' (or c[0]='(1 2)'). d='($a)' e='($(uname))' declare $options a[1]='(1 2)' however sets a[1] to '(1 2)' regardless of whether -a, -A or neither is passed. 2. words that don't form valid assignments. Those are parsed as normal command line arguments. declare \a=(...) would get a syntax error because of the unexpected ( like with echo \a=(...). And the resulting words after expansion are treated like in 1.1 above. So, the main difference with the current behaviour would be that declare a='(1 2 3)' or: declare 'a=(1 2 3)' or declare a='([0]=a [1]=b)' would not be an array (or hash) assignment if a was previously declared as a hash or array. declare -a a='(1 2)' would still be but be deprecated in favour of: declare -a a=(1 2) or declare a=(1 2) declare -a a=$external_input would still be a command injection contrary to my earlier proposal, but in this case, it's unlikely one would do that (doesn't really make sense), or if they did (for instance because they did actually mean: eval declare -a a=$external_input then it's quite obvious that the input should be sanitised. declare a=([$external_input]=foo) declare a=([external_input]=foo) are also command injection vulnerabilities, but that's the same story with all arithmetic evaluation. -- Stephane
Re: declare a=$b if $a previously set as array
2014-12-16 21:49:58 +, Stephane Chazelas: [...] 1.1 litteral array assignment. [...] 1.2 litteral scalar assignment [...] 2. words that don't form valid assignments. Those are parsed as normal command line arguments. declare \a=(...) would get a syntax error because of the unexpected ( like with echo \a=(...). And the resulting words after expansion are treated like in 1.1 above. [...] Sorry, I meant like in 1.2, not like in 1.1. -- Stephane
Re: declare a=$b if $a previously set as array
So I'm still getting caught up on this thread, but hasn't this issue been done
to death in previous threads? Back when I was examining bash's strange declare
-a, you were the very first person I found to notice its quasi-keyword behavior
10 years ago
(https://lists.gnu.org/archive/html/bug-bash/2004-09/msg00110.html). Perhaps
you didn't realize what this was doing at the time?
To me the biggest problem is what happens when you explicitly request a scalar
assignment. (I even specified a[1] to ensure it's not an a vs. a[0] issue.)
bash -c 'typeset -a a; b=(foo bar); typeset a[1]=$b; typeset -p a;
printf %s ${a[@]}; echo'
declare -a a='([0]=foo [1]=bar)'
foo bar
This doesn't fit with my understanding of how it should work. Otherwise I think
the way declare's arguments are interpreted based upon their form and current
set of attributes is pretty well understood, albeit undocumented.
I agree that ksh sometimes makes more sense. I'd add that although ksh's
typeset will never evaluate its non-literal parts of assignment arguments as
assignment syntax after the initial expansion, in ksh it isn't possible to
modify the _type_ of a variable after declaration to begin with, because ksh
has a confusing distinction between types and attributes. Every time you
use a _different_ declaration command, you're wiping all attributes and
effectively unsetting then declaring a whole new variable, while attributes
remain mutable.
# In this case we're modifying attributes
$ ksh -c 'typeset -a x=(foo); print ${@x}; typeset -r x=; print
${@x}'
typeset -a
typeset -r -a
# In this case we're modifying an attribute, then declaring a new
variable
# (and then implicitly modifying its attribute, though ${@a} fails to
# report it correctly).
$ ksh -c 'typeset -a a=(foo); print ${@a}; typeset -T A=(typeset x);
A a=(x=foo); print ${@a}; a+=(x=bar); print ${@a}; typeset -p a'
typeset -a A
A
A -a a=((x=foo) (x=bar))
If it were redesigned today there would surely be no attributes, only types.
--
Dan Douglas
Re: declare a=$b if $a previously set as array
Sorry I did a bad job copy/pasting then editing that last example.
ksh -c 'typeset -a x=(foo); print ${@x}; typeset -T A=(typeset y); A
x=(y=foo); print ${@x}; x+=(y=bar); print ${@x}; typeset -p x'
typeset -a
A
A
A -a x=((y=foo) (y=bar))
--
Dan Douglas
Re: declare a=$b if $a previously set as array
2014-12-15 05:41:51 -0600, Dan Douglas: So I'm still getting caught up on this thread, but hasn't this issue been done to death in previous threads? Back when I was examining bash's strange declare -a, you were the very first person I found to notice its quasi-keyword behavior 10 years ago (https://lists.gnu.org/archive/html/bug-bash/2004-09/msg00110.html). Perhaps you didn't realize what this was doing at the time? [...] I can't say I remember that one, but note that was a different case that was later fixed. That was about: declare -a a=($b) where the content of $b was further evaluated. $ b='$(uname2)' bash-2.05b -c 'declare -a a=($b)' Linux That one was more clearly a bug as no one could really be expected to escape the content of $b there. I probably did not notice at the time that it was also the case with: declare -a a=(...) though it would never have occurred to me to use that syntax (although it's true that's the syntax that is used by declare -p) As previously mentionned, another related bug that was also reported over 10 years ago: http://lists.gnu.org/archive/html/bug-bash/2003-10/msg00065.html That one was about: a=($var) or a=(*) (without or without declare) where the resulting words may be of the form: [0]=foo (and then, I had not realised at the time that it was a code injection vulnerability as well): $ a='[0$(uname2)]=foo' bash-2.05b -c 'b=($a)' Linux -- Stephane
Re: declare a=$b if $a previously set as array
On Wednesday, December 10, 2014 10:43:45 AM Stephane Chazelas wrote:
David Korn mentions that local scoping a la ksh was originally rejected by
POSIX because ksh88 originally implemented dynamic scoping (like most
shells now do, but most other languages don't). ksh changed to lexical
scoping in ksh93 (probably partly to address that POSIX /requirement/) and
now everyone else wants dynamic scoping because that's how most shells have
implemented it in the interval.
A local builtin and dynamic scope is in the current beta of ksh.
I don't care very much for how it works though. It's only enabled as a part of
bash mode, which actively disables a bunch of ksh features and doesn't allow
you to pick and choose. Bash mode is all or nothing at least as of the last
snapshot, with no tunables like zsh's setopt.
$ ksh -c 'function f { typeset x=in f; g; typeset -p x; }; function
g { x=in g; }; x=global; f; typeset -p x'
x='in f'
x='in g'
$ ( exec -a bash ksh -c 'function f { typeset x=in f; g; typeset -p
x; }; function g { x=in g; }; x=global; f; typeset -p x' )
x='in g'
x=global
I would have much preferred if David Korn followed his original proposal of
adding dynamic scope for only POSIX-style function definitions across the
board. At least porting bash scripts will be slightly easier in the very
specific case that ksh is invoked correctly.
--
Dan Douglas
Re: declare a=$b if $a previously set as array
On 12/15/14 6:41 AM, Dan Douglas wrote:
To me the biggest problem is what happens when you explicitly request a scalar
assignment. (I even specified a[1] to ensure it's not an a vs. a[0] issue.)
bash -c 'typeset -a a; b=(foo bar); typeset a[1]=$b; typeset -p a;
printf %s ${a[@]}; echo'
declare -a a='([0]=foo [1]=bar)'
foo bar
This doesn't fit with my understanding of how it should work. Otherwise I
think
the way declare's arguments are interpreted based upon their form and current
set of attributes is pretty well understood, albeit undocumented.
If you use a subscript with typeset, even when you're attempting an
assignment, the following text from the manual page applies:
To explicitly declare an indexed array, use declare -a name (see SHELL
BUILTIN COMMANDS below). declare -a name[subscript] is also accepted;
the subscript is ignored.
Bash treats `typeset name[subscript]' as identical to `typeset -a name'
or `typeset -a name[subscript]'.
It's syntax picked up from old versions of ksh, which actually used the
subscript to size the array. Maybe it should attempt subscript assignment,
but it never has.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
Chet Ramey wrote:
On 12/8/14 4:56 AM, Stephane Chazelas wrote:
I'm saying that if a script writer writes:
declare a=$b
I don't think it's unreasonable for a script writer to expect that
this does not unset a; it's not documented to do that, and assignment
without the `declare' doesn't unset it. Assignment without declare
also pays attention to the variable's state: assigning to `a' as if it
were a scalar doesn't magically convert it from an array to a scalar;
it assigns element 0, as you know.
Similarly, if you use 'set', or 'shopt', they doesn't reset
all the state variables addressable w/those commands. declare can
be used to toggle those variable attributes w/o destroying the rest.
Though I just ran into a bit of weirdness (in 4.2.45)
(output is commented out and indented):
env -i /bin/bash --norc --noprofile
declare -a ar=(ONE TWO THREE)
declare -p ar
# declare -a ar='([0]=ONE [1]=TWO [2]=THREE)'
add 'l', and note output:
declare -al ar=(${ar[@]})
declare -p ar
# declare -al ar='([0]=ONE [1]=TWO [2]=THREE)' # Note - no
conversion
# ok, now set export:
declare -x ar=(${ar[@]})
declare -p ar
declare -axl ar='([0]=one [1]=two [2]=three)' # now -l
takes effect
---
Same thing happened w/integers... delayed conversion.
Weird. prolly fixed. (?)
- it will work in most of the cases (and that's one aspect why
it's dangerous, because it's hard to detect).
- but you've got a code injection vulnerability (in the very
special case where $b starts with (.
- for no good reason. See ksh for a better syntax that doesn't
have the issue.
It is code injection because the script writer is using unchecked
input.
That's why I thought it might be a way out to evolve bash to
know what vars come from the user or the environment -- then
shopt -o no_eval_tainted
would throw an error on *any* eval (whether it uses 'eval' or
not). (obviously not happening overnight, but it would be
more than a simple band-aid correcting this one use case
It's still a script writer's responsibility to check input
before blindly using it: this isn't considered a bug in
the C language -- but it is considered unsafe practice:
char * buff = getenv(PATH);
printf(buff);
Do you really think BASH should be changed because of
bad practice?
Re: declare a=$b if $a previously set as array
On 12/14/14 4:44 PM, Stephane Chazelas wrote: There's still a (security) issue with declare -x/-l/-r that may still be used in non-function contexts (not export or readonly), and as result for things like: saved=$(export -p var) ... eval $saved And with declare -g in functions, so probably still worth addressing. What's your proposal? Let's see if we can get to something concrete. Something like what I proposed in one of my previous messages would probably work to make declare -a var=value closer to declare -a var; var=value -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On Sunday, December 14, 2014 02:39:29 PM Chet Ramey wrote:
And we get to the fundamental issue. Is it appropriate to require
arguments to declaration commands to be valid assignment statements when
the parser sees them, instead of when the builtin sees them, at least
when using the compound assignment syntax.
I'm ok with tightening the rules and saying that it is, but it's not
backwards compatible and changes the behavior of things like
declare -a foo='(1 2 3)'
which I imagine plenty of people have used and has been a supported way
to do compound assignment for many years.
That would be one way but I think this can be solved without going quite
so far. How do you feel about these rules?
1. If a word that is an argument to declare is parsed as a valid
assignment, then perform the assignment immediately as it appears
lexically. If such a word is parsed as a simple assignment (with or
without an index) then bash treats it as a simple scalar assignment
to
the variable or array element as any ordinary assignment would. If
word
is parsed as an assignment with array initializer list then bash
treats
it as such.
2. Any words that do not parse as valid assignments lexically during
step
1 undergo the usual set of expansions for simple commands and the
resulting words are saved.
3. After evaluation of all words from steps 1-2, those that have
undergone
expansion per 2 are passed as arguments to declare. Declare then
parses
each of these arguments, testing for assignments that became valid as
a result of expansion. During this step, for assignments to variables
that have an array attribute set, or for all assignments if declare
has
been given the -a or -A option, declare will treat assignments that
have
initializer lists as array assignments accordingly. Otherwise,
declare
will treat all assignments as scalar assignments to a variable or
array
element. Words that still fail to parse as valid assignments are an
error.
I think this covers all the bases and is very close to the current behavior
with one exception. All of the problems Stephane and I have brought up deal
with the situation where a word that parses lexically as a scalar assignment
isn't treated as such, so that is the only required change. I don't think
that change will break many scripts.
Some observations:
* This solves the problem without breaking backwards compatibility and allows
declare -p output to continue to function.
* These rules will make the following become compatible with current ksh
versions:
$ ksh -c 'function f { typeset x[1]=(foo bar); typeset -p x; }; f'
typeset -a x=([1]='(foo bar)')
$ bash -c 'function f { typeset x[1]=(foo bar); typeset -p x; }; f'
declare -a x='([0]=foo [1]=bar)'
* Treating words that parse as scalar assignments as scalar assignments is
consistent with bash's usual treatment of a[idx]=(foo). If bash sees
declare foo[1]=(foo), it should print:
bash: foo[1]: cannot assign list to array member.
* Note the fine print in step 3. It deals with this case in bash's current
behavior and doesn't need to change (-a vs no -a):
$ bash -c 'x=[4]=foo [6]=bar; declare -a y=($x); declare -p y'
declare -a y='([4]=foo [6]=bar)'
$ bash -c 'x=[4]=foo [6]=bar; declare y=($x); declare -p y'
declare -- y=([4]=foo [6]=bar)
--
Dan Douglas
Re: declare a=$b if $a previously set as array
On 12/7/14 5:36 AM, Stephane Chazelas wrote:
In this case, it's not about sanitizing external input, at least
as far as the $external_input variable is concerned.
In a way, it is. It's understanding the context in which external
input is going to be used. However, I think that you're right, and
that the rules concerning assignment statements as arguments to
what the bash documentation refers to as `declaration commands' should
be closer to what they are for standalone assignment statements.
When a script writer writes:
declare -l a=$external_input
he's entitled to expect $a to contain the lower case version of
$external_input whatever $external_input contain.
If $external_input happens to contain ($(echo FOO)), I'd
expect $a to contain ($(echo foo)), not foo just because
$external_input happens to follow a specific pattern.
So the question is under which conditions assignment statements can be
considered as such when they are arguments to declare. The situation
is complicated by the fact that declare is a builtin, so one round of
word expansion is performed before declare sees the arguments. Bash
does do some analysis of the arguments to declaration builtins so it can
inhibit word splitting and filename generation, and it has internal
provisions to pass additional information to the builtin, so I'm sure I
can implement whatever conditions are appropriate.
Where the script writer may be to blame is for running declare
when that a variable was previous declared as an array (though
that could have been done by code he invoked from 3rd party
libraries), or not anticipating that his code would be invoked
in contexts where that a variable may have been declared as an
array.
It's true. I agree that it's better to decide what the appropriate
rules are and document them.
But even then, he can be excused not to imagine that
declare -l a=$external_input
could ever run an arbitrary commands.
Maybe. That's actually a straightforward consequence of a standard set
of rules: declare is a builtin, and so its arguments are run through the
usual word expansions before declare sees them (including quote removal);
declare accepts the usual name=word assignment syntax, including compound
assignment; and assigning to associative arrays involves running subscripts
through a defined set of expansions.
The question we're really considering is how much further to move
declaration commands away from the semantics of other builtins, at least
when using the compound assignment syntax.
And even if he intended to declare a as an array beforehand,
he could reasonably expect
declare -l a=$external_input
to do the same as
declare -l a; a=$external_input
That is, the same as a[0]=$external_input.
And we get to the fundamental issue. Is it appropriate to require
arguments to declaration commands to be valid assignment statements when
the parser sees them, instead of when the builtin sees them, at least
when using the compound assignment syntax.
I'm ok with tightening the rules and saying that it is, but it's not
backwards compatible and changes the behavior of things like
declare -a foo='(1 2 3)'
which I imagine plenty of people have used and has been a supported way
to do compound assignment for many years.
~$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
~$ b='($(uname))' bash --norc -c 'declare -a a; declare a=$b; printf
%s\n $a ${a[0]}'
Linux
(on Linux Mint amd64).
$ ./bash --version
GNU bash, version 4.3.30(15)-release (x86_64-apple-darwin13.1.0)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ b='($(uname))' ./bash --norc -c 'declare -a a; declare a=$b; printf
%s\n $a ${a[0]}'
Darwin
Darwin
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/7/14 11:16 PM, Eduardo A. Bustamante López wrote:
On Sun, Dec 07, 2014 at 07:34:53PM -0800, Linda Walsh wrote:
Only if you properly quote external input.
Well, that's the whole point, as a script writer, I don't expect to get
arbitrary code execution here:
| dualbus@hp:~/t$ unset var; value='[$(ls -l)]=1 [2]=2'; declare -a
var=($value); declare -p var
| bash: total 0: syntax error in expression (error token is 0)
Yeah, that's what we're discussing.
Or here:
| dualbus@hp:~/t$ a=(1 2 3); k='a[$(ls -l)]'; echo ${a[k]}
| bash: total 0: syntax error in expression (error token is 0)
And I *shouldn't* have to worry about that.
But the ship has sailed on this one. Every shell that implements indexed
arrays does what bash does here.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/8/14 4:56 AM, Stephane Chazelas wrote:
I'm saying that if a script writer writes:
declare a=$b
intending to declare the *scalar* varible $a as a copy of the
scalar variable $b (and remember that in ksh/bash, scalar
variables and arrays are not very differentiated, $a being
${a[0]}), and overlooked (or is not aware of (because that was
done by 3rd party code for instance)) the fact that the variable
was used as an array before (for instance because he used
a[0]=foo instead of a=foo for instance), then:
I don't think it's unreasonable for a script writer to expect that
this does not unset a; it's not documented to do that, and assignment
without the `declare' doesn't unset it. Assignment without declare
also pays attention to the variable's state: assigning to `a' as if it
were a scalar doesn't magically convert it from an array to a scalar;
it assigns element 0, as you know.
- it will work in most of the cases (and that's one aspect why
it's dangerous, because it's hard to detect).
- but you've got a code injection vulnerability (in the very
special case where $b starts with (.
- for no good reason. See ksh for a better syntax that doesn't
have the issue.
Well, I don't think it's particularly a syntax issue. It's the question
I wrote in my previous message: how much further should we move compound
assignment away from the execution semantics associated with builtins.
- and it's not consistent when the same assignment is done
without declare (and no, I don't agree declare is a mere
builtin as it's already parsed halfway between a builtin and an
assignment).
I understand what you're saying. It's true that bash treats arguments
to declaration commands differently from arguments to other builtins.
The question is twofold: the appropriate semantics and the amount of
backwards compatibility to sacrifice.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/8/14 3:48 PM, Linda Walsh wrote:
intending to declare the *scalar* varible $a as a copy of the
scalar variable $b
How does the script writer **KNOW** $a and $b are scalar?
I agree that the script writer needs to pay attention to the types and
values of variables he is using if he doesn't unset them first.
I'm using 'a=fmt' and 'b=out' for clarity in my example
but say I want to insert a line to print the current UID/EUID/time hostname:
#!/bin/bash
### add lowercase info line of currently executing user:
unset fmt out
declare -x fmt='($(echo uid:$UID) $(echo euid:$EUID) $(date) $(uname -n))'
declare -a out
declare -l out=$fmt
echo out=${out[@]}
sudo bash --norc -c 'declare -a out;declare -l out=$fmt; echo out=${out[@]}'
This is exactly the backwards compatibility issue I'm talking about. The
question is how much of this to give up and how to make the functionality
available, even if it's not by default. The simplest thing is to require
`eval declare -l out=$fmt', but change to the default behavior is going to
require some change to some scripts. Adding an option to change the
default behavior is least disruptive, but will probably not sufficiently
address the `vulnerability' to satisfy security people.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/8/14 4:31 PM, Eduardo A. Bustamante López wrote: If you want delayed evaluation, use 'eval', that's what it's for. Your use case is a lame excuse for a feature that's clearly more evil than useful. I stopped reading right here. In the end, this is a minor argument about a relatively obscure technology issue. Couching it in moral or ethical terms cheapens the argument and is a pretty good indicator that the writer lacks perspective. Take this as preachy if you like, but there are things out there that really deserve to be called evil, and this isn't one of them. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/8/14 9:51 PM, konsolebox wrote: On Sun, Dec 7, 2014 at 7:24 AM, Stephane Chazelas stephane.chaze...@gmail.com wrote: $ b='($(uname2))' bash -c 'a=(); declare -r a=$b' Linux I actually also don't like that form of implementation. Even if declare is a builtin and not a reserved word, it still has some special treatments that's different from a simple command. For example in a simple command like echo, something like 'echo a=(1 2 3 4)' would cause syntax error. Word splitting and pathname expansion also doesn't happen unless when assigning elements in an array. If declare is able to recognize assignments separated by spaces, I think it would have not been difficult avoiding $b to be evaluated since it's really not meant to be. It clearly is a simple form of assignment. Imo even if declare is a builtin (which is actually a little special over a normal one), it should have just emulated behavior of normal assignments and did nothing more than it besides declaring the type (and optionally have some other features not related to assignment). It does implement `emulated behavior of normal assignments'. The question is whether or not it should do that after having had its arguments undergo one round of word expansion. (But I think those who used to recommend using declare as a general tool for assigning values remotely over evil eval would love it since it makes assignments like `declare -g $var=$something` possible which naturally should have been illegal.) Why should this have been illegal? Even Posix acknowledges that constructs like `export $one=$two' are valid. The question is what to do about compound assignment. Anyhow, this problem would only happen if a variable in a specific scope (and only in that scope) has already been set previously as an array and it doesn't always happen if you write your scripts well. It may be worth recommending people do a unset var before doing a declare [-option] var I actually find it better to mention that on a simple assignment like declare var=$something, var's type doesn't change if it has already been declared previously as an array. As I mentioned in a previous message, there's no reason for a script writer to expect that `declare x=y' unsets x. Assignment without using `declare' doesn't. Maybe it is worth saying that nothing will change an array variable's type, though it's possible to promote a variable to an array. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/9/14 5:59 AM, Stephane Chazelas wrote: BTW, it's worth noting (and maybe documenting) that word splitting and filename generation are performed in: declare -g $var=$something (or declare ''var=$something or declare f\oo=$bar...) So it must be indeed: declare -g $var=$something Again, this is the difference between builtins and keywords. Arguments to builtin commands undergo the usual set of word expansions. Posix acknowledges and allows this. These aren't assignment statements as bash (and Posix) defines them, so they're not treated specially when used as arguments to declaration commands. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/9/14 7:55 AM, konsolebox wrote: Other shells certainly does not support local scopes. Pretty much every other shell except the straight descendants of the System V Bourne shell supports local scopes. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
2014-12-14 14:57:15 -0500, Chet Ramey:
[...]
Well, I don't think it's particularly a syntax issue. It's the question
I wrote in my previous message: how much further should we move compound
assignment away from the execution semantics associated with builtins.
[...]
IMO, the ksh behaviour is the closest one can get to a
consistent approach and still allow assignments to arrays:
act like an assignment modifier except when the arguments don't
form valid assignment (simple command parsing then, and only
scalar assignments for arguments that contain =).
declare var=(1 2 3)
declares and sets an array only as long as var=( and ) are
litteral (as in not the result of any expansion) and unquoted
and var is a valid identifier.
declare var=(1 2 3)
or
declare var=(1 2 3)
would be syntax errors.
declare var=$foo
doesn't perform word splitting but assigns the content of $foo
as a string to $var (or ${var[0]} if an array, or ${var[0]} if
a hash).
declare var=$foo
declare var=$foo
declare $foo=$foo
would perform split+glob on $foo. (parsed like a normal command)
declare var=([123]=bar)
parsed like a scalar assignment (no globbing on that unquoted
[123], no hash/array assignment.
In short, all the assignments you can do without declare should
work the same with declare. And things that would not be treated
as an assignment without declare (like \a=foo, ${1+a=$1}...) are
parsed normally as arguments to declare, and declare by itself
only does scalar assignments including to array members, never
array or hash assignment.
ksh supports:
$ ksh -c 'typeset a[1]=(1 2 3); typeset -p a'
typeset -a a=([1]=(1 2 3) )
but that's because it supports nested arrays. It makes
sense for bash to just return an error there (as it does
already).
declare would still end up do some indirect evaluation as part
of arithmetic evaluation in array subscript:
foo='a[0$(uname2)]=foo'
declare $foo
would still run uname before assigning foo to a[0]. I'd rather
it doesn't do that (only expand $(...) when litteral), but if we
fix it here, we'd need to fix it everywhere else as well.
Does that make sense?
BTW:
$ bash -c 'declare a[1]=(1 2 3); declare -p a'
declare -a a='([0]=1 [1]=2 [2]=3)'
$ bash -c 'declare a+=([1]=1 2 3); declare -p a'
bash: line 0: declare: `a+': not a valid identifier
declare -a a='([1]=1 2 3)'
How do I do ATM if I want a[1] to contain (1 2 3) with
declare?
--
Stephane
Re: declare a=$b if $a previously set as array
On 12/9/14 5:51 PM, konsolebox wrote: On Tue, Dec 9, 2014 at 7:29 AM, Linda Walsh b...@tlinx.org wrote: Instead of dumbing down bash, why not lobby for bash to record which variables contain tainted input -- and throw an error they are eval'ed (based on an option setting, of course)? For compatibility's sake I think it's a good idea to have an option (through shopt [and set / a command-line argument]) to make a strict behavior of declare in which assignment of variables are strictly the same as the way they are normally assigned without it. This is unnecessarily limiting. There's no reason to completely disallow constructs like `declare -x $one=$two' or even `declare -l a=$value'. The question is what to do about potentially dangerous -- from some perspectives -- uses of those constructs. So far we've identified compound assignment as one of those uses; assignment to an associative array using a subscript containing a command substitution might be another. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/10/14 5:43 AM, Stephane Chazelas wrote: It's ironic that local scoping is now implemented in virtually all POSIX shells but not specified by POSIX. It almost was, once. It's one of my great regrets that I got rid of my paper copy of Posix 1003.2 draft 9, which specified `local'. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
On 12/10/14 8:27 AM, konsolebox wrote: Note that that's the Bourne shell, not a POSIX sh. I was actually wanting to say that heirloom-sh is still a modernized shell (in code) despite being a strict clone of the original sh. It's not a `strict' clone'. It is a modified version of the SVR4 bourne shell as it appeared in Solaris. And that is where I made the assumption about other shells. I don't really study POSIX and I don't really care about it because I don't see it as the general standard to base from when creating compatible scripts. I'm curious about what you see as performing that function. Something Linux-specific is disqualified as being not cross-platform. Moreover I still respect bourne shell. That's fine, but note that the bourne shell has nothing to contribute to this discussion. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
2014-12-14 14:39:29 -0500, Chet Ramey:
[...]
~$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
~$ b='($(uname))' bash --norc -c 'declare -a a; declare a=$b; printf
%s\n $a ${a[0]}'
Linux
(on Linux Mint amd64).
$ ./bash --version
GNU bash, version 4.3.30(15)-release (x86_64-apple-darwin13.1.0)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ b='($(uname))' ./bash --norc -c 'declare -a a; declare a=$b; printf
%s\n $a ${a[0]}'
Darwin
Darwin
[...]
Yes I don't know what Mint (Ubuntu?) did, but it's a bit broken
see also:
$ bash -c 'f() { declare a[1]=foo; declare -p a; }; a[0]=x; f; declare -p a'
bash: line 0: declare: a: not found
declare -a a='([0]=x)'
(sorry about the confusion there).
--
Stephane
Re: declare a=$b if $a previously set as array
On 12/14/14 4:19 PM, Stephane Chazelas wrote:
Yes I don't know what Mint (Ubuntu?) did, but it's a bit broken
see also:
$ bash -c 'f() { declare a[1]=foo; declare -p a; }; a[0]=x; f; declare -p a'
bash: line 0: declare: a: not found
declare -a a='([0]=x)'
(sorry about the confusion there).
It might have been changed by a subsequent patch, but I can't see any
obvious candidates.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: declare a=$b if $a previously set as array
2014-12-14 20:48:45 +, Stephane Chazelas:
[...]
$ bash -c 'declare a[1]=(1 2 3); declare -p a'
declare -a a='([0]=1 [1]=2 [2]=3)'
$ bash -c 'declare a+=([1]=1 2 3); declare -p a'
bash: line 0: declare: `a+': not a valid identifier
declare -a a='([1]=1 2 3)'
How do I do ATM if I want a[1] to contain (1 2 3) with
declare?
[...]
I'm just realising the issue is nowhere as bad as I initially
thought it was.
x='($(uname2))' bash -c 'a[0]=x; declare a=$x'
is indeed a problem, but:
x='($(uname2))' bash -c 'f() { a[0]=x; declare a=$x; }; f'
is not, because when in a function, declare ignores variables by
the same name that have not been declared in that same scope
(even if they have been *set* in that context)
So it's more unlikely to be a problem than I thought it would be
as declare is mostly used in functions.
x='($(uname2))' bash -c 'f() { declare a[0]=x; declare a=$x; }; f'
would be a problem, but then why would one do that?
There's still a (security) issue with
declare -x/-l/-r
that may still be used in non-function contexts (not export or
readonly), and as result for things like:
saved=$(export -p var)
...
eval $saved
And with declare -g in functions, so probably still worth
addressing.
(the possible source of confusion issue, and compatibility
with ksh93 to consider as well)
--
Stephane
Re: declare a=$b if $a previously set as array
On 12/14/2014 08:45 PM, Chet Ramey wrote:
On 12/7/14 11:16 PM, Eduardo A. Bustamante López wrote:
On Sun, Dec 07, 2014 at 07:34:53PM -0800, Linda Walsh wrote:
Only if you properly quote external input.
Well, that's the whole point, as a script writer, I don't expect to get
arbitrary code execution here:
| dualbus@hp:~/t$ unset var; value='[$(ls -l)]=1 [2]=2'; declare -a
var=($value); declare -p var
| bash: total 0: syntax error in expression (error token is 0)
Yeah, that's what we're discussing.
Or here:
| dualbus@hp:~/t$ a=(1 2 3); k='a[$(ls -l)]'; echo ${a[k]}
| bash: total 0: syntax error in expression (error token is 0)
And I *shouldn't* have to worry about that.
But the ship has sailed on this one. Every shell that implements indexed
arrays does what bash does here.
This matches my observations. That's why I recommend not to use array
variables in shell scripts:
http://docs.fedoraproject.org/en-US/Fedora_Security_Team/1/html/Defensive_Coding/sect-Defensive_Coding-Shell-Types.html
--
Florian Weimer / Red Hat Product Security
Re: declare a=$b if $a previously set as array
On Mon, Dec 15, 2014 at 5:12 AM, Chet Ramey chet.ra...@case.edu wrote: On 12/10/14 8:27 AM, konsolebox wrote: Note that that's the Bourne shell, not a POSIX sh. I was actually wanting to say that heirloom-sh is still a modernized shell (in code) despite being a strict clone of the original sh. It's not a `strict' clone'. It is a modified version of the SVR4 bourne shell as it appeared in Solaris. I see. That was just my immediate impression of it sorry. And that is where I made the assumption about other shells. I don't really study POSIX and I don't really care about it because I don't see it as the general standard to base from when creating compatible scripts. I'm curious about what you see as performing that function. Something Linux-specific is disqualified as being not cross-platform. There's one well-known project I know that used to make itself compatible across many systems without minding compatibility about POSIX. I'm not sure if it has changed already both on target systems and in code (even if the description says so). http://sf.net/projects/rkhunter Moreover I still respect bourne shell. That's fine, but note that the bourne shell has nothing to contribute to this discussion. Sorry, I was actually also wanting to end the discussion myself. I just can't avoid making a reply when something is needing one. Cheers, konsolebox
Re: declare a=$b if $a previously set as array
On Mon, Dec 15, 2014 at 5:05 AM, Chet Ramey chet.ra...@case.edu wrote: On 12/9/14 5:51 PM, konsolebox wrote: On Tue, Dec 9, 2014 at 7:29 AM, Linda Walsh b...@tlinx.org wrote: Instead of dumbing down bash, why not lobby for bash to record which variables contain tainted input -- and throw an error they are eval'ed (based on an option setting, of course)? For compatibility's sake I think it's a good idea to have an option (through shopt [and set / a command-line argument]) to make a strict behavior of declare in which assignment of variables are strictly the same as the way they are normally assigned without it. This is unnecessarily limiting. There's no reason to completely disallow constructs like `declare -x $one=$two' or even `declare -l a=$value'. The question is what to do about potentially dangerous -- from some perspectives -- uses of those constructs. So far we've identified compound assignment as one of those uses; assignment to an associative array using a subscript containing a command substitution might be another. I'm not sure how about the former but 'declare -l a=$value' or `declare -i x='1 + 1'` is good to me. My only real concern about those constructs is that values assigned to variables should not be re-interpreted to make it appear as if it's part of the raw construct itself besides when explicitly required e.g. $value should remain as $value when assigned to `a` (besides transformation of uppercase characters to lowercase forms) and `'1 + 1'` should literally be assigned to `x` besides another transformation through evaluation of the arithmetic expression. Not only are those values can be allowed for transformation because they are explicitly required but also because they are safe in the sense that their form is not in the form of assignments like those of indexed arrays and associative arrays. They are also not synonymous to how eval does it. The best thing as well is that they can be re-evaluated after with another common function that's not necessarily part of declare's implementation that declare itself would be the one to manage the transformations i.e. `declare` would only care about assigning the raw value and let other functions handle the transformation of the variable's value after it. Any other variable that is not declared to have transformations should have values assigned to them -as is-. If it's a normal variable or an array variable, assignments like `a='[1]=2'` or `a=$something` should only give the variable the literal value of `[1]=2` and `$something` with no transformation at all. If `a` turns out to be an array variable (may it be an indexed or an associative array variable), `[1]=2` or value of `$something` should be assigned to index or key 0 instead (a[0]) therefore making it synonymous to `a[0]='[1]=2'` and `a[0]=$something`. No other forms of interpretation should be allowed. And naturally if you have both -i and -a or -A on declaration, values assigned to each element can also be transformed with arithmetic expression. This form of implementation should be prioritized. The former (declare -x $one=$two) is optional for the sake of compatibility or flexibility if needed. If we would allow forms like `declare -x $one=$two`, we should make sure that $two remains intact as a single value. If $one does not represent a valid variable name (a form with a key or index should be illegal), declare should simply throw an error. $one should not be subject to any form of transformation whether word splitting, pathname expansion, arithmetic evaluation, command substitution, etc. Of course, so is $two. Last thing is `declare -x $a_complete_assignment` should not be allowed; even quoted assignments like `declare -x $one=$two` simply because they are already a reinterpreted form i.e. they are already evaluated with a pair of single or double-quotes before declare tries to evaluate them and it is were declare's behavior becomes inconsistent and it is were everything starts going wrong. If we do allow it, how would we apply the proper rules to the expanded form of $any? - knowing that $any is already a terrible combination of raw constructs and values were literal values meant to be literal values become subject to misinterpretation as forms of assignment to an array. And what if `$a_complete_assignment` contain `x=$something`, would we expand $something or not? This just makes implementation of declare way more complicated, difficult and unclear. And yes, I do imply that declare should just act like a reserved word instead because it's for the best. Cheers, konsolebox
Re: declare a=$b if $a previously set as array
2014-12-10 06:33:00 +0800, konsolebox: [...] Not sure what you mean by that last sentence, but all of dash, yash, mksh, pdksh, ATT ksh, posh, zsh (so virtually all modern Bourne-like shells) do support local scopes each with variations of behaviour and syntax. I know ksh, pdksh and zsh supports local but I didn't know that dash does it too - or maybe I forgot. I always remember that it's a pure POSIX shell but apparently it's not. dash (and posh) follow (or at least used to) another standard, the Debian policy which is more or less POSIX with a few additions (from XSI, UP and its owns). In particular, it mandates a local builtin. So sh scripts that use local are guaranteed to work in Debian. However note that behaviour for local foo=bar is unspecified there, so sh scripts for Debian should use local foo; foo=bar (even those POSIX specifies export foo=bar (though not clearly enough)). Last time I checked, the Debian spec didn't specify the exact variable scoping (for instance whether dynamic á la ksh88/bash/dash/yash/pdksh or lexical a la ksh93). There have been discussions on specifying local scopes in the past, but it's hard to get a consensus. See the discussion at http://thread.gmane.org/gmane.comp.standards.posix.austin.general/8371/focus=8377 David Korn mentions that local scoping a la ksh was originally rejected by POSIX because ksh88 originally implemented dynamic scoping (like most shells now do, but most other languages don't). ksh changed to lexical scoping in ksh93 (probably partly to address that POSIX /requirement/) and now everyone else wants dynamic scoping because that's how most shells have implemented it in the interval. And sorry I actually meant some other. Consider a more conservative shell like heirloom-sh. [...] Note that that's the Bourne shell, not a POSIX sh. It's ironic that local scoping is now implemented in virtually all POSIX shells but not specified by POSIX. -- Stephane
Re: declare a=$b if $a previously set as array
2014-12-10 10:43:45 +, Stephane Chazelas: [...] In particular, it mandates a local builtin. So sh scripts that use local are guaranteed to work in Debian. However note that behaviour for local foo=bar is unspecified there, so sh scripts for Debian should use local foo; foo=bar (even those POSIX specifies export foo=bar (though not clearly enough)). [...] Sorry, that's wrong. Debian does allow local foo=bar (https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts), either it has changed or I was confusing it with LSB, I'm not sure now. I thought LSB was mandating local, but can't find a reference to it either. -- Stephane
Re: declare a=$b if $a previously set as array
On Wed, Dec 10, 2014 at 6:43 PM, Stephane Chazelas stephane.chaze...@gmail.com wrote: 2014-12-10 06:33:00 +0800, konsolebox: [...] Not sure what you mean by that last sentence, but all of dash, yash, mksh, pdksh, ATT ksh, posh, zsh (so virtually all modern Bourne-like shells) do support local scopes each with variations of behaviour and syntax. I know ksh, pdksh and zsh supports local but I didn't know that dash does it too - or maybe I forgot. I always remember that it's a pure POSIX shell but apparently it's not. dash (and posh) follow (or at least used to) another standard, the Debian policy which is more or less POSIX with a few additions (from XSI, UP and its owns). ... Thank you for the information. And sorry I actually meant some other. Consider a more conservative shell like heirloom-sh. [...] Note that that's the Bourne shell, not a POSIX sh. I was actually wanting to say that heirloom-sh is still a modernized shell (in code) despite being a strict clone of the original sh. I believe many people still base their scripts from it so I also see that as a shell to consider despite not being POSIX. It's ironic that local scoping is now implemented in virtually all POSIX shells but not specified by POSIX. And that is where I made the assumption about other shells. I don't really study POSIX and I don't really care about it because I don't see it as the general standard to base from when creating compatible scripts. Moreover I still respect bourne shell. See https://sourceforge.net/p/loader/code/ci/base/tree/loader.sh Cheers, konsolebox
Re: declare a=$b if $a previously set as array
Thanks konsolebox and Eduardo, glad to see I'm not the only one
seeing an issue in that.
A few additions inline:
2014-12-09 10:51:51 +0800, konsolebox:
[...]
(But I think those who used to recommend using declare as a general tool
for assigning values remotely over evil eval would love it since
it makes assignments like `declare -g $var=$something` possible
which naturally should have been illegal.)
[...]
BTW, it's worth noting (and maybe documenting) that word
splitting and filename generation are performed in:
declare -g $var=$something
(or declare ''var=$something or declare f\oo=$bar...)
So it must be indeed:
declare -g $var=$something
Anyhow, this problem would only happen if a variable in a specific
scope (and only in that scope) has already been set previously as an
array and it doesn't always happen if you write your scripts well.
It may be worth recommending people do a unset var before
doing a declare [-option] var
I actually find it better to mention that on a simple assignment like
declare var=$something, var's type doesn't change if it has already been
declared previously as an array.
Strictly speaking,
if it has already been declared *or assigned* (as in a[0]=x or
a=()) previously as an array *in the current scope*,
otherwise, the variable is declared as a scalar (but see below
for more complications), :
$ bash -c 'f() { declare a=1; declare -p a; }; a=(); f'
declare -- a=1
However, if a was exported or readonly even in an outer scope
before, it's still exported or readonly (that doesn't seem to be
the case for any of the other variable attributes):
$ a=3 bash -c 'f() { declare a=1; declare -p a; }; a=(); f'
declare -x a=1
$ a=3 bash -c 'f() { declare a=1; declare -p a; }; declare -r a=2; f'
bash: line 0: declare: a: readonly variable
declare -rx a=2
What that means is the local or declare is not a complete
solution to your scoping problem. You still need to resolve to
name spaces. See the eternal discussion between ksh and bash
about dynamic vs static scope for local variables as well.
ksh:
$ a=3 ksh -c 'function f { typeset a=1; echo $a; env | grep ^a; }; typeset
-r a=2; f'
1
(a in the function is not exported but the original a was
removed from the environment, not sure it's a lot better, but
it's probably as good as we can reasonably get without major
change of design).
$ a=0 ksh -c 'function f { typeset a=1; g; h; echo f: $a; }; function g {
a=2; }; function h { typeset a=3; };f'
f: 1
f doesn't have to worry about alien functions changing its
local variables under its feet.
[...]
Shared functions otoh should be responsible of making sure that their
work variables are placed as local. This is also a common
practice.
[...]
That doesn't apply if you want to use a function library written
in sh (the POSIX sh language doesn't have local scope except
via subshells).
Then again, that library should use its own namespace.
So, if good practice is followed all over the board, then that
issue is very unlikely to be hit.
Then again, I see no compelling reason not to fix it. Again, the
issue is aggravated by the fact that it's hard to detect (your
scripts will still work but will have a hidden vulnerability).
--
Stephane
Re: declare a=$b if $a previously set as array
2014-12-09 10:59:14 +, Stephane Chazelas:
[...]
$ a=0 ksh -c 'function f { typeset a=1; g; h; echo f: $a; }; function g {
a=2; }; function h { typeset a=3; };f'
f: 1
f doesn't have to worry about alien functions changing its
local variables under its feet.
[...]
(for clarification)
It still has to worry about alien functions declared using the
Bourne syntax though (as those are interpreted in their parent's
scope and don't have a separate scope except for the positional
parameters):
$ ksh -c 'function f { typeset a=1; g; echo $a; }; g() a=2; f'
2
(or sourced scripts or evaled variables...).
--
Stephane
Re: declare a=$b if $a previously set as array
On Tue, Dec 9, 2014 at 6:59 PM, Stephane Chazelas stephane.chaze...@gmail.com wrote: BTW, it's worth noting (and maybe documenting) that word splitting and filename generation are performed in: declare -g $var=$something (or declare ''var=$something or declare f\oo=$bar...) So it must be indeed: declare -g $var=$something It actually should be completely avoided. eval is the right tool for that job (besides a variable declared with -n in 4.3+). Anyhow, this problem would only happen if a variable in a specific scope (and only in that scope) has already been set previously as an array and it doesn't always happen if you write your scripts well. It may be worth recommending people do a unset var before doing a declare [-option] var I actually find it better to mention that on a simple assignment like declare var=$something, var's type doesn't change if it has already been declared previously as an array. Strictly speaking, if it has already been declared *or assigned* (as in a[0]=x or a=()) previously as an array *in the current scope*, otherwise, the variable is declared as a scalar (but see below for more complications), : Actually I referred to declared as to being declared by type. If it comes to assigning values, it is more of defining. As for bash, using an array form of assignment (e.g. a[0]=x; a=(x)) is an implied form of declaration + definition. Taking a quick glance at some of Bash's codes you would see that variables are dynamically converted from a non-array to an array type when it encounters an assignment (or perhaps and an expansion) that requires one. So saying that if it has already been declared previously as an array should suffice on any form of declaration whether there's an assignment of value or none. See http://en.wikipedia.org/wiki/Declaration_%28computer_programming%29. Clearly declare is already doing things more than what its name say. Not that I'm saying it should no longer assign values, but at the very least it should not be re-evaluating values. As for the other things, I was actually only concerned about the dangerous behavior that's synonymous to eval $b so I didn't really thought about discussing them. And I'm only referring to Bash when I made the suggestions. Other shells certainly does not support local scopes. Cheers, konsolebox
Re: declare a=$b if $a previously set as array
On Tue, Dec 09, 2014 at 08:55:02PM +0800, konsolebox wrote: declare -g $var=$something It actually should be completely avoided. eval is the right tool for that job (besides a variable declared with -n in 4.3+). Unfortunately, declare -n is the same as eval. We've had this discussion before, and I haven't changed my stance. At least with eval, you know what you're getting. declare -n looks like something different, but it's not. That's where the danger lies.
Re: declare a=$b if $a previously set as array
On Tue, Dec 9, 2014 at 7:29 AM, Linda Walsh b...@tlinx.org wrote: Instead of dumbing down bash, why not lobby for bash to record which variables contain tainted input -- and throw an error they are eval'ed (based on an option setting, of course)? For compatibility's sake I think it's a good idea to have an option (through shopt [and set / a command-line argument]) to make a strict behavior of declare in which assignment of variables are strictly the same as the way they are normally assigned without it. And I don't see it as dumbing down, but more of correcting. To me it's actually a wrong implementation more than a feature (i.e. having a builtin-like behavior like test / unset over a reserved word like behavior like [[ ]]. If there would have been a tool for that, it can never be declare as its function to me would give conflict. Better have another builtin tool for it that behaves similar to what I suggested before. Cheers, konsolebox
Re: declare a=$b if $a previously set as array
2014-12-07 19:34:53 -0800, Linda Walsh:
[...]
Stephane Chazelas wrote:
declare -l a=$external_input
he's entitled to expect $a to contain the lower case version of
$external_input whatever $external_input contain.
---
Only if you properly quote external input.
If you properly quote the external input I don't see the problem:
Does this example demonstrate your setup?
declare -a a=(1 2 3)
b='($(echo FOO))'
printf -v qb %q $b# here must quote the raw 'external input' string
declare -l a=$qb # redefining 'a' to be lower case
read c $a # read the quoted value printf %s\n
$c
($(echo foo)) # no execution -- just the case lowering you want
Am I missing something?
I think you're missing my point.
I'm saying that if a script writer writes:
declare a=$b
intending to declare the *scalar* varible $a as a copy of the
scalar variable $b (and remember that in ksh/bash, scalar
variables and arrays are not very differentiated, $a being
${a[0]}), and overlooked (or is not aware of (because that was
done by 3rd party code for instance)) the fact that the variable
was used as an array before (for instance because he used
a[0]=foo instead of a=foo for instance), then:
- it will work in most of the cases (and that's one aspect why
it's dangerous, because it's hard to detect).
- but you've got a code injection vulnerability (in the very
special case where $b starts with (.
- for no good reason. See ksh for a better syntax that doesn't
have the issue.
- and it's not consistent when the same assignment is done
without declare (and no, I don't agree declare is a mere
builtin as it's already parsed halfway between a builtin and an
assignment).
Now, if the script writer intends to make the *array* variable $a
a copy (well potentially changing the indices) of $b, he would
simply write:
declare -la a=(${b[@]})
I certainly don't expect him to have to resort to `printf %q`
here for that.
--
Stephane
Re: declare a=$b if $a previously set as array
Stephane Chazelas wrote:
2014-12-07 19:34:53 -0800, Linda Walsh:
[...]
Stephane Chazelas wrote:
declare -l a=$external_input
he's entitled to expect $a to contain the lower case version of
$external_input whatever $external_input contain.
---
Only if you properly quote external input.
If you properly quote the external input I don't see the problem:
Does this example demonstrate your setup?
declare -a a=(1 2 3)
b='($(echo FOO))'
printf -v qb %q $b# here must quote the raw 'external input' string
declare -l a=$qb # redefining 'a' to be lower case
read c $a # read the quoted value printf %s\n
$c
($(echo foo)) # no execution -- just the case lowering you want
Am I missing something?
I think you're missing my point.
I'm saying that if a script writer writes:
declare a=$b
intending to declare the *scalar* varible $a as a copy of the
scalar variable $b
How does the script writer **KNOW** $a and $b are scalar?
I'm using 'a=fmt' and 'b=out' for clarity in my example
but say I want to insert a line to print the current UID/EUID/time
hostname:
#!/bin/bash
### add lowercase info line of currently executing user:
unset fmt out
declare -x fmt='($(echo uid:$UID) $(echo euid:$EUID) $(date) $(uname -n))'
declare -a out
declare -l out=$fmt
echo out=${out[@]}
sudo bash --norc -c 'declare -a out;declare -l out=$fmt; echo
out=${out[@]}'
Run that and you get:
out=uid:5013 euid:5013 mon dec 8 12:39:06 pst 2014 ishtar
out=uid:0 euid:0 mon dec 8 12:39:06 pst 2014 ishtar
All of the terms are evaluated at the time of final execution.
--- This usage ***depends*** on delayed evaluation -- which you
claim is code injection. This is the way shell is supposed to
operate. The programmer ***HAS*** to choose when to cause the expression
to be evaluated depending on their need.
They need to ensure inputs are clean.
You may not like that programmers have to deal with such, and if it bothers
you enough, go to a more encapsulated language and environment (java,
javascript,
perl (has tainting tracking to help w/this), et al). But whatever the
language,
they all still have problems -- but at least are designed with more
protection
from their beginnings.
Shell wasn't. It was a way to run commands and allow
for variable and powerful behaviors based on quoting and substitution rules.
How would you address your problem w/o affecting programs like the above?
Re: declare a=$b if $a previously set as array
2014-12-08 12:48:05 -0800, Linda Walsh:
[...]
declare -x fmt='($(echo uid:$UID) $(echo euid:$EUID) $(date) $(uname -n))'
declare -a out
declare -l out=$fmt
echo out=${out[@]}
sudo bash --norc -c 'declare -a out;declare -l out=$fmt; echo
out=${out[@]}'
Run that and you get:
out=uid:5013 euid:5013 mon dec 8 12:39:06 pst 2014 ishtar
out=uid:0 euid:0 mon dec 8 12:39:06 pst 2014 ishtar
All of the terms are evaluated at the time of final execution.
--- This usage ***depends*** on delayed evaluation -- which you
claim is code injection. This is the way shell is supposed to
operate. The programmer ***HAS*** to choose when to cause the expression
to be evaluated depending on their need.
[...]
Hi Linda,
then change it to:
sudo bash --norc -c 'declare -a out;declare -l out; out=$fmt; echo
out=${out[@]}'
And you get the behaviour *I* expect (assign the content of $fmt
to out[0] (aka out)).
Would you then say that bash is broken there?
If I want the shell to evaluate the content of a variable as
code, I use eval. And I know it's a dangerous command and
that I should use it carefully. I don't expect declare to do
the job of eval, I don't expect declare to run commands, I
only expect it to declare variables (and possibly assign values
to them).
(BTW, don't forget to add back the Defaults env_reset to your
/etc/sudoers as your system is currently probably vulnerable to
local privilege escalation if you're using restricted sudoer
commands).
--
Stephane
Re: declare a=$b if $a previously set as array
On Mon, Dec 08, 2014 at 12:48:05PM -0800, Linda Walsh wrote: --- This usage ***depends*** on delayed evaluation -- which you claim is code injection. This is the way shell is supposed to operate. The programmer ***HAS*** to choose when to cause the expression to be evaluated depending on their need. If you want delayed evaluation, use 'eval', that's what it's for. Your use case is a lame excuse for a feature that's clearly more evil than useful. Also, it's *not* the way it's supposed to operate. Arrays are not POSIX, how can this be the way it's supposed to operate? Where do you get that from? | dualbus@hp ~/t % ls | bar foo | dualbus@hp ~/t % for shell in bash ksh93 mksh pdksh; do $shell -c 'typeset -a a b; x=(\$(ls)); typeset a=$x; b=$x; typeset -p a b'|sed s|^|$shell: |; done | bash: declare -a a='([0]=bar [1]=foo)' | bash: declare -a b='([0]=(\$(ls)))' | ksh93: a='($(ls))' | ksh93: typeset -a b='($(ls))' | mksh: typeset a='($(ls))' | mksh: typeset b='($(ls))' | pdksh: typeset a='($(ls))' | pdksh: typeset b='($(ls))' Bash is *evidently* the only shell in the family that does that delayed evaluation (i.e. *evaluates code when it SHOULD NOT*). Also, why is: declare foo=$bar different from foo=$bar The difference in ksh93 makes a little sense, you're resetting the variable's attributes with typeset, or defining a new variable. Introducing code evaluation, nope, doesn't make sense. The point of this kind of discussion is precisely to: - Hey, this feature is dangerous and kind of pointless - We should remove it, or at least disable it by default Not: - Hey, this feature is dangerous and kind of pointless - Sorry, that's how the shell is, live with it!
Re: declare a=$b if $a previously set as array
Eduardo A. Bustamante López wrote: On Mon, Dec 08, 2014 at 12:48:05PM -0800, Linda Walsh wrote: --- This usage ***depends*** on delayed evaluation -- which you claim is code injection. This is the way shell is supposed to operate. The programmer ***HAS*** to choose when to cause the expression to be evaluated depending on their need. If you want delayed evaluation, use 'eval', that's what it's for. Your use case is a lame excuse for a feature that's clearly more evil than useful. Evil? You are going to base your argument on religion? I figured out the problem in your program almost instantly because I have had to go about deciding to quote or not quote array args and because I save off arrays between processes so I can re-instantiate them in child processes (in lieu of exporting). Also, it's *not* the way it's supposed to operate. Arrays are not POSIX, how can this be the way it's supposed to operate? Where do you get that from? Of the shells you mention, which implemented arrays first? Bash was designed to be an extension in features and usability above /bin/sh -- the standard shell at the time and try to maintain posix compatibility while providing those extra features. Using the excuse that other, less advanced shells don't implement things the same way is a reason for some prefer bash, and others to prefer other shells. Trying to dumb down bash to the lowest common feature set of the less advanced shells you mention seems like going backwards. BTW, if you want to compare shells, please don't leave out 'zsh', as it seems to be the only other shell that has tried to implement advanced features and not just be another /bin/[k]sh compat-looking posix standard wannabe. Instead of dumbing down bash, why not lobby for bash to record which variables contain tainted input -- and throw an error they are eval'ed (based on an option setting, of course)?
Re: declare a=$b if $a previously set as array
On Sun, Dec 7, 2014 at 7:24 AM, Stephane Chazelas stephane.chaze...@gmail.com wrote: $ b='($(uname2))' bash -c 'a=(); declare -r a=$b' Linux I actually also don't like that form of implementation. Even if declare is a builtin and not a reserved word, it still has some special treatments that's different from a simple command. For example in a simple command like echo, something like 'echo a=(1 2 3 4)' would cause syntax error. Word splitting and pathname expansion also doesn't happen unless when assigning elements in an array. If declare is able to recognize assignments separated by spaces, I think it would have not been difficult avoiding $b to be evaluated since it's really not meant to be. It clearly is a simple form of assignment. Imo even if declare is a builtin (which is actually a little special over a normal one), it should have just emulated behavior of normal assignments and did nothing more than it besides declaring the type (and optionally have some other features not related to assignment). (But I think those who used to recommend using declare as a general tool for assigning values remotely over evil eval would love it since it makes assignments like `declare -g $var=$something` possible which naturally should have been illegal.) Anyhow, this problem would only happen if a variable in a specific scope (and only in that scope) has already been set previously as an array and it doesn't always happen if you write your scripts well. It may be worth recommending people do a unset var before doing a declare [-option] var I actually find it better to mention that on a simple assignment like declare var=$something, var's type doesn't change if it has already been declared previously as an array. As for the unusual evaluation of $b, I believe that that implementation should be changed and it's not enough to just warn users about it. (though in that case one may argue that they should use their own context and make sure they don't call other functions that modify variables in their parent context or be aware of what variables the functions they call may modify (recursively)). And to add, if a script is meant to be sourced, it should make sure that no unshared variable is still set after its execution ends. If it has to have variables that are meant to remain after it is sourced, then it should make sure that those variables use their own namespaces like having a prefix. As a shared script it is natural to avoid conflicts with variables of other scripts that would call you. This is always something you would always do whether declare is badly implemented or not. Shared functions otoh should be responsible of making sure that their work variables are placed as local. This is also a common practice. As for shared variables that would contain the results of the function, those that are meant to contain array values should only contain array values, and those non-array variables should always have non-array values e.g. __A0 for array and __ for non-array. Cheers, konsolebox
Re: declare a=$b if $a previously set as array
2014-12-06 20:19:21 -0500, Chet Ramey:
On 12/6/14 6:24 PM, Stephane Chazelas wrote:
Hiya,
this is potentially dangerous:
If $a was previously used as an array (or hash), then:
declare a=$external_input
So what you're saying is that blindly using external input can sometimes
have negative consequences?
In this case, it's not about sanitizing external input, at least
as far as the $external_input variable is concerned.
When a script writer writes:
declare -l a=$external_input
he's entitled to expect $a to contain the lower case version of
$external_input whatever $external_input contain.
If $external_input happens to contain ($(echo FOO)), I'd
expect $a to contain ($(echo foo)), not foo just because
$external_input happens to follow a specific pattern.
Where the script writer may be to blame is for running declare
when that a variable was previous declared as an array (though
that could have been done by code he invoked from 3rd party
libraries), or not anticipating that his code would be invoked
in contexts where that a variable may have been declared as an
array.
But even then, he can be excused not to imagine that
declare -l a=$external_input
could ever run an arbitrary commands.
And even if he intended to declare a as an array beforehand,
he could reasonably expect
declare -l a=$external_input
to do the same as
declare -l a; a=$external_input
That is, the same as a[0]=$external_input.
I'd say we have at least a documentation issue.
[...]
At some point, you have to put some burden of responsibility onto the
script writer.
Yes, definitely.
It's up to everyone to try and make this digital world a
safer place. In this case though, bash can easily do its bit by
changing this behaviour (I can't imagine anyone relying on the
behaviour as it is now).
If not changed, that behaviour may be worth documented.
declare's behavior is already documented:
Arrays are assigned to using compound assignments of the form
name=(value1 ... valuen), where each value is of the form
[subscript]=string. ... This syntax is also accepted by the declare
builtin.
a=$external_input
is not of the form name=(value ... valuen), it's of the form
var=$other_var.
And when doing it without declare, it works as expected.
The problem is when using declare
Similarly,
a=(foo)
Assigns (foo) to the element of indice 0 in the array a and
one could expect:
declare a=(foo)
to do the same.
ksh behaviour that only accepts ( when litteral and unquoted
(or zsh that doesn't accept it at all) is a bit saner IMO.
declare is a builtin, not a reserved word. At the point when it is
invoked, its arguments have already been expanded. The question is
how much argument-mangling you want to do, and how much is worth it.
It's already parsed differently from other builtins.
declare a=(foo bar)
works
echo a=(foo bar)
doesn't (and declare a=$b doesn't do word splitting on $b for
instance)
$ b='[0]=foo' bash -c 'declare a=($b); echo $a'
[0]=foo
does already the right thing (remember you fixed that after I
reported the bug a few years ago). I argue it's a similar issue
here.
[...]
$ b='($(uname))' bash -c 'declare -a a; declare a=$b; printf %s\n
$a ${a[0]}'
Linux
$ b='($(uname))' bash -c 'a=(); declare a=$b; printf %s\n $a
${a[0]}'
Linux
Linux
(I'd expect the same result for both).
You don't say what version of bash you're using, but I get the same results
for both commands back to bash-4.0 (when I quit testing).
~$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
~$ b='($(uname))' bash --norc -c 'declare -a a; declare a=$b; printf %s\n
$a ${a[0]}'
Linux
(on Linux Mint amd64).
It may be worth recommending people do a unset var before
doing a declare [-option] var unless they do intend to retain the
previous value and/or type
Sure, that's good programming practice. It should be recommended in a
bash programming guide -- and there are several excellent ones.
[...]
At least the bash doc could document what happens when using
declare on a variable that was already assigned or declared
(in the current context and not).
--
Stephane
Re: declare a=$b if $a previously set as array
Stephane Chazelas wrote: declare -l a=$external_input he's entitled to expect $a to contain the lower case version of $external_input whatever $external_input contain. --- Only if you properly quote external input. If you properly quote the external input I don't see the problem: Does this example demonstrate your setup? declare -a a=(1 2 3) b='($(echo FOO))' printf -v qb %q $b# here must quote the raw 'external input' string declare -l a=$qb # redefining 'a' to be lower case read c $a # read the quoted value printf %s\n $c ($(echo foo)) # no execution -- just the case lowering you want Am I missing something? If $external_input happens to contain ($(echo FOO)), I'd expect $a to contain ($(echo foo)), not foo just because $external_input happens to follow a specific pattern. Does the above work for you?
Re: declare a=$b if $a previously set as array
On Sun, Dec 07, 2014 at 07:34:53PM -0800, Linda Walsh wrote:
Only if you properly quote external input.
Well, that's the whole point, as a script writer, I don't expect to get
arbitrary code execution here:
| dualbus@hp:~/t$ unset var; value='[$(ls -l)]=1 [2]=2'; declare -a
var=($value); declare -p var
| bash: total 0: syntax error in expression (error token is 0)
Or here:
| dualbus@hp:~/t$ a=(1 2 3); k='a[$(ls -l)]'; echo ${a[k]}
| bash: total 0: syntax error in expression (error token is 0)
And I *shouldn't* have to worry about that.
I know that I can do many things to avoid that, but, why? It's tedious, and
error prone. It shouldn't be this painful to write scripts that take arbitrary
user input. We have enough with word-splitting and quote-all-the-things, to
also be worrying about that.
declare a=$b if $a previously set as array
Hiya,
this is potentially dangerous:
If $a was previously used as an array (or hash), then:
declare a=$external_input
(or:
declare -r a=$external_input
for instance (or -l...)... but not:
readonly a=$external_input
(or export)
)
becomes a code injection vulnerability:
$ b='($(uname2))' bash -c 'a=(); declare -r a=$b'
Linux
That could be an issue in scripts that are meant to be sourced
or functions that call other functions before using declare.
That's aggravated by the fact that the issue doesn't show up
unless $external_input starts with (.
If not changed, that behaviour may be worth documented.
ksh behaviour that only accepts ( when litteral and unquoted
(or zsh that doesn't accept it at all) is a bit saner IMO.
In ksh:
$ typeset -a a=(2); echo $a
(2)
There also seems to be a minor bug there in bash:
$ b='($(uname))' bash -c 'declare -a a; declare a=$b; printf %s\n $a
${a[0]}'
Linux
$ b='($(uname))' bash -c 'a=(); declare a=$b; printf %s\n $a ${a[0]}'
Linux
Linux
(I'd expect the same result for both).
Note that's also the case when the variable was previously
defined as integer (and that's also the case for mksh) but those
cases would usually be detected I think:
$ b='z[0$(uname2)]' bash -c 'declare -i a; declare a=$b; echo $a'
Linux
0
It may be worth recommending people do a unset var before
doing a declare [-option] var unless they do intend to retain the
previous value and/or type especially if they don't have control
on what was executed beforehand in the same context (though in
that case one may argue that they should use their own context
and make sure they don't call other functions that modify
variables in their parent context or be aware of what variables
the functions they call may modify (recursively)).
--
Stephane
Re: declare a=$b if $a previously set as array
On 12/6/14 6:24 PM, Stephane Chazelas wrote:
Hiya,
this is potentially dangerous:
If $a was previously used as an array (or hash), then:
declare a=$external_input
So what you're saying is that blindly using external input can sometimes
have negative consequences?
(or:
declare -r a=$external_input
for instance (or -l...)... but not:
readonly a=$external_input
(or export)
)
becomes a code injection vulnerability:
$ b='($(uname2))' bash -c 'a=(); declare -r a=$b'
Linux
That could be an issue in scripts that are meant to be sourced
or functions that call other functions before using declare.
That's aggravated by the fact that the issue doesn't show up
unless $external_input starts with (.
At some point, you have to put some burden of responsibility onto the
script writer.
If not changed, that behaviour may be worth documented.
declare's behavior is already documented:
Arrays are assigned to using compound assignments of the form
name=(value1 ... valuen), where each value is of the form
[subscript]=string. ... This syntax is also accepted by the declare
builtin.
ksh behaviour that only accepts ( when litteral and unquoted
(or zsh that doesn't accept it at all) is a bit saner IMO.
declare is a builtin, not a reserved word. At the point when it is
invoked, its arguments have already been expanded. The question is
how much argument-mangling you want to do, and how much is worth it.
In ksh:
$ typeset -a a=(2); echo $a
(2)
In ksh, typeset is essentially a reserved word. Doing a trace on that
command is illustrative; it appears that ksh performs the assignment
before setting the variable's attributes, which means it can more
closely emulate assignment statement behavior. I'm not sure how faithfully
that reflects what ksh is doing internally.
There also seems to be a minor bug there in bash:
$ b='($(uname))' bash -c 'declare -a a; declare a=$b; printf %s\n $a
${a[0]}'
Linux
$ b='($(uname))' bash -c 'a=(); declare a=$b; printf %s\n $a ${a[0]}'
Linux
Linux
(I'd expect the same result for both).
You don't say what version of bash you're using, but I get the same results
for both commands back to bash-4.0 (when I quit testing).
It may be worth recommending people do a unset var before
doing a declare [-option] var unless they do intend to retain the
previous value and/or type
Sure, that's good programming practice. It should be recommended in a
bash programming guide -- and there are several excellent ones.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
