Re: Unset array doesn't work

[Robert Elz]

Date:Fri, 2 Mar 2018 14:43:02 -0500
From:Chet Ramey 
Message-ID:  <4502a0e5-0600-d294-9af2-4e9eeb0a0...@case.edu>

My final comments on this subject:

  | Perhaps. But bash has never done this. Not from day one. That's 30 years.

That a bug (be it a design bug, or a coding bug) has existed a long tiime
does not make it any less a bug.

I have been using bash for essentially all that time (from before you took
over maintainership) and I never knew it worked like that.   From comments
here (where some people far more knowledgable about bash and its
internals than I are to be found) I suspect that very few other people know
about it either.

  | This is how bash dynamic scoping works. The exception for the declaration/
  | unset at the current scope was added 16 years ago, and the existing
  | behavior was already entrenched.

And yet when that change to the entrenched behaviour was made,
there were no complaints?   And there's no option to switch back to
the previous way?   Kind of suggests just how important everyone
believes the original method was, doesn't it?

  | I can see doing this and allowing it to be toggled by a shell option.

A suggestion:   Do that for bash 5, and in the alpha release, make
the option default to cause things to work the opposite way than
happens now (so the option needs to be explicitly changed to get
the current behaviour).   I know that's the opposite of what would
usually be done in order to retain backwards compat, but for this,
I think it would be a useful test to see if anyone notices the difference.
You can always change it for beta/final releases if there are issues.
If not, perhaps the option can just go away (then or later.)

  | > Lastly, where does the notion of "remove" come from?
  |
  | As a way to describe the historical bash behavior, it works.

Yes, that I understand.   My issue is that I believe this is colouring
your thoughts on just what "unset" is - same as the "appear/be"
(trivial seeming) semantic issue you commented on in another message.

That is, it appears to me as if you believe that "unset" (as a state, not
the command here) implies "non-existing".   That's never been correct.

The converse is correct - a variable that does not exist appears as
an unset variable when referenced.

There are (even ignoring the unset command) too many ways
(in bash, as well as other shells) to get variables that patently
obviously "exist" in some form or other but are unset.

The most obvious example is

export newvar

after that

echo ${newvar-unset}

prints "unset".   Sometime later if we give newvar a value, it, and its
new value are exported - demonstrating that the export attribute was
remembered (ie: "newvar" existed before it was set - it must have done
in order to retain an atttribute).

jinx$ export newvar
jinx$ echo ${newvar-unset}
unset
jinx$ newvar=set
jinx$ printenv newvar
set
jinx$ echo $BASH_VERSION
4.4.12(1)-release

All shells that function correctly behave that way.   In bash (and in
several other shells, though not all, due to unrelated differences)
the same is true of the local command in a function...

var=set
func() { local var; echo ${var-unset}; }
echo $var ; func; echo $var

prints "set" "unset" "set"  showing that in the function there is
a var (which must still exist as its value is retained in the global
scope) which is unset.

Even if the model was that the "var" in the function is something
completely unrelated to the global "var" (as it would be in C for
example) we can still see that the local command must have
created some state for it, as otherwise the echo would be
referring to the global "var".

This can be more obviously seen in

var=set
func() { [ -n "$1" ] && local var; echo ${var-unset}; }
echo $var ; func x; func; echo $var

The same code either accesses the local or global var
depending upon whether we happened to have executed
the local command or not - indicating that command sets
some atttibute, and still makes an unset var.

Again this is all as it should be (given the assumption that "local"
creates an unset version of the variable, which is a rational choice,
if not the one made by all shells - a topic we have discussed in
another context.)

The best model for all of this is that set/unset is an atttibute of a
shell variable, just like exported, read-only (and any more that
exist in various shells) and that when we reference a variable
which does not (or perhaps even just did not) exist at all, then
it is made to appear as if it were an unset variable, and this is
the special case, rather than considering leaving some state
around to remember attributes of vars that happen to be unset
but must continue to exist as being what is unusual or special.

An analogy is the unlink sys call, which is not "remove".   If you
think of it as remove, and that its function is to remove files, then
things get way too complex to describe the behaviour.   But if
it just 

Re: misleading error message from variable modifier

[don fong]

Chet, thanks for the explanation about CHANGES.  i am not familiar with the
bash release process.

as for this:

> I didn't think the tests were necessary.

what standard are you using to judge whether tests are necessary?  does the
bash project have any coverage metrics?

on general principles, when someone makes a change, there ought to be a
test to show that it does what it's supposed to.  that's just good
engineering.  there also ought to be sufficient pre-existing tests to show
that the change didn't break anything else.  that goes double when patches
are being submitted by outsiders and newcomers (like me).

before i submitted my patch, i looked around in the test directory to see
if there were any tests that covered the behavior i was changing.  i didn't
see any tests of the variable modifiers at all.  that struck me as a
situation that should be changed.  so i added some tests to cover my own
change plus a bit more.

as far as i can tell, the vast majority of the C code here seems to lack
unit tests as well.

for a widely used program like bash, IMHO it'd be worthwhile to have
automated tests (either unit tests or functional tests) that cover every
documented feature.

if that hasn't been the case in the past, i propose that it be adopted as a
new goal.  there should at least be a rule that all new code should be
covered by tests.  i'm astonished to learn that the tests that i
contributed were simply discarded.






On Fri, Mar 2, 2018 at 12:21 PM, Chet Ramey  wrote:

> On 3/1/18 11:37 PM, don fong wrote:
> > Chet, thanks.  in subst.c there is code that looks similar to what i had
> > suggested.  but i don't see the tests that i submitted.  i also don't see
> > the change listed in CHANGES?
>
> I didn't think the tests were necessary. And the ChangeLog file is the
> one to look at (eventually resolves to CWRU/CWRU.chlog, as Clark noted).
> The CHANGES file is updated as part of release engineering; as you might
> suspect, I'm working on bash-5.0-alpha.
>
> Chet
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>

Re: misleading error message from variable modifier

[Chet Ramey]

On 3/1/18 11:37 PM, don fong wrote:
> Chet, thanks.  in subst.c there is code that looks similar to what i had
> suggested.  but i don't see the tests that i submitted.  i also don't see
> the change listed in CHANGES?

I didn't think the tests were necessary. And the ChangeLog file is the
one to look at (eventually resolves to CWRU/CWRU.chlog, as Clark noted).
The CHANGES file is updated as part of release engineering; as you might
suspect, I'm working on bash-5.0-alpha.

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Typo (?) in man page

[Chet Ramey]

On 3/2/18 10:00 AM, Fabrizio Di Pilla wrote:

> I think it should say:
> 
> command2  is  executed  if,  and only if, command1 returns a non-zero exit
> status.
> 
> Notice the added ',' so it will be same as the AND example.
> I don't know if this is really a typo since English is not my primary
> language.

Either way is acceptable, but there's no reason not to make the two cases
consistent. Thanks for the report.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Unset array doesn't work

[Chet Ramey]

On 2/28/18 5:38 PM, Robert Elz wrote:

> Were I you, I would simply change the "in a previous scope" behaviour to
> match the "in the current scope" behaviour.   That makes it consistent,
> and rational.

I can see doing that with the new behavior controlled by a shell option.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Unset array doesn't work

[Chet Ramey]

On 2/28/18 5:38 PM, Robert Elz wrote:

> I would change "appear" to "be" in the description of the current scope
> behaviour.

Now we're straying into semantics. If the variable were `really' unset,
as opposed to just appearing unset, that could be construed as meaning
the global instance (or any previous local instances) would be gone, too.
I could see pairing it with text that says the variable at the current or
nearest previous instance is unset and remains so until that function
returns.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Unset array doesn't work

[Chet Ramey]

On 2/28/18 2:00 PM, Robert Elz wrote:
> Date:Wed, 28 Feb 2018 10:27:23 -0500
> From:Chet Ramey 
> Message-ID:  
> 
>   | These are two different cases -- same context vs. a previous context. Your
>   | example is not the same as the original poster's.
> 
> OK, though I am not sure why that should make a difference.

It does.


>   | The global var is left unchanged, and satisfies references to $var with 
> v1.
> 
> The second clause of that is what is brain dead.   There should be a simple
> invariant in all shell contexts, which is
> 
>   unset X
>   Y=$X
> 
> assigns the null string to Y (except in the case where X is read only, which
> is not a case of concern here.)

Perhaps. But bash has never done this. Not from day one. That's 30 years.

> Always.
> 
> No (other) exceptions.
> 
> None.

Never done it that way. Ever. Not once. (We can do this all day. But levity
aside.)

> 
> Similarly the expression ${X-unset} should always produce "unset" if X
> has been unset and not subsequently given a value.
> 
> Whatever else is believed about how variables should work, that part should
> remain true - if we cannot rely upon unset working to create an unset 
> variable,
> then we're lost.

This is how bash dynamic scoping works. The exception for the declaration/
unset at the current scope was added 16 years ago, and the existing
behavior was already entrenched.


>   | You can mark any local instances in previous
>   | scopes (or the nearest previous scope that has a local instance of
>   | that variable) with a special `unset' attribute that will cause
>   | references to it to behave as if it were unset, leaving the global
>   | value unchanged. 
> 
> Yes.   That's the correct way I believe.

I can see doing this and allowing it to be toggled by a shell option.

> Lastly, where does the notion of "remove" come from?

As a way to describe the historical bash behavior, it works.  An instance
of a variable with a given name at a previous scope is removed, and another
instance of that variable at another scope is revealed.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: add generated files to .gitignore

[Chet Ramey]

On 2/27/18 11:46 AM, don fong wrote:
> Chet, thanks for the suggestion.
> 
> i still wonder what's the objection to changing .gitignore?

I don't think it will be useful to me, since I curate the commits I
make to the various branches, but I don't have any real objection if
it will help others.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: tilde character not interpreted in variable set

[Nick Chambers]

On 3/2/18, 9:03 AM, "bug-bash on behalf of Koncz, Istvan (Extern)" 
 wrote:


Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
-DCONF_VENDOR='unknown'
-DLOCALEDIR='/home/konczi.ext/apps/bin/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -g -O2 -Wno-
parentheses -Wno-format-security
uname output: Linux OED-EX-DIG-L 4.13.0-36-generic #40-Ubuntu SMP Fri
Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.4
Patch Level: 18
Release Status: release

Description:
when i use ~/ for home folder it works in command line e.g. ls
-la ~/.vimrc works, but when i add it to a variable e.g. foo="~/.vimrc"
then ~/ will not be replaced to $HOME value like in command line, but i
think it should be. ~ is a special character so must be escapped like
\~ if i want the char, but without escape it should provide the home
like in ls -la ~/ command. In my oppinion it is a bug.
Current workaround is in shell script to use $HOME not ~/ but would be
better if ~/ would be consistent and behave everywhere as same.

Are you sure this is a bug? Consider the following:

ubuntu:~ nchambers$ echo ~
/home/nchambers
ubuntu:~ nchambers$ echo "~"
~
ubuntu:~ nchambers$ tmp=~
ubuntu:~ nchambers$ echo "$tmp"
/home/nchambers
ubuntu:~ nchambers$ tmp="~"
ubuntu:~ nchambers$ echo "$tmp"
~
ubuntu:~ nchambers$

~ is not an actual path, but rather a symbolic value expanded by the shell to 
whatever value is in HOME. Wrapping it in quotes prevents this expansion.

Repeat-By:
foo="~/.vimrc"
echo $foo
~/.vimrc
ls -la $foo
ls: cannot access '~/.vimrc': No such file or directory

ls -la ~/.vimrc
-rw-r--r-- 1 konczi.ext users 19 Jan  2 12:49 /home/konczi.ext/.vimrc

Fix:
i would replace ~/ or ~ with $HOME value when there is a
variable definition. If \~ is used, no replacement is needed because of
the escaping.



--
Mit freundlichen Grüßen/Kind regards

Istvan Koncz




Istvan Koncz
OEDIV Digital Services (ODS)

OEDIV Oetker Daten- und Informationsverarbeitung KG
Bechterdisser Str. 10
33719 Bielefeld, Germany

phone:   +49 521
fax: +49 521 26050 445
mailto:   istvan.ko...@oediv.de
web:   www.oediv.de


Gesellschaftssitz: Bielefeld
Registergericht: Amtsgericht Bielefeld HRA 13532
Der Umwelt zuliebe: E-Mails und Anhänge nur ausdrucken, wenn unbedingt 
nötig.
Save paper - print e-mails and attachments only when absolutely necessary.

_

Dieses E-Mail kann Informationen enthalten, die vertraulich sind und/oder 
dem Berufsgeheimnis unterfallen.
Diese Information ist nur für den Gebrauch durch die in diesem E-Mail 
benannte Person oder Rechtseinheit
bestimmt. Jede(r) unautorisierte Durchsicht, Gebrauch, Verwendung, 
Offenlegung oder Verbreitung ist verboten.
Falls Sie nicht der beabsichtigte Empfänger sind, bitten wir Sie, den 
Absender durch Antwort E-Mail zu
benachrichtigen und das empfangene E-Mail dauerhaft zu löschen sowie alle 
Kopien hiervon zu vernichten.
Vielen Dank.
Da über das Internet versandte E-Mails während des Übermittlungsprozesses 
leicht verfälscht und/oder unter
fremden Namen erstellt werden können, übernehmen wir keine Verantwortung 
für den Inhalt der E-Mail oder der
Anhänge und folglich kann der Inhalt des E-Mails kein rechtlich bindendes 
Angebot und keine rechtlich
bindende Annahme eines Angebots begründen sofern nicht ausdrücklich 
schriftlich anders vereinbart. Diese
E-Mail dient ausschließlich dem Informationsaustausch. Es gelten unsere 
Allgemeinen Geschäftsbedingungen.
Wir unternehmen alle Anstrengungen, um unser Netzwerk von Viren 
freizuhalten. Dennoch sollten Sie dieses
E-Mail und seine Anhänge auf Viren überprüfen, da wir keine Verantwortung 
für Computerviren übernehmen,
die durch dieses Mail unbeabsichtigt übermittelt werden könnten.

This e-mail may contain information that is confidential and/or privileged. 
This information is intended only
for the use of the individual or entity named in this e-mail. Any 
unauthorized review, use, disclosure or
distribution is prohibited. If you are not the 

Re: tilde character not interpreted in variable set

[Chet Ramey]

On 3/2/18 7:14 AM, Koncz, Istvan (Extern) wrote:

> Bash Version: 4.4
> Patch Level: 18
> Release Status: release
> 
> Description:
> when i use ~/ for home folder it works in command line e.g. ls
> -la ~/.vimrc works, but when i add it to a variable e.g. foo="~/.vimrc"
> then ~/ will not be replaced to $HOME value like in command line, but i
> think it should be. ~ is a special character so must be escapped like
> \~ if i want the char, but without escape it should provide the home
> like in ls -la ~/ command. In my oppinion it is a bug.

It's not a bug. The tilde is quoted, and is not one of the special
characters that is interpreted inside double quotes.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

tilde character not interpreted in variable set

[Koncz, Istvan (Extern)]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
-DCONF_VENDOR='unknown'
-DLOCALEDIR='/home/konczi.ext/apps/bin/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -g -O2 -Wno-
parentheses -Wno-format-security
uname output: Linux OED-EX-DIG-L 4.13.0-36-generic #40-Ubuntu SMP Fri
Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.4
Patch Level: 18
Release Status: release

Description:
when i use ~/ for home folder it works in command line e.g. ls
-la ~/.vimrc works, but when i add it to a variable e.g. foo="~/.vimrc"
then ~/ will not be replaced to $HOME value like in command line, but i
think it should be. ~ is a special character so must be escapped like
\~ if i want the char, but without escape it should provide the home
like in ls -la ~/ command. In my oppinion it is a bug.
Current workaround is in shell script to use $HOME not ~/ but would be
better if ~/ would be consistent and behave everywhere as same.

Repeat-By:
foo="~/.vimrc"
echo $foo
~/.vimrc
ls -la $foo
ls: cannot access '~/.vimrc': No such file or directory

ls -la ~/.vimrc
-rw-r--r-- 1 konczi.ext users 19 Jan  2 12:49 /home/konczi.ext/.vimrc

Fix:
i would replace ~/ or ~ with $HOME value when there is a
variable definition. If \~ is used, no replacement is needed because of
the escaping.



--
Mit freundlichen Grüßen/Kind regards

Istvan Koncz



Istvan Koncz
OEDIV Digital Services (ODS)

OEDIV Oetker Daten- und Informationsverarbeitung KG
Bechterdisser Str. 10
33719 Bielefeld, Germany

phone:   +49 521
fax: +49 521 26050 445
mailto:   istvan.ko...@oediv.de
web:   www.oediv.de

Gesellschaftssitz: Bielefeld
Registergericht: Amtsgericht Bielefeld HRA 13532
Der Umwelt zuliebe: E-Mails und Anhänge nur ausdrucken, wenn unbedingt nötig.
Save paper - print e-mails and attachments only when absolutely necessary.

_

Dieses E-Mail kann Informationen enthalten, die vertraulich sind und/oder dem 
Berufsgeheimnis unterfallen.
Diese Information ist nur für den Gebrauch durch die in diesem E-Mail benannte 
Person oder Rechtseinheit
bestimmt. Jede(r) unautorisierte Durchsicht, Gebrauch, Verwendung, Offenlegung 
oder Verbreitung ist verboten.
Falls Sie nicht der beabsichtigte Empfänger sind, bitten wir Sie, den Absender 
durch Antwort E-Mail zu
benachrichtigen und das empfangene E-Mail dauerhaft zu löschen sowie alle 
Kopien hiervon zu vernichten.
Vielen Dank.
Da über das Internet versandte E-Mails während des Übermittlungsprozesses 
leicht verfälscht und/oder unter
fremden Namen erstellt werden können, übernehmen wir keine Verantwortung für 
den Inhalt der E-Mail oder der
Anhänge und folglich kann der Inhalt des E-Mails kein rechtlich bindendes 
Angebot und keine rechtlich
bindende Annahme eines Angebots begründen sofern nicht ausdrücklich schriftlich 
anders vereinbart. Diese
E-Mail dient ausschließlich dem Informationsaustausch. Es gelten unsere 
Allgemeinen Geschäftsbedingungen.
Wir unternehmen alle Anstrengungen, um unser Netzwerk von Viren freizuhalten. 
Dennoch sollten Sie dieses
E-Mail und seine Anhänge auf Viren überprüfen, da wir keine Verantwortung für 
Computerviren übernehmen,
die durch dieses Mail unbeabsichtigt übermittelt werden könnten.

This e-mail may contain information that is confidential and/or privileged. 
This information is intended only
for the use of the individual or entity named in this e-mail. Any unauthorized 
review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please 
immediately contact the sender by
reply e-mail and permanently delete the original message and destroy all copies 
thereof. Thank you.
We do not enter into binding agreements via e-mail absent express written 
consent. As any e-mail sent over the
internet can be improperly altered electronically during the process of 
transmission or be sent under the name
of a third person, we assume no responsibility for the content of the e-mail or 
any of its attachments and,
consequently, the content of this e-mail shall not constitute a legally binding 
offer or acceptance of an
offer, unless otherwise agreed in writing. This e-mail is only intended to 
exchange information. Our standard
terms and conditions are applicable. We make every effort to keep our network 
free from viruses. However, you
do need to scan this e-mail and any attachments to it for viruses as we can 
take no responsibility for any
computer virus which might be 

Typo (?) in man page

[Fabrizio Di Pilla]

Hi bash!

First of all, just want to thank you all for all the great work you are
doing.

I was reading the bash man page and found something that I think is a typo.
Under the paragraph of *Lists*, when talking about the AND list, it says:

command2 is executed if, and only if, command1 returns an exit status of
zero.


A few lines bellow, when talking about the OR list, it says:

command2  is  executed  if  and only if command1 returns a non-zero exit
status.


I think it should say:

command2  is  executed  if,  and only if, command1 returns a non-zero exit
status.

Notice the added ',' so it will be same as the AND example.
I don't know if this is really a typo since English is not my primary
language.


What do you think?

Regards.