Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Martin Kealey]

On Fri, 23 Feb 2024, Robert Elz wrote:
>   | Yes, that's exactly the point, to *avoid* dynamic scoping. I want the
>   | equivalent of Perl's "my", rather than Perl's "local".
>
> Lexical scoping does not, that is, cannot, work with the shell
> language as it is defined, if you want that you need to go and
> invent a whole new language.

I'm beginning to wonder if people are reading some different proposal from
what I wrote.

I'll say it again: unless a variable is explicitly marked as 'lexically
local' (however that's achieved) then it would continue to behave as it does
now. Any lexically local symbol table would only be consulted when an
appropriate declaration has been encountered.

You already know I'm keen on maintaining backwards compatibility, even for
stuff that appears at first blush to be "just a stupid bug", so if it sounds
like I'm suggesting breaking existing code, either I haven't explained it
properly, or the reader hasn't understood what I've written, or both.

> As it is thare are too many places in sh where correct operation
> requires access to specific vars, which makes no sense (ie: would
> not work as intended with lexically scoped local vars) - consider
> PATH CDPATH PWD OPTIND OPTARG REPLY (for shells that have it)
> and try to explain how they can be made to work as intended if
> made local, with lexical scoping.

Sure, and since you're not declaring them as lexicals, there's no problem
continuing with that.

(If someone declares a lexical variable that masks a reserved name, well
they get what they ask for. That doesn't make my entire proposal broken.)

> Regardless of how much people have tried to attempt to impose a more
> programming language model on sh vars, it simply does not work.

If you *take away* the current 'local', then of course it's not going to
work. That is not what I'm proposing.

> sh was designed [...]

Let's face it, sh was barely 'designed' at all; rather, it was defined by
"whatever the implementation does", and *that* was "mash together a macro
processing language, a batch processing language, and a command line
interface, and try and make something coherent".

I've mentioned Perl repeatedly because it provides excellent prior art for a
language moving from dynamic lifetime to lexical scope, while still keeping
the former as an option. The existing guts of Bash make it a bigger job than
if we were simply starting with a sane "programming" model, but it's
certainly not impossible.

>   | new code can avoid the craziness that comes from "everything is global,
>
> crazy or not, that is the sh model.  If you don't like it, use
> something different -- tclsh (or with graphics added, wish) is much
> more rigorous, and also much less friendly interactively.

Another way to think about this is that Bash already has "enable -f" that
can load a new function that was written in C and compiled to a binary
shared library.

What I'm proposing amounts to a new way to define functions, similar to ones
loaded by "enable -f", except that (a) they're interpreted rather than
complied, and (b) they haves almost the same syntax as the normal shell, but
(c) their internal variables can be completely hidden from the rest of the
program/script.

When such a function needs to reach out and touch a variable in the global
namespace, arguably it should have to try just a little harder to do so.

By "trying harder", I mean for one of these new-style functions to access an
outer variable Foo, you might need to 'import' it with a declaration, or you
might need to qualified the name where it's used, perhaps by adding a
prefix.

Here's an example I've just dreamed up (loosely borrowing some syntax from
ksh) simply to prove that this is possible:

* to reach up to the root of the function call stack ("main" or "global"):
 * .top.Foo=newvalue
 * ${.top.Foo}

* to reach into the calling function's local variables:
 * .caller.Foo=newvalue
 * ${.caller.Foo}

* to reach outside the innermost dynamic local (however far up the function
call chain that might be):

 * .local.out.Foo=newvalue
 * ${.local.out.Foo}

(Note that Perl simply defines that a list of two dozen or so built-in
variables are forced into the 'main::' package; the equivalent here would be
to make an exception (to the "trying harder" rule) for the existing special
variables, including any future additions whose name starts with 'BASH'.)

* to reach the variables of a lexically enclosing function:
 * .outer.Foo=newvalue
 * ${.outer.Foo}

I'm in two minds whether to 'fall back' to the global scope if there are no
lexical matches; obviously if it's not marked as a new-style function, then
the answer must be yes; but if it *is* recognisable as a new-style function,
sound software engineering principles would come down firmly on 'no'.

>   | even when we claim it's local" and "unset can even mess with the poor
>   | protection afforded by 'local'".
>
> I'm guessing you mean bash's bizarre unset definition wrt locals.
> I consider 

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Robert Elz]

Date:Fri, 23 Feb 2024 10:45:01 -0500
From:Chet Ramey 
Message-ID:  <3544bf1c-e8f6-4358-a281-24bd1be7d...@case.edu>

  | Do you mean that unset can mark local variables at the same
  | scope as unset,

no, but see below.

  | that unset can remove local variables at a previous scope,

yes, that one, but again, see below.   But note, had you said
'unset' there instead of 'remove', it would all be fine.

  | or that unset in a shell function can unset global variables?

no, of course not.

But I think the real issue here is that we would use the term 'scope'
with respect to shell variables at all, it really isn't appropriate.

The non-standard command 'local' which we all have, in slightly
different forms, but with similar enough effect that most simple
usages work fine in all shells, would better be called 'save'
(or something similar) as its real purpose is to preserve the
current state of a variable so that can be restored when the
functuon exits.  Once that's done, everything else (unset included)
just operates in the global var as if the 'local' (or 'save')
never happened.

This turns out to be quite similar to dynamic scoping, with
some subtle differences, but fits the shell's model of variable
use (everything is global, there is exactly one var named PATH
or IFS (etc)) better than actual dynamic scoping.

kre

ps: it is immaterial here what other effects the local command
has on the variable, once the previous val/state has been saved,
also having it unset the var, or assign it '', or alter
attributes (remove export for example) is unimportant.  Clearly
what, if anything, happens makes (or might) a difference to
the script, but none of that kind of variation affects the model.

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 2/23/24 8:13 AM, Robert Elz wrote:


   | even when we claim it's local" and "unset can even mess with the poor
   | protection afforded by 'local'".

I'm guessing you mean bash's bizarre unset definition wrt locals.
I consider that to just be a bug.  Unsetting a 'local' variable
should be identical to unsetting a global, as that is what
actually happens (should happen).  That there is a saved copy that
will be restored later is irrelevant.


Same question:

Do you mean that unset can mark local variables at the same scope as unset,
that unset can remove local variables at a previous scope, or that unset in
a shell function can unset global variables?

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 2/22/24 6:15 PM, Martin D Kealey wrote:


On Wed, 21 Feb 2024 at 08:09, Chet Ramey > wrote:


On 2/20/24 4:11 AM, Martin D Kealey wrote:
 > Ideally each function invocation would have its own variable namespace,
 > only using the global namespace as a fall-back, but that creates
 > complications with exported variables, so let's take baby steps to
get there.

That doesn't work with dynamic scoping at all.


Yes, that's exactly the point, to /avoid/ dynamic scoping. I want the 
equivalent of Perl's "my", rather than Perl's "local".


OK. Implementing static scoping for functions is not on my list.

Code that currently relies on dynamic scoping would continue to work, while 
new code can avoid the craziness that comes from "everything is global, 
even when we claim it's local" and "unset can even mess with the poor 
protection afforded by 'local'".


Do you mean that unset can mark local variables at the same scope as unset,
that unset can remove local variables at a previous scope, or that unset in
a shell function can unset global variables?

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Robert Elz]

Date:Fri, 23 Feb 2024 09:15:09 +1000
From:Martin D Kealey 
Message-ID:  


  | Yes, that's exactly the point, to *avoid* dynamic scoping. I want the
  | equivalent of Perl's "my", rather than Perl's "local".

Lexical scoping does not, that is, cannot, work with the shell
language as it is defined, if you want that you need to go and
invent a whole new language.

As it is thare are too many places in sh where correct operation
requires access to specific vars, which makes no sense (ie: would
not work as intended with lexically scoped local vars) - consider
PATH CDPATH PWD OPTIND OPTARG REPLY (for shells that have it)
and try to explain how they can be made to work as intended if
made local, with lexical scoping.

Regardless of how much people have tried to attempt to impose
a more programming language model on sh vars, it simply does
not work.   sh was designed to operate using an entirely global
namespace - the best way to think of 'local' is 'save the value
and attributes of this var as they are now, and restore them when
the current function invocation terminates' - and absolutely no
more than that.  All operations continue to use a single global
namespace, and because of that, everything simply works as it
should without issues.  Deviate even a little and there get to be
all kinds of problems.

  | new code can avoid the craziness that comes from "everything is global,

crazy or not, that is the sh model.  If you don't like it, use
something different -- tclsh (or with graphics added, wish) is much
more rigorous, and also much less friendly interactively.

  | even when we claim it's local" and "unset can even mess with the poor
  | protection afforded by 'local'".

I'm guessing you mean bash's bizarre unset definition wrt locals.
I consider that to just be a bug.  Unsetting a 'local' variable
should be identical to unsetting a global, as that is what
actually happens (should happen).  That there is a saved copy that
will be restored later is irrelevant.

kre

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Martin D Kealey]

On Wed, 21 Feb 2024 at 08:09, Chet Ramey  wrote:

> On 2/20/24 4:11 AM, Martin D Kealey wrote:
> > Ideally each function invocation would have its own variable namespace,
> > only using the global namespace as a fall-back, but that creates
> > complications with exported variables, so let's take baby steps to get
> there.
>
> That doesn't work with dynamic scoping at all.
>

Yes, that's exactly the point, to *avoid* dynamic scoping. I want the
equivalent of Perl's "my", rather than Perl's "local".

Perhaps I should clarify that I'm using "global namespace" to mean what
Perl calls a "package" (except Bash has only one), and that it *includes*
the dynamic effects of dynamic "local" statements. Therefore "using the
global namespace as a fall-back" means precisely that variables that are
not explicitly lexically scoped would continue to behave as they do now.

Code that currently relies on dynamic scoping would continue to work, while
new code can avoid the craziness that comes from "everything is global,
even when we claim it's local" and "unset can even mess with the poor
protection afforded by 'local'".

-Martin

PS:
Lexical variable scope implies that nested functions require multiple
'local' symbol tables to be active concurrently, effectively as in a chain.
That in turn means the local variables associated with a function
invocation would need to outlive that invocation if there are lexically
nested functions that access them.

If lexical variable declarations were declarative (immediately effective
during parsing) rather than procedural (only becoming effective when the
flow of control passes over them), that would be advantageous: a single
local symbol table could be kept with the parse tree for each function,
with an index into a per-invocation storage array. Then some time in the
future, symbol look-ups could be done during the parsing phase, leaving a
'local reference by index' entry in the parse tree. (Having a fixed symbol
table implies that the behaviour of the 'unset' command would subtly
change; for the global symbol table it would continue to behave as now, but
for the symbol tables associated with functions, it would place an 'unset'
marker in the slot, rather than deleting the name. There may be corner
cases where that's a detectable change, but since it's opt-in I think
that's acceptable. And that means some new mechanism to implement 'upvar'
would be needed.)

Being effective during parsing would argue for 'local' being a keyword,
albeit one that mostly behaves as if it's a command, but I can see that
some would argue that it's too much change, so I wouldn't object to leaving
'local' as it is and defining a new keyword for this purpose; my preference
would be one of 'my', 'var', or 'use local' (where 'use' is a general
'during parsing' keyword).

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 2/20/24 4:11 AM, Martin D Kealey wrote:

So yes please, I'd like "local" to push a new variable definition that 
inherits nothing from any outer one: not name-ref, not read-only, not array 
(of any kind), not assignment modifiers (integer, upper-case, lower-case), 
and above all, not any previous values.


You're in luck.


Ideally each function invocation would have its own variable namespace, 
only using the global namespace as a fall-back, but that creates 
complications with exported variables, so let's take baby steps to get there.


That doesn't work with dynamic scoping at all.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Koichi Murase]

2024年2月20日(火) 18:12 Martin D Kealey :
> So yes please, I'd like "local" to push a new variable definition that
> inherits nothing from any outer one: not name-ref, not read-only, not array
> (of any kind), not assignment modifiers (integer, upper-case, lower-case),
> and above all, not any previous values.

In my understanding, Bash behaves as you hope from the beginning
except for the global readonly variables (2.04+) and the export
attribute. I also checked the behavior of old versions now, but Bash
seems to behave in that way at least in Bash 2.0. If I'd raise some
exceptions, I thought in the past there was a bug that some array
attribute or behavior were inherited to a local variable (but don't
remember exactly), but that's just a bug found in a very limited
situation. Also, the recently added option `shopt -s localvar_inherit'
turns on the behavior you hate, but it's not enabled by default.

> Ideally each function invocation would have its own variable namespace,
> only using the global namespace as a fall-back,

That breaks so many existing scripts.

> Maybe this would be twisting 'local' too much, and it warrants creating a
> new keyword such as 'var'?

Perl's `local' is dynamic scoping, but Perl later added `my' for the
lexical scoping. Eric Blake suggested `local -s' [1]. Alex xmb
suggested `local -L' [2].

[1] https://lists.gnu.org/archive/html/bug-bash/2010-11/msg00036.html
[2] https://lists.gnu.org/archive/html/help-bash/2021-05/msg00094.html

--
Koichi

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Martin D Kealey]

On Sat, 17 Feb 2024 at 02:32, Chet Ramey  wrote:

> Let's say we take the approach of restricting attribute changes on readonly
> variables to export/trace/local.
>
> Should it be an error to attempt to set other attributes (it already is
> with nameref), or should declare silently ignore it?
>

I would prefer to make "local" behave as much as possible like a true
lexically scoped declaration in "regular" languages.

Much as I hate the effect on backwards compatibility, I hate the current
situation even more: it's not possible to write a re-usable general-purpose
utility function because the function has to avoid overriding outer
variables that might differ in their attributes from what the function
needs. Arrays and read-only are particularly problematic.

So yes please, I'd like "local" to push a new variable definition that
inherits nothing from any outer one: not name-ref, not read-only, not array
(of any kind), not assignment modifiers (integer, upper-case, lower-case),
and above all, not any previous values.

Ideally each function invocation would have its own variable namespace,
only using the global namespace as a fall-back, but that creates
complications with exported variables, so let's take baby steps to get
there.

Maybe this would be twisting 'local' too much, and it warrants creating a
new keyword such as 'var'?

-Martin

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 2/16/24 12:14 PM, Grisha Levit wrote:


Let's say we take the approach of restricting attribute changes on readonly
variables to export/trace/local.

Should it be an error to attempt to set other attributes (it already is
with nameref), or should declare silently ignore it?



I think an error (probably just the usual sh_readonly one) would be good.


I agree.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Grisha Levit]

On Fri, Feb 16, 2024 at 11:32 AM Chet Ramey  wrote:
>
> On 2/13/24 12:41 PM, Grisha Levit wrote:
> > On Wed, Jan 31, 2024, 14:10 Chet Ramey  > > wrote:
> >
> > Well, is this a "my arm hurts when I do this" problem, or should bash
> > restrict the types of attributes that can be set on readonly variables?
> >
> > For the attributes that only affect future assignments ([iluc]), I think
> > think it makes sense to either (a) restrict the ability to change these
> > attributes on readonly variables or (b) make internal assignments that
> > force new values on readonly variables ignore such attributes.
>
> Let's say we take the approach of restricting attribute changes on readonly
> variables to export/trace/local.
>
> Should it be an error to attempt to set other attributes (it already is
> with nameref), or should declare silently ignore it?


I think an error (probably just the usual sh_readonly one) would be good.

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 2/13/24 12:41 PM, Grisha Levit wrote:
On Wed, Jan 31, 2024, 14:10 Chet Ramey > wrote:


Well, is this a "my arm hurts when I do this" problem, or should bash
restrict the types of attributes that can be set on readonly variables?


For the attributes that only affect future assignments ([iluc]), I think 
think it makes sense to either (a) restrict the ability to change these 
attributes on readonly variables or (b) make internal assignments that 
force new values on readonly variables ignore such attributes.


Let's say we take the approach of restricting attribute changes on readonly
variables to export/trace/local.

Should it be an error to attempt to set other attributes (it already is
with nameref), or should declare silently ignore it?

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Grisha Levit]

On Wed, Jan 31, 2024, 14:10 Chet Ramey  wrote:

> Well, is this a "my arm hurts when I do this" problem, or should bash
> restrict the types of attributes that can be set on readonly variables?
>

For the attributes that only affect future assignments ([iluc]), I think
think it makes sense to either (a) restrict the ability to change these
attributes on readonly variables or (b) make internal assignments that
force new values on readonly variables ignore such attributes.

Option (a) makes the behavior more transparent to the user, but I can't say
if the added complexity is worth it. My ASS_NOEVAL patch was meant as an
attempt at option (b).

If code were to be added to restrict adding the above attributes, then
restricting [aA] might make sense too. While converting a readonly scalar
variable to an array does have a user-visible consequence -- an exported
scalar variable is no longer present in the environment after being
converted to an array -- one can already toggle the export attribute of
readonly variables.

This leaves the trace attribute, which doesn't do anything for variables,
but I've occasionally found it useful for debugging Bash issues so would
put in a weak vote to not restrict its modification.

>

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Chet Ramey]

On 1/24/24 7:06 PM, Emanuele Torre wrote:

I recently discovered that it is possible to set attributes on readonly
variables


You've been able to set attributes on readonly variables forever. Even the
historical Bourne shell allowed you to export readonly variables.



   $ declare -p PPID
   declare -ir PPID="759437"
   $ declare -l PPID
   $ declare -p PPID
   declare -irl PPID="759437"

I noticed SHELLOPTS and BASHOPTS among the default readonly variables
set by bash.
They are modified by bash whenever you set a set option or shopt.

I tried setting the -i attribute on them, and got this result:


Well, is this a "my arm hurts when I do this" problem, or should bash
restrict the types of attributes that can be set on readonly variables?


The arithmetic syntax error caused by trying to assign the new list of
options to SHELLOPTS prevents bash from changing its value, and
re-adding the readonly attribute to SHELLOPTS.


That's clearly a problem that needs to be fixed.


Then SHELLOPTS can be modified, and unset freely. :-)

The same is possible with BASHOPTS.


Sure, it's a problem that you can do it, but it's not really harmful.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/