Re: [feature request] Add ".sh" or ".bash" extension to tmpfile generated by `fc`

[Koichi Murase]

2024年9月21日(土) 1:09 Chet Ramey :
> On 9/20/24 2:41 AM, shynur . wrote:
> > `fc` will create a temporary file named something
> > like "bash-fc.Esf9by", which seldom benefits from
> > editors that use *suffixes* to infer what syntax
> > highlighting should be enabled.
>
> Do these editors have an option that sets the language for syntax checking?

Emacs has `auto-mode-alist'. VS Code has `files.associations'.

Re: please make the commit log clean

[Koichi Murase]

2024年8月20日(火) 5:52 Koichi Murase :
> 2024年8月20日(火) 2:25 Martin D Kealey :
> > Perhaps a compromise would be to put the documentation in a directory
> > that's not inside the source code directory, so it's easier to `git diff`
> > just one or the other.  (In practice, that would mean moving some of the
> > code into a new subdirectory.)
>
> One can put " diff=no" in .git/info/attributes to
> ignore these files in `git diff'. Mine is something like

I forgot to mention that I I have the following lines in my ~/.gitconfig:

[diff "no"]
command = /bin/true

Re: please make the commit log clean

[Koichi Murase]

2024年8月20日(火) 2:25 Martin D Kealey :
> Perhaps a compromise would be to put the documentation in a directory
> that's not inside the source code directory, so it's easier to `git diff`
> just one or the other.  (In practice, that would mean moving some of the
> code into a new subdirectory.)

One can put " diff=no" in .git/info/attributes to
ignore these files in `git diff'. Mine is something like

/configure diff=no
*.cache diff=no
/m4/* diff=no
/po/*.gmo diff=no
/po/*.po diff=no
/doc/bashref.aux  diff=no
/doc/bashref.bt   diff=no
/doc/bashref.bts  diff=no
/doc/bashref.cp   diff=no
/doc/bashref.cps  diff=no
/doc/bashref.dvi  diff=no
/doc/bashref.fn   diff=no
/doc/bashref.fns  diff=no
/doc/bashref.html diff=no
/doc/bashref.info diff=no
/doc/bashref.ky   diff=no
/doc/bashref.log  diff=no
/doc/bashref.pdf  diff=no
/doc/bashref.pg   diff=no
/doc/bashref.ps   diff=no
/doc/bashref.rw   diff=no
/doc/bashref.rws  diff=no
/doc/bashref.tmp  diff=no
/doc/bashref.toc  diff=no
/doc/bashref.tp   diff=no
/doc/bashref.vr   diff=no
/doc/bashref.vrs  diff=no
/doc/bash.html diff=no
/doc/bash.info diff=no
/doc/bash.pdf  diff=no
/doc/bash.ps   diff=no
/doc/builtins.0   diff=no
/doc/builtins.pdf diff=no
/doc/builtins.ps  diff=no
/doc/bash.0  diff=no
/doc/rbash.0 diff=no

Re: Potentially misleading documentation of SECONDS variable

[Koichi Murase]

2024年8月7日(水) 0:24 Bash-help via Bug reports for the GNU Bourne Again
SHell :
> Reading the manual at
> 
> regarding the SECONDS variable, it states that
> " [..] Assignment to this variable resets the count to the value assigned,
> and the expanded value becomes the value assigned plus the number
> of seconds since the assignment. [..]"
>
> [...]
>
> As we sleep less than a full second the expanded value of SECONDS should 
> never be greater than 0 but it sometimes is. I guess this is because the 
> assignment might occur, say X.7 seconds and the expanded value will then read 
> (X+1).2 which would be rounded down to (X+1).

I think the next sentence in the manual

https://www.gnu.org/software/bash/manual/bash.html#Bash-Variables
> [...] The number of seconds at shell
> invocation and the current time are always determined by querying the
> system clock. [...]

is supposed to imply that technically. It says "The number of seconds
at shell invocation", so the starting time point is considered to have
only the "second" resolution and doesn't have the subsecond
resolution. Then, when a value is assigned, the starting time point
would be updated to the number of "seconds" at the assignment.

For example, when the assignment occurs at time X.7, the recorded time
becomes just X (with the fraction part being truncated). Then, the
value obtained from SECONDS is incremented when the clock becomes
(X+1).0. This is consistent with the observed behavior.

However, I agree that this is implicit and ambiguous. A minimal
modification is probably something like

diff --git a/doc/bash.1 b/doc/bash.1
index 1f0a23d3..3ace21e9 100644
--- a/doc/bash.1
+++ b/doc/bash.1
@@ -2091,7 +2091,7 @@ the value returned upon subsequent
 references is
 the number of seconds since the assignment plus the value assigned.
 The number of seconds at shell invocation and the current time are always
-determined by querying the system clock.
+determined by querying the system clock at the resolution of a second.
 If
 .SM
 .B SECONDS
diff --git a/doc/bashref.texi b/doc/bashref.texi
index 510b43f6..6ee0083b 100644
--- a/doc/bashref.texi
+++ b/doc/bashref.texi
@@ -7038,7 +7038,7 @@ Assignment to this variable resets the count to
the value assigned, and the
 expanded value becomes the value assigned plus the number of seconds
 since the assignment.
 The number of seconds at shell invocation and the current time are always
-determined by querying the system clock.
+determined by querying the system clock at the resolution of a second.
 If @env{SECONDS}
 is unset, it loses its special properties,
 even if it is subsequently reset.

--
Koichi

[PATCH] fix `shopt -u force_fignore' affecting unrelated parts

[Koichi Murase]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -march=native -O3
uname output: Linux chatoyancy 6.5.12-100.fc37.x86_64 #1 SMP
PREEMPT_DYNAMIC Mon Nov 20 22:28:44 UTC 2023 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.3
Patch Level: 0
Release Status: alpha

Description:

  The filtering by `shopt -u force_fignore' is also applied to the
  suppression of completons unrelated to FIGNORE, which results in
  strange behaviors in command- and directory-name completions with
  `shopt -u force_fignore'.  This is because `_ignore_completion_names
  ()' (bashline.c) keeps the original completion list with `shopt -u
  force_fignore' regardless of whether the current filtering is that
  by FIGNORE.  The `force_fignore == 0' should take effect only for
  the filtering by FIGNORE.

  Bash 3.0..devel are affected.  I haven't confirmed it with the
  actual binary, but I think Bash 2.02..2.05b are also affected when
  the compiler option `-DNO_FORCE_FIGNORE' is specified.

  The problem was identified after the report originally submitted by
  Maëlan  to bash-completion:
  https://github.com/scop/bash-completion/issues/1229 .

Repeat-By (case 1 - bash_progcomp_ignore_filenames):

  $ bash --norc
  $ rm -rf tmpdir && mkdir tmpdir && cd tmpdir
  $ touch hello.txt
  $ shopt -s force_fignore
  $ compgen -d  <-- Nothing is generated as expected
  $ shopt -u force_fignore
  $ compgen -d
  hello.txt <-- This is unexpected.

  In this case, `_ignore_completion_names ()' is called by
  `bash_progcomp_ignore_filenames ()' (bashline.c) to remove all
  non-directory filenames.

Repeat-By (case 2 - bash_ignore_filenames):

  $ bash --norc
  $ rm -rf tmpdir && mkdir tmpdir && cd tmpdir
  $ touch nonexistent-command-name.txt
  $ shopt -s force_fignore
  $ nonexistent-command-name[TAB]  <-- Nothing happens as expected
  $ shopt -u force_fignore
  $ nonexistent-command-name[TAB]  <-- This completes the line as follows:
  $ nonexistent-command-name.txt   <-- This is unexpecdted.

  In this case, `_ignore_completion_names ()' is called by
  `bash_ignore_filenames ()' (bashline.c) to suppress non-directory
  filenames in the current directory for the command-name completions.

Repeat-By (case 3 - bash_ignore_everything):

  $ bash --norc
  $ rm -rf tmpdir && mkdir tmpdir && cd tmpdir
  $ touch hello.txt
  $ shopt -s no_empty_cmd_completion
  $ shopt -s force_fignore
  $ [TAB]  <-- Nothing happens as expected
  $ shopt -u force_fignore
  $ [TAB]  <-- This completes the line as follows:
  $ hello.txt  <-- This is unexpected.

  In this case, `_ignore_completion_names ()' is called by
  `bash_ignore_everything ()' (bashline.c) to implement `shopt -s
  no_empty_command_completion'.

Fix:

  In this patch, `_ignore_completion_names ()' is changed to restore
  the original completion list with `shopt -u force_fignore' only when
  the completion list becomes empty by the FIGNORE filtering.  This
  patch was created and tested with the devel branch.

  The function `_ignore_completion_names ()' (bashline.c) is called by
  the four functions in bashline.c:

  * filename_completion_ignore ()
  * bash_progcomp_ignore_filenames ()
  * bash_ignore_filenames ()
  * bash_ignore_everything ()

  The first one is the function that performs the filtering by
  FIGNORE, and its behavior should be kept.  The second to fourth
  functions are the ones related to case 1..3 in Repeat-By section of
  this report, respectively, so I think their behavior should not be
  affecetd by `shopt -s force_fignore'. The function
  `_ignore_completion_names ()' can detect whether the current
  filtering is that by FIGNORE by checking its second argument,
  `name_func'.

Change history:

  The current strange behaviors seem to be present from Bash 3.0 where
  `force_fignore' is first introduced, and are still present in the
  devel branch.  The filtering by force_fignore was introduced by
  commit d3a24ed2 (2011-11-29, "Initial devel branch import from
  bash-3.0-alpha").  This commit includes everything from the
  3.0-alpha release, so a finer change history is unavailable.  Before
  that version, `force_fignore' was implemented as a C-macro `#define
  NO_FORCE_FIGNORE', which was undefined (and hence effectively the
  same as `shopt -s force_fignore') by default.  The C-macro was
  introduce in 2.02 (commit cce855bc, 1998-04-18, "Imported from
  ../bash-2.02.tar.gz.").  Before that, the behavior was always like
  `shopt -s force_fignore'.

---
 bashline.c | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/bashline.c b/bashline.c
index 9cdd9bc4..174f5311 100644
--- a/bashline.c
+++ b/bashline.c
@@ -3072,6 +3072,8 @@ _ignore_completion_names (char **names, sh_ignore_func_t 
*name_func)
   size_t idx, nidx;
   char **o

Re: proposed BASH_SOURCE_PATH

[Koichi Murase]

2024年6月20日(木) 5:47 Léa Gris :
> Look like you did not find a proper answer there. Here is one simple
> that involve no sub-shell at all and does exactly what your sub-shell
> version does.
>
> declare -r SCRIPT_DIR=${BASH_SOURCE[0]%/*}

This doesn't work as explained by Will. BASH_SOURCE doesn't contain a
slash when the script is specified as e.g. `bash a.sh' or `source
a.sh'.

> [...] it is not system agnostic anymore.

POSIX 2024 containing realpath(1) was just published, though I'm not
sure when/whether it will be available in very old systems.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月17日(月) 15:56 Léa Gris :
> Le 16/06/2024 à 23:44, Zachary Santer écrivait :
> Anyway, this could be handled with existing bash string replacement feature:
>
> localeFormatted=3,1415
> declare -i numvar=${localeFormatted/,/.}

Is the radix character in an arbitrary locale ensured to be either `.'
or `,'? I think we should in general need to do

  declare -i numvar=${localeFormatted/[!0-9]/.}

Anyway, this is non-trivial for users. I anticipate that people in
typical locales won't care about it, so many scripts would do
`numvar=$localeFormatted` and are tested under en_US.UTF-8 or locales
with `.' being the radix character Then, those scripts are later
broken under random locales.

In my opinion, the locale-formatted strings should only appear in the
strings presented to users, such as in the output of `printf'. It is
easier and maintainable to normalize all the internal representations
to be the C floating-point literals.

--
Koichi

Re: set -a leads to truncated output from ps

[Koichi Murase]

2024年6月15日(土) 22:48 Koichi Murase :
> 2024年6月15日(土) 13:48 Oğuz :
> > On Saturday, June 15, 2024, Koichi Murase  wrote:
> >> Also, the behavior of the `ps' command honoring COLUMNS to format its
> >> output is also an expected one.
> >
> > The fact that programs giving precedence to COLUMNS over the
> > terminal API are so common is an argument against exporting COLUMNS
> > behind the user's back though.
>
> I'm not sure how it is against exporting COLUMNS. I agree that it may
> cause unexpected behaviors when a user or a script uses the utilities
> using COLUMNS in the wrong way, but the existence of the programs
> using COLUMNS implies that those programs expect the users to export
> it when the user likes to do it. If those programs would not expect
> COLUMNS to be exported, why do these programs read the environment
> variable COLUMNS in the first place?

Sorry, I think I misunderstood the part "behind the user's back". Now I
guess you were talking about the behavior of `checkwinsize' when `set
-a' is enabled? In that part of my original reply, I talked about the
case where a user or a script intentionally exports those variables.

--
Koichi

Re: set -a leads to truncated output from ps

[Koichi Murase]

2024年6月15日(土) 13:48 Oğuz :
> On Saturday, June 15, 2024, Koichi Murase  wrote:
>> Also, the behavior of the `ps' command honoring COLUMNS to format its
>> output is also an expected one.
>
> The fact that programs giving precedence to COLUMNS over the
> terminal API are so common is an argument against exporting COLUMNS
> behind the user's back though.

I'm not sure how it is against exporting COLUMNS. I agree that it may
cause unexpected behaviors when a user or a script uses the utilities
using COLUMNS in the wrong way, but the existence of the programs
using COLUMNS implies that those programs expect the users to export
it when the user likes to do it. If those programs would not expect
COLUMNS to be exported, why do these programs read the environment
variable COLUMNS in the first place?

> Right now, if you're dealing with such a program while `set -a' is in effect,

Is this a reply to me? My argument was that the script is still broken
even when the user does *not* use `set -a'. Or you repeat the
arguments to the general audience.

> in order to suppress COLUMNS you need to unexport it before every command:

In Bash, when you know you are going to use programs honoring COLUMNS
and you want to disable that, you can just run `shopt -u checkwinsize'
and `unset -v COLUMNS' only once at the beginning of the script.

However, for the portable scripts, I now agree that it is a problem
that Bash seems to enable `checkwinsize' even in the POSIX mode.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月15日(土) 17:24 Léa Gris :
> Indeed printf only knows the string value. But Bash know the variable
> has a numeric flag when doing the value expansion, so it expands it
> using the current LC_NUMERIC locale in this specific case.

Ah, OK. I misunderstood your suggestion somehow differently (such that
the expansion result would still use the period as a radix character,
but the `print' builtin would automatically convert the format to the
one with the locale-dependent radix character before it starts to
interpret the arguments).

> # Numeric format use comma as radix separator
> LC_NUMERIC=fr_FR.UTF8
>
> # Numeric variable uses the C locale format internally
> declare -i floatVar=3.1415
>
> # Printf does not have to know about the variable type
> # as it is expanded in the current LC_NUMERIC format
> printf '%.2f\n' "$floatVar"
> # Prints 3,14

In this case, what is actually stored in the variable floatVar? `3.14'
or `3,14' (probably the former if I do not misunderstand)? In other
words, at which point does the conversion happens, in the assignment
phase or the parameter expansion phase? If the conversion happens in
the assignment phase, `floatVar' would not be able to use in
arithmetic expressions, which seems unreasonable. If the conversion
happens in the parameter expansion phase, it is different from the
traditional behavior of parameter expansions.

> # Same when it is expanded to assign another variable
> # stringVar=3,1415
> # Since the string expansion of numeric variable uses the
> # LC_NUMERIC format
> stringVar=$floatVar
>
> # Same rule applies when expanding a numerical context to a string:
> echo "$((2 * 3.14))"
> # Prints the string 6,28 because LC_NUMERIC=fr_FR.UTF8
>
> # Do not use string expansion to assign numeric value or it will fail
> # with different locale. Use a numeric context to assign.
> # Do:
> (( arc = 2 * floatVar ))
> # Don't:
> # arc=$((2 * floatVar ))

So this means that the arithmetic expansions $(()) would produce a
format incompatible with the arithmetic expression. This seems
inconsistent to me, but it might be a valid option if there is no
other option.



I still feel it would be best if POSIX could be updated to allow the
implementations to extend the interpretation when the conversion by
strtod(3) fails. The current restriction is not an explicit one but
something deduced from the four statements of the POSIX:

1) the floating-point conversion (%f, etc) of the printf utility is
performed by strtod(3),
2) strtod(3) uses LC_NUMERIC to determine the radix character,
3) strtod(3) "fails" by setting endptr = nptr when the string does not
have an expected form,
4) the printf utility needs to print a diagnostic message and fail
when the argument is not converted to a value appropriate to the
conversion specification (%f, etc.).

I doubt that the current restriction on the printf utility not
allowing C floating-point literals is intentional. Is there a reason
that we should not interpret C floating-point literals (except the
passive reason that a non-trivial combination of POSIX statements does
not allow it)? I feel it is better to allow the printf implementations
to extend the conversion rather than trying to invent strange and
inconsistent behaviors among the arithmetic expressions, the
arithmetic expansions, and the parameter expansions just to work
around the problem.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月14日(金) 16:18 Léa Gris :
> Another elegant option would be to expand the existing variables' i flag
> to tell the variable is numeric rather than integer.
>
> Then have printf handle argument variables with the numeric flag as
> using the LC_NUMERIC=C floating-point format with dot radix point.

In that design, how does the `printf' builtin know that the argument
string is generated by a specific variable with the numeric flag?  The
`printf' builtin is a (builtin) command that receives arguments as
strings.  The arguments do not carry the contexts of how the strings
are generated, (though some of the exceptions are the array
assignments of the assignment builtins).

--
Koichi

Re: set -a leads to truncated output from ps

[Koichi Murase]

2024年6月14日(金) 16:30 Alain BROSSARD :

Even if the behavior of `set -a' is modified, your script (or the
third-party script that you rely on?) is still broken when the user
explicitly gives COLUMNS and LINES the export attribute, which is a
*typical usage of those variables*.  This should be a bug in the
script expecting `ps ax' to not consider the environment COLUMNS.



> COLUMN is clearly meant to be an internal variable of bash given the
> description of how it is used;

What do you mean by ``the internal variable''?  COLUMNS and LINES are
traditionally variables to store the terminal width and height, and
these are expected to be updated to have the correct values.  Bash's
behavior of updating COLUMNS and LINES after the termination of
foreground jobs and SIGWINCH is consistent with that convention.
Also, the behavior of the `ps' command honoring COLUMNS to format its
output is also an expected one.  The problem is that the script tries
to used the ``formatted'' output of `ps ax' (specifically, column
COMMAND) to extract information.

I have to say it is typical to make COLUMNS and LINES environment
variables.  For the shell frameworks using those variables to work,
those variables do not need to be exported.  However, they need to be
exported for CUI applications written in languages that don't provide
a way to get the terminal width, such as AWK, and pure C without
expecting POSIX's TIOCGSIZE.  Then, users are expected to export those
variables if the users want to use such programs.  Actually the `ps'
command is one of them.

You can also observe this convention explicitly in the output of the
`resize' command (provided by xterm).  The `resize' command without
arguments outputs the following text:

  $ resize
  COLUMNS=239;
  LINES=75;
  export COLUMNS LINES;

which are expected to be used as

  $ eval "$(resize)"

> Honestly, I don't know where to go with this issue. Bash's behavior
> should be consistent and easily understood in order to have reliable
> scripts. Clearly this case doesn't respect this.

I'm not sure which one is more consistent, but exporting COLUMNS on
checkwinsize seems also consistent with the POSIX requirement on `cd',
`read', and other commands implicitly setting variables.  The POSIX
standard reads the ``assignments'' includes all the implicit changes
to the variables, so it is consistent that the implicit change of
COLUMNS by checkwinsize is also considered for `-a'.

--
Koichi

Re: Echoing commands

[Koichi Murase]

2024年6月13日(木) 5:20 Angelo Borsotti :
> This is not the same as debugging, for which set -x is devoted.
> "set -x" makes the ensuing commands be printed, but prefixed
> with "+ ", which makes the result look ugly,

PS4= (as Greg has replied)

> not to mention that
> the following "set +x" is echoed too (there are hacks to suppress
> the "set +x" output, but they are just hacks).

What are the hacks? What are the problems with them? You can have aliases:

alias @echo-on='set -x'
alias @echo-off='{ set +x; } 2>/dev/null'

> I would stress the importance of this: the purpose of scripts is
> to execute commands, informing the caller of what they execute,
> purged of all the builtins and other calculations the script does
> in order to come to the commands.

I don't think the purpose of scripts is to execute only the external
commands in general. The tasks implemented by builtins and other
calculations are allowed to be the purpose of scripts.

> Many solutions are posted in the internet, all of them are hacks with
> lots of edge cases that make them fail in presence of commands
> containing special characters, redirections, substitutions, etc.

Have you tried `set -v'? `set -v' is not a hack, (though there doesn't
seem to be a way to suppress echoing set +v).

The detailed design of how the logged commands should be filtered and
formatted depends on the purpose of each user. We already have `set
-x' and `set -v'. We can adjust the format by PS4 for `set -x'. It's
unrealistic to add a new option for every new preference of detailed
filtering and formatting. If it is really needed, you can implement
your filtering through the DEBUG trap, but it would finally be best to
explicitly write `echo COMMAND' before every COMMAND that you think is
important (unless you are lazy). It's stable and flexible.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月6日(木) 18:09 Léa Gris :
> Le 06/06/2024 à 10:29, Koichi Murase écrivait :
> > 2024年6月6日(木) 15:59 Léa Gris :
> >> Le 05/06/2024 à 17:09, Koichi Murase écrivait :
> >>> 2024年6月5日(水) 21:41 Zachary Santer :
> >>>> Bash could potentially detect floating point literals within
> >>>> arithmetic expansions and adjust the operations to use floating point
> >>>> math in that case. [...]
> >>>
> >>> ksh and zsh are already behaving in that way, and if Bash would
> >>> support the floating-point arithmetic, Bash should follow their
> >>> behavior.
> >>
> >> Bash isn't even consistent with the floating point data/input format
> >> when using printf '%f\n' "$float_string" as it depends on LC_NUMERIC 
> >> locale.
> >
> > Maybe I miss your point, but literals in arithmetic expressions and
> > the printf format are unrelated.
>
> They are in Bash. Bash use the locale format for floating-point number
> arguments.

The arguments of `printf' are not in an arithmetic context. This is
the case even for the currently supported integers. If it were an
arithmetic context, « printf %d 1+1 » should have printed `2', but the
actual result is an error because the arguments of the `printf'
builtin are unrelated to the arithmetic expression.

Though, I see your point. It is inconvenient that we cannot pass the
results of arithmetic evaluations to the `printf' builtin. This
appears to be an issue of the printf builtin. I think the `printf'
builtin should be extended to interpret both forms of the numbers, the
locale-dependent formatted number and the floating-point literals.

A problem is POSIX. I checked POSIX which states that the arguments of
the `printf' utility for %f, etc. shall be interpreted by strtod(),
and that strtod() shall use the locale-defined radix character and
shall fail if it is in an unexpected format. By combining these
statements, POSIX does not allow extension of the number format in the
arguments of the printf builtin, which seems an unreasonable
restriction.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月6日(木) 15:59 Léa Gris :
> Le 05/06/2024 à 17:09, Koichi Murase écrivait :
> > 2024年6月5日(水) 21:41 Zachary Santer :
> >> Bash could potentially detect floating point literals within
> >> arithmetic expansions and adjust the operations to use floating point
> >> math in that case. [...]
> >
> > ksh and zsh are already behaving in that way, and if Bash would
> > support the floating-point arithmetic, Bash should follow their
> > behavior.
>
> Bash isn't even consistent with the floating point data/input format
> when using printf '%f\n' "$float_string" as it depends on LC_NUMERIC locale.

Maybe I miss your point, but literals in arithmetic expressions and
the printf format are unrelated.

Even if Bash is not consistent with ksh/zsh for the printf format, the
floating-point arithmetic of Bash can be made consistent with ksh/zsh.
The floating-point support in arithmetic expressions is not specified
by POSIX/C. Bash is not required anything by POSIX/C about its support
for floating-point arithmetic.

Actually, arithmetic expressions are defined as a subset of C language
in POSIX, so it is natural to follow the floating-point literals in C
if Bash would support floating-point arithmetic. The floating-point
literals in C are unrelated to the result of printf and independent of
the locale where the C compiler runs. E.g. the decimal point is always
`.' in the literals in the C source code. In this sense, even from the
perspective of POSIX/C, there is no reason to adopt the output format
of `printf' for the floating-point numbers in arithmetic expressions
and the results of their evaluation.

--
Koichi

Re: REQUEST - bash floating point math support

[Koichi Murase]

2024年6月5日(水) 21:41 Zachary Santer :
> Bash could potentially detect floating point literals within
> arithmetic expansions and adjust the operations to use floating point
> math in that case. [...]

ksh and zsh are already behaving in that way, and if Bash would
support the floating-point arithmetic, Bash should follow their
behavior.

Actually, there's already a stub for the double-precision
floating-point number for shell-variable values in variables.h [1],
though it seems to be there for more than 20 years since 2.05b.

[1] 
https://git.savannah.gnu.org/cgit/bash.git/tree/variables.h?h=devel&id=dbb48b978671394bcb33c9f34656a9aadf40a318#n75

> For completeness, a floating point variable attribute could
> potentially be added. 'declare -i var' will cause var to be treated as
> an integer, though var can be referenced within an arithmetic
> expansion without this attribute. declare -f and -r (real, anybody?)
> are already taken for other things, so I'm not sure what the natural
> choice of option would be.

ksh uses -E for the double-precision floating-point numbers, and `-E'
is not used by Bash.

By the way, people are writing shell libraries to perform the
floating-point arithmetic in pure Bash, though I've never tried them
and am unsure about their quality, performance, and API experience.

https://github.com/clarity20/shellmath
https://github.com/m1ndflay3r/bash-float

We can also do the fixed-point arithmetic more easily and efficiently
(except for special functions such as sin, cos, tan, exp, log, etc.).
These are examples for specific purposes without needing precisions
(and thus not all the operations are implemented):

https://github.com/aneeshdurg/bash-raytracer/blob/main/math.sh
https://github.com/akinomyoga/blesh-contrib/blob/master/colorglass.bash#L247-L381

--
Koichi

Re: [PATCH] fix uneccesary copy of function bodies for BASH_SOURCE

[Koichi Murase]

2024年6月4日(火) 15:11 konsolebox :
> I haven't looked at this but it will keep lazy functions safe to
> implement right?

Right. The fix shouldn't change the observable behavior.

The functions are saved in two hash tables, `shell_functions' and
`shell_function_defs'. The function bodies are supposed to be put in
the former hash table, and the metadata (the source-file location and
the line where it is defined) are supposed to be put in the latter
hash table. However, we reuse the data structure for the function
declaration (containing both bodies and metadata) to keep the
metadata, and the function bodies were also contained in the latter
hash table unintentionally. We've been saving the data that has never
been used.

--
Koichi

Re: [PATCH] fix uneccesary copy of function bodies for BASH_SOURCE

[Koichi Murase]

To add some contexts not in the commit message, I knew that recent
versions of Bash consume more memory when the same shell script is
loaded.  After bisecting, I realized that the increase in memory use
was actually caused by a single commit ba4ab055 (see the attached
image.  The vertical axis is the increase of RSS measured by comparing
the results of `ps -o rss "$$"' before and after `source 

[PATCH] fix uneccesary copy of function bodies for BASH_SOURCE

[Koichi Murase]

The function `bind_function_def' (variables.c) stores the function
metadata in the hash map `shell_function_defs' in variables.c.  The
saved information is used by BASH_SOURCE (execute_cmd.c) and the
output of `declare -F func' with `shopt -s extdebug'
(builtin/declare.def) through the interface `find_function_def'
(variables.c).  The function body (i.e., funcdef->command) saved in
the hash map is not used anywhere and does not need to be saved in the
hash map.  In fact, the function body is not saved in the first call
of `bind_function_def' for the function name.

The problem is that the function body is saved for the second and
later calls of `bind_function_def' for the same function name.  When a
function is defined, `bind_function_def' is always called twice in its
parsing phase (`make_function_def' in make_cmd.c) and its defining
phase (`execute_intern_function' in execute_cmd.c).  As a result, the
function body is always saved in the hash map `shell_function_defs'
after the second call.  This doubles the memory used for shell
functions because the function bodies are already intended to be saved
in a separate hash map `shell_functions' (variables.c).

This patch fixes `bind_function_def' so that it skips the function
body also for the second and later calls.  This reduces the memory
uses of Bash for shell functions by almost half.

---
 variables.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/variables.c b/variables.c
index 84b30d93..0e785742 100644
--- a/variables.c
+++ b/variables.c
@@ -3507,7 +3507,11 @@ bind_function_def (const char *name, FUNCTION_DEF 
*value, int flags)
   if (entry && (flags & 1))
 {
   dispose_function_def_contents (entry);
+
+  cmd = value->command;
+  value->command = 0;
   entry = copy_function_def_contents (value, entry);
+  value->command = cmd;
 }
   else if (entry)
 return;
-- 
2.45.0

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月25日(土) 8:45 Lawrence Velázquez :
> On Fri, May 24, 2024, at 9:04 AM, Koichi Murase wrote:
> > 2024年5月24日(金) 19:18 Martin D Kealey :
> >> On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:
> >>> However, I personally do not think the FPATH mechanism is useful
> >>> because a file can only contain one function per file. Significantly
> >>> non-trivial functions are usually implemented by a set of helper
> >>> functions or sub-functions.
> >>
> >> Defining extra (private) functions in a file loaded from FPATH does no 
> >> harm,
> >> as long as its name doesn't conflict.
> >
> > [...] and registers it through `autoload -U func', it would result in a big 
> > function
> > containing function definitions:
> >
> >   func() { internal1() { ...; }; internal2() { ...; }; func() { ...; }; }
> >
> > This is not an expected one.
>
> Moving the extra "func" call into the definition itself takes that
> off the caller's hands [...]
>
> % cat /tmp/func
> internal1() { echo internal1 }
> internal2() { echo internal2 }
> func() { internal1; internal2 }
> func

I thought we couldn't include multiple public functions and create
symbolic links that way.

However, I now tried $funcstack, and it seems to be working:

  internal1() { echo i1; }
  internal2() { echo i2; }
  public1() { printf p1; (($#)) && printf '<%s>' "$@"; echo; internal1; }
  public2() { printf p2; (($#)) && printf '<%s>' "$@"; echo;
internal1; internal2; }
  $funcstack[1] "$@"

> while maintaining the ability to use "func"
> as a standalone script (the motivation for the zsh style).

I see, I was wondering about the motivation.

--
Koichi

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月24日(金) 19:18 Martin D Kealey :
> On Tue, 21 May 2024 at 23:16, Koichi Murase  wrote:
>> 2024年5月21日(火) 14:56 Phi Debian :
>> > 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
>> > know if that would be good enough for the initial purpose.
>>
>> There are already shell-function implementations at
>> /examples/functions/autoload* in the Bash source. They reference FPATH
>> to load functions, though one needs to call `autoload' for each
>> function in advance (by e.g. `autoload "$fpath_element"/*' ).
>
> My solution to this was to call 'autoload --all', which would gather all the
> filenames in FPATH (*1) and create autoload stubs for them.

Thank you for your comments.

These were just a comment on the implementations /example/functions/autoload*,
but yeah, I haven't noticed that `autoload.v4' actually supports the equivalent
option `autoload -f'.  If you would define your own `autoload', of course, you
can define the useful feature.

I think there is still a minor (and maybe negligible?) issue when a new path is
added to FPATH after performing `autoload --all'.  One would need to
additionally call `autoload "$fpath_added_element"/*' (or one could run
`autoload --all' again, but this would scan all the directories that were
already processed).

> Alternatively one could define a commandnotfound function to defer this until
> actually needed.

I thought about it before, but `command_not_found_handle' is executed in a
subshell, so the loaded functions will not be reflected in the original shell
process.

> (*1 I actually used a different variable name, since I wasn't providing
> exactly the same semantics as ksh, but that's relatively cosmetic)

I intended to talk about the specific semantics of ksh93/zsh's FPATH, but if we
extend the discussion to different variable names and different semantics, I
actually have my own `autoload' [1] referring the variable
`bleopt_import_path', where the function names are specified to `autoload' with
an explicit module name defining them.  An example use is found in
e.g. Ref. [2], although the module is specified by an absolute path in this
case.

[1] 
https://github.com/akinomyoga/ble.sh/blob/b72d78a97e6867bbf43de6a645dd6e387525443f/src/util.sh#L5071
[2] 
https://github.com/akinomyoga/ble.sh/blob/b72d78a97e6867bbf43de6a645dd6e387525443f/lib/core-syntax-def.sh#L62-L79

>> However, I personally do not think the FPATH mechanism is useful
>> because a file can only contain one function per file. Significantly
>> non-trivial functions are usually implemented by a set of helper
>> functions or sub-functions.
>
> Defining extra (private) functions in a file loaded from FPATH does no harm,
> as long as its name doesn't conflict.

I was assuming the Zsh's implementation of $FPATH.  I thought the Zsh
implementation of $FPATH doesn't allow it, but I now learned that ksh's
implementation of $FPATH is different.  I'm not a user of Zsh, so maybe I miss
something, but Zsh's default autoload assumes that each file contains the
*body* of the function (instead of the function declaration).  So if one puts a
set of functions in a file `func':

  internal1() { ...; }; internal2() { ...; }; func() { ...; }

and registers it through `autoload -U func', it would result in a big function
containing function definitions:

  func() { internal1() { ...; }; internal2() { ...; }; func() { ...; }; }

This is not an expected one.  However, TIL one can put function definitions in
ksh's style, and ksh's style can be used also in Zsh with `autoload -Uk func'.

>> Also, in libraries, we usually have a set
>> of functions that are closely related to one another and share the
>> implementations. I don't think it is practical to split those
>> functions into dozens or hundreds of files.
>
> I would hesitate to call what I've done "a manager", but my approach has been
> to allow a file to "declare" all the public functions it defines, and then
> simply have symlinks (or even hard links) to a single underlying file.

OK, this should work as ksh's FPATH implementation allows including function
definitions.

In that sense, this is just my personal preference, but I prefer to prepare a
`header' script file (such as [2]) that contains `autoload' declarations and is
intended to be sourced on the initialization stage instead of creating as many
symbolic links as the public functions.

--
Koichi

Re: printf -u "$fd"?

[Koichi Murase]

2024年5月22日(水) 0:15 Zachary Santer :
> Additionally, I was hoping the discussion of having a way to make fds
> not CLOEXEC without a loadable builtin[3][4] would get some more
> attention.

I haven't followed the discussion in the original thread for multiple
coprocs, but you can remove CLOEXEC relatively easily [1]:

  # Requires Bash >= 4.0
  function remove-cloexec {
local fd=$1
if ((fd!=0)); then
  builtin eval -- "exec 0<&$fd $fd<&- $fd<&0" &$fd $fd>&- $fd>&1" >/dev/null
fi
  }

This uses Bash's behavior of removing CLOEXEC for standard streams
0,1,2. (This behavior is probably undocumented, but I haven't checked
the document carefully). Adding CLOEXEC is more complicated and
relying on some assumptions.

[1] 
https://github.com/akinomyoga/ble.sh/blob/59787ee5528a8489e78ba09f712ed4b1f5a8e62f/src/util.sh#L3208-L3215
[2] 
https://github.com/akinomyoga/ble.sh/blob/59787ee5528a8489e78ba09f712ed4b1f5a8e62f/src/util.sh#L3114-L3207

--
Koichi

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月21日(火) 14:56 Phi Debian :
> Apparently konsolebox wrote a package manager, and survived the source 'as
> is', yet he still advocate for 'enhancement'. I do have a package manager
> too, and survived source 'as is' and don't require any enhancement.

I'm not sure if it is the case for konsolebox, but I think it is valid
to suggest including the features that turned out to be useful in
maintaining a package manager as built-in ones. Those suggested
features are meant to replicate the useful part of the features of the
package manager, so it is not surprising that the suggested features
are not needed for the package manager itself.

That said, I think it would be difficult to reach an agreement on the
detailed behavior of the suggested features to make them consistent
with many existing package managers.

> 'May be' bash could investigate the ksh93/zsh $FPATH autoload, but don't
> know if that would be good enough for the initial purpose.

There are already shell-function implementations at
/examples/functions/autoload* in the Bash source. They reference FPATH
to load functions, though one needs to call `autoload' for each
function in advance (by e.g. `autoload "$fpath_element"/*' ).

However, I personally do not think the FPATH mechanism is useful
because a file can only contain one function per file. Significantly
non-trivial functions are usually implemented by a set of helper
functions or sub-functions. Also, in libraries, we usually have a set
of functions that are closely related to one another and share the
implementations. I don't think it is practical to split those
functions into dozens or hundreds of files. It would also be slow to
read many different files, which requires access to random positions
on the disk.

--
Koichi

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月17日(金) 4:36 Greg Wooledge :
> On Thu, May 16, 2024 at 11:31:55AM -0400, Chet Ramey wrote:
> > On 5/15/24 11:31 AM, Koichi Murase wrote:
> > > Maybe it was not clear to use `source name' and `source -i name' to
> > > describe the idea. I meant I assumed the search domain being
> > >
> > > * BASH_SOURCE_PATH + PATH + PWD without the option
> >
> > It seems to me that this isn't useful.

I tried to convince myself about the behavior that the existence of
BASH_SOURCE_PATH disables PATH and PWD. However, after rethinking, I
still think there is a compatibility issue with it. I still think the
search domain of the source builtin being

* BASH_SOURCE_PATH + PATH + PWD without the option
* BASH_SOURCE_PATH with the option

> > The whole reason to have and use
> > BASH_SOURCE_PATH is not to use PATH; a fallback to PATH if something
> > isn't found in BASH_SOURCE_PATH doesn't allow that.

For this point, I'm convinced that only providing "BASH_SOURCE_PATH +
PATH + PWD" wouldn't satisfy the whole reason for adding
BASH_SOURCE_PATH. Maybe we need the option.

> > And if you're using
> > BASH_SOURCE_PATH, you have to set it, and if you want `.' in there,
> > add it.
>
> Yes, I'm inclined to agree with this.  If you want it to fall back to
> $PATH, you can append the contents of $PATH to BASH_SOURCE_PATH as well.
> You get the most control if BASH_SOURCE_PATH is used exclusively when
> it's set.

The problem is that this changes the global behavior of the `source'
builtin, and setting BASH_SOURCE_PATH breaks other libraries that
assume PATH.

On Thu, May 16, 2024 at 11:31:55AM -0400, Chet Ramey wrote:
> (from the user's perspective;
> I'm not sure what I think about a loaded `library' changing the global
> user settings by setting it),

If the loaded "library" is a single-file library, there is no reason
for the library to set BASH_SOURCE_PATH. However, the original
motivation for adding BASH_SOURCE_PATH is to provide a mechanism to
specify the directory of a library that consists of many files, where
only the files necessary for the current purpose are loaded. Maybe I
should call it such a library as "a set of libraries" or "a library
framework". Such a library framework for Bash wants to add its own
path to BASH_SOURCE_PATH.

However, another library framework can similarly set PATH to achieve a
similar effect (despite the problem of the name conflicts in the mixed
namespace with the executable files). Such a library framework can be
an old library framework designed for Bash < 5.3. Or it can be a
library framework designed for POSIX sh.

If setting BASH_SOURCE_PATH disables PATH and PWD (on which a library
framework for POSIX sh would rely) in the search for the conventional
use of the `source' builtin, it means that we cannot use the two types
of the library framework in a session or a program: i.e., library
frameworks utilizing BASH_SOURCE_PATH vs library frameworks for POSIX
sh. To allow the existence of both types of library frameworks,

a. The search can fall back to PATH and PWD when the path is not found
in BASH_SOURCE_PATH. When the library framework for Bash wants to
restrict the search, the library can use the option as `source -i'
.etc (the letter `i' can be a different one). The POSIX-sh library
framework can continue to use the plain `source filename' without the
option.

b. Another option would be to require everyone to include `PATH'
and `.' in BASH_SOURCE_PATH. However, if we require that, "the whole
reason to add BASH_SOURCE_PATH" is again lost. In addition, a later
change of PATH wouldn't be reflected in BASH_SOURCE_PATH that way.

c. Even another option is to require people not to globally set
BASH_SOURCE_PATH and only use it through `BASH_SOURCE_PATH=
source filename'. However, I anticipate that if we add this feature,
library frameworks or distribution bashrcs would start to set
BASH_SOURCE_PATH, which would break library frameworks assuming PATH.

d. Is there any better solution under the design that BASH_SOURCE_PATH
disables PATH and PWD even without the option?

I now changed my mind so that the `source' builtin should have a
separate way to restrict the search in BASH_SOURCE_PATH. While
`source' without the option continues to consider also PATH and PWD,
`source' with the option can disable PATH and PWD.

If BASH_SOURCE_PATH is required to disable PATH and PWD in the
conventional uses of the `source' builtin (in the form `source
filename' without the option), maybe I'm opposed to adding
BASH_SOURCE_PATH at all.

--
Koichi

Re: [PATCH v2 6/8] builtins/source: use source path in isolated mode

[Koichi Murase]

2024年5月15日(水) 22:28 Chet Ramey :
> On 5/14/24 5:26 PM, Koichi Murase wrote:
> > I see. I was assuming that users who want to completely turn off the
> > new feature would set an empty BASH_SOURCE_PATH (for both cases
> > `source name' and `source -i name').
>
> If you don't want to use BASH_SOURCE_PATH, don't set BASH_SOURCE_PATH,
> and unset it explicitly if you want to be sure.
>
> > `BASH_SOURCE_PATH=' could just mean
> > the current working directory skipping steps 1 and 2.
>
> That is consistent with other PATH-like variables.

I now agree.

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月15日(水) 22:21 Chet Ramey :
> On 5/14/24 7:15 PM, Koichi Murase wrote:
> > In case we talk about different designs, I assume the following one:
> >
> > * `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
> > current working directory.
>
> No, I'm proposing that this behavior be the default, without the option:

Yes, that's my intent. I mean "without the option" by the above `source name'.

> > * `source -i name', if it would be added, searches the files in
> > BASH_SOURCE_PATH (excluding PATH and the current working directory
> > unless `.' is included in BASH_SOURCE_PATH).

Maybe it was not clear to use `source name' and `source -i name' to
describe the idea. I meant I assumed the search domain being

* BASH_SOURCE_PATH + PATH + PWD without the option
* BASH_SOURCE_PATH with the option (if the option is supported)

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月15日(水) 22:13 Chet Ramey :
> On 5/14/24 4:48 PM, Koichi Murase wrote:
> > However, at the same time, I think the new option could be justified
> > from the viewpoint of language design. If we have a new namespace
> > ``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
> > to have an option to specify the namespace exclusively.
>
> Should these be the default semantics for BASH_SOURCE_PATH?

Even if the option is not supported, I think the default semantics
should still reference PATH and the current working directory as
fallbacks because some libraries may want to set BASH_SOURCE_PATH,
while some other libraries (such as an old library for <= 5.2 or a
POSIX-sh library) would still assume the current behavior of the
`source' builtin searching PATH and the current working directory. If
setting BASH_SOURCE_PATH would disable PATH and the current working
directory, those libraries cannot be mixed, which seems a troublesome
restriction.

> After all, if
> you want to search the current directory, put "." into the value.

Yes, that's technically correct. However, that requires every user or
framework to set BASH_SOURCE_PATH properly as

  BASH_SOURCE_PATH=xxx:${BASH_SOURCE_PATH:-$PATH:.}

(unless without a specific reason). Otherwise, the setup can be easily
broken by a not-so-well-designed.library in a dependency.

> > `unset -v var' is not required.
>
> That was a POSIX invention as a compromise between implementations.

Hmm. Then, is the example of `unset' irrelevant?

Re: proposed BASH_SOURCE_PATH

[Koichi Murase]

2024年5月14日(火) 15:09 Martin D Kealey :
> A common pattern is to unpack a script with its associated library & config
> files into a new directory, which then leaves a problem locating the
> library files whose paths are only known relative to $0 (or
> ${BASH_SOURCE[0]}).
>
> 1. I therefore propose that where a relative path appears in
> BASH_SOURCE_PATH, it should be taken as relative to the directory
> containing $0 (after resolving symlinks), rather than relative to $PWD.

After thinking about it again, I'm wondering what would be the assumed
use case of proposal 1 for the relative paths in BASH_SOURCE_PATH. I
think a way to simplify

  source "$__dir__/xxx/yyy/libzzz.bash"

(where $__dir__ is resolved to be the location of the current script
file) would be useful. However, I think what is generally achieved by
proposal 1 would be

  source "$__dir__//xxx/yyy/libzzz.bash"

This might be useful when the middle path elements of the library
location are ambiguous, yet the candidates are common with different
$__dir__. However, I don't have an idea about whether this has a
significant demand. What would be the use case? I naively think only
`.' is normally useful for the suggested interpretation of relative
paths.

Re: proposed BASH_SOURCE_PATH

[Koichi Murase]

2024年5月15日(水) 13:13 Peng Yu :
> On Tue, May 14, 2024 at 3:35 AM Koichi Murase  wrote:
> > I totally agree that we should have a way to specify the files in
> > subdirectories by the path relative to the current script file.
>
> I currently use something like this to source script files relatively.
>
> declare -g -A __sourced__files__
> __sourced__files__[$BASH_SOURCE]=$(realpath "$BASH_SOURCE")
> # This is to ensure to the relative path is to the actual bash file
> instead of a symlink to the bash file.
> source 
> "${__sourced__files__[$BASH_SOURCE]%/*}/dirname1/dirname2/${BASH_SOURCE##*/}"
> # My convention is that the file sourced in the subdirectory has the
> same filename as the parent file.
> # The difference between them are expressed in the directory path,
> i.e., dirname1/dirname2

Yeah, as far as we resolve the paths by ourselves and specify the
absolute path to the `source' builtin, it would reliably work. Even
BASH_SOURCE_PATH is unnecessary.

By the way, I got an inspiration from your example. You can define
name references to automate the resolution of the directory. With the
following definitions somewhere in a framework:

  declare -gA _Impl_file _Impl_dir _Impl_empty
  declare -gn 
__file__='_Impl_file[${_Impl_empty[x${_Impl_file[${BASH_SOURCE-$0}]=$(realpath
"${BASH_SOURCE-$0}")}]-}${BASH_SOURCE-$0}]'
  declare -gn 
__dir__='_Impl_dir[${_Impl_empty[x${_Impl_dir[${BASH_SOURCE-$0}]=$(dirname
"$__file__")}]-}${BASH_SOURCE-$0}]'

you can simply call

  source "$__dir__/xxx/yyy/libzzz.bash"

in each file intended to be sourced. The value of __dir__ always
reflects the location of the current script file. The names of the
variables can be adjusted to suit your taste.

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

In case we talk about different designs, I assume the following one:

* `source name' searches the files in BASH_SOURCE_PATH, PATH, and the
current working directory.
* `source -i name', if it would be added, searches the files in
BASH_SOURCE_PATH (excluding PATH and the current working directory
unless `.' is included in BASH_SOURCE_PATH).

I think it is clearer to see the patches. I attach modified patches:
0001 adds BASH_SOURCE_PATH. 0002 is an optional patch if we are to add
the command-line option to the `source' builtin.

In rebasing the patches, I noticed the stub in the latest devel push.
Should BASH_SOURCE_PATH be affected by `shopt -s sourcepath'? I think
it is more useful to be able to control PATH (through sourcepath) and
BASH_SOURCE_PATH (through whether its state is unset or empty)
separately.
From c29cf5ec2a79ef40eb09dcec3ae7402142850838 Mon Sep 17 00:00:00 2001
From: Matheus Afonso Martins Moreira 
Date: Mon, 13 May 2024 07:37:26 -0300
Subject: [PATCH 1/2] feat: support BASH_SOURCE_PATH

* shell: restrict BASH_SOURCE_PATH when appropriate

Make the BASH_SOURCE_PATH variable read-only and unsettable
when the shell is operating in restricted mode. This variable
should be restricted for the same reasons why PATH is restricted.

Modifications:

* When we are not in the POSIX mode, we consider BASH_SOURCE_PATH if
  present even without the new option `-i'.

* The new variable BASH_SOURCE_PATH has been disabled at all in the
  restricted shell since this would introduce a new loophole in
  existing setups of the restricted shells.  If the new variable were
  enabled in the restricted shells, all the existing setups would need
  to be updated to care about the variable BASH_SOURCE_PATH, which
  would be an unreasonable requirement.  In addition, considering the
  purpose of the restricted shells, we would not need to provide the
  new feature to the restricted shells.

Co-authored-by: Koichi Murase 
---
 builtins/source.def | 21 -
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/builtins/source.def b/builtins/source.def
index 51dddec6..4498ac9e 100644
--- a/builtins/source.def
+++ b/builtins/source.def
@@ -116,7 +116,7 @@ int
 source_builtin (WORD_LIST *list)
 {
   int result;
-  char *filename, *debug_trap, *x;
+  char *filename, *source_path, *debug_trap, *x;
 
   if (no_options (list))
 return (EX_USAGE);
@@ -143,20 +143,15 @@ source_builtin (WORD_LIST *list)
 filename = savestring (list->word->word);
   else if (absolute_pathname (list->word->word))
 filename = savestring (list->word->word);
-  else if (source_uses_path)
-{
-#if 0
-  char *spath;
+  else if (
+!posixly_correct &&
 #if defined (RESTRICTED_SHELL)
-  if (restricted == 0 && posixly_correct == 0 && (spath = path_value ("BASH_SOURCE_PATH", 1)))
-#else
-  if (posixly_correct == 0 && (spath = path_value ("BASH_SOURCE_PATH", 1)))
-#endif
-	filename = find_in_path (list->word->word, spath, FS_READABLE);
-  else
+!restricted &&
 #endif
-	filename = find_path_file (list->word->word);
-}
+(source_path = path_value ("BASH_SOURCE_PATH", 1)) != NULL)
+filename = find_in_path (list->word->word, source_path, FS_READABLE);
+  if (filename == 0 && source_uses_path)
+filename = find_path_file (list->word->word);
   if (filename == 0)
 {
   if (source_searches_cwd == 0)
-- 
2.43.0

From 172218c620adbbeca7a1dfda8eae759d38fecbc2 Mon Sep 17 00:00:00 2001
From: Matheus Afonso Martins Moreira 
Date: Mon, 13 May 2024 07:37:23 -0300
Subject: [PATCH 2/2] feat: support "source -i"

* builtins/source: parse the -i option

Passing the -i option to the source builtin
enables isolated sourcing mode which restricts
its search path to the directories defined by
the BASH_SOURCE_PATH variable. This also has
the added benefit of not touching PATH at all.

Modifications:

* If the option is explicitly supplied, we try to use BASH_SOURCE_PATH
  exclusively even in POSIX mode.  This is consistent with portable
  scripts because they are supposed not to specify the -i option.  It
  has been Bash's convention to allow a non-POSIX feature even in
  POSIX mode when the script explicitly requests it.

Co-authored-by: Koichi Murase 
---
 builtins/source.def | 62 ++---
 1 file changed, 42 insertions(+), 20 deletions(-)

diff --git a/builtins/source.def b/builtins/source.def
index 4498ac9e..251f6c02 100644
--- a/builtins/source.def
+++ b/builtins/source.def
@@ -117,9 +117,31 @@ source_builtin (WORD_LIST *list)
 {
   int result;
   char *filename, *source_path, *debug_trap, *x;
+  int opt, isolated_mode = 0;
+
+  reset_internal_getopt ();
+
+  while ((opt = internal_getopt (list, "i")) != -1)
+{
+  switch (opt)
+{
+case '

Re: Re: [PATCH v2 1/8] findcmd: parameterize path variable in functions

[Koichi Murase]

2024年5月14日(火) 12:32 Matheus Afonso Martins Moreira :
> > The patches touch the interface of many functions
>
> I added one external function: find_in_path_var.
> The other changes are all static and local.
>
> > It seems essentially equivalent to just calling
> > find_in_path (list->word->word, path_value ("BASH_SOURCE_PATH", 1), 
> > FS_READABLE);
>
> It is. I'm just very averse to chaining functions like that in C
> since you can't write f(g(x)) without worrying about freeing
> the value returned by g.

If you are talking about the current specific case of path_value, as
Chet has replied, path_value doesn't return a newly allocated block.
It just returns a pointer to an existing variable cell owned and
managed by the global variable context whose root is static. It
doesn't return or move an ownership of the pointer.

If you are talking about just the coding style in general, it seems
like just your personal style. In addition, the patch just
encapsulates it in another function h(x) := f(g(x)) (as done in
`_find_user_command_internal()'), and the logical structure is the
same. One could argue that `_find_user_command_internal()' is using a
separate form `var = g(x); return f(var);', but then one can also use
the separate form at the caller's side. Whether to use the combined
form or the separate form seems to be orthogonal to whether we do it
inline or in a new function.

> > The cases where path_value() returns NULL or an empty string would be
> > anyway handled by find_in_path()
>
> With find_in_path_var there's no need to even consider that.

Yes, but even without `find_in_path_var', there's no need to consider
that because it is handled elsewhere.

> > I doubt it's worth adding those changes in
> > `findcmd.c'. So are the changes in [PATCH v2 2/8].
>
> You don't think they improve the code?

I don't think so. There are always pros and cons, and the choice would
depend on many factors including the maintenance cost and the
preference of the maintainer. The inter-modular interface should be
minimized to reduce the coupling unless the module is very stable and
immutable. You might think preparing as many "buttons" as possible
would be a good interface, but any additional interface becomes a
constraint or a boundary condition to the module and adds a
maintenance cost for future changes. The final choice of how the code
is maintained would be determined by the maintainer; it is similar to
the coding style. If one observes the Bash codebase, one notices that
it is not maintained in your style.

Re: [PATCH v2 6/8] builtins/source: use source path in isolated mode

[Koichi Murase]

2024年5月15日(水) 5:16 Chet Ramey :
> On 5/13/24 10:53 PM, Matheus Afonso Martins Moreira wrote:
> >> I think we should restrict the search in
> >> BASH_SOURCE_PATH and shouldn't fall back to local files.
> >
> > I concur but thought it'd be wise to discuss it first so I left it in.
>
> Searching the current directory is the existing behavior (albeit when using
> PATH) when not in posix mode.

I agree that we should fall back to the files in the current working
directory for `source name'. However, in the above, I was talking
about the behavior of the suggested option to turn off the fallback.

Re: [PATCH v2 6/8] builtins/source: use source path in isolated mode

[Koichi Murase]

2024年5月15日(水) 5:02 Chet Ramey :
> > In that case,
> > we need to make sure that `find_in_path_var()' returns NULL in the
> > case of empty BASH_SOURCE_PATH.
>
> Do you mean BASH_SOURCE_PATH= ?

Yes. Or more precisely, I intended to include both `BASH_SOURCE_PATH='
and `unset -v BASH_SOURCE_PATH'.

> That's the standard behavior of PATH-like
> variables: a null element anywhere is equivalent to `.'. Does this
> require a special case?

I see. I was assuming that users who want to completely turn off the
new feature would set an empty BASH_SOURCE_PATH (for both cases
`source name' and `source -i name'). Suppose we consider, for `source
name', 1) BASH_SOURCE_PATH, 2) PATH, and 3) the current directory in
order (which is actually different from the submitted patch). If we
are to use `find_in_path_var' or `find_in_path' for step 1, we can
return NULL to fall back to step 2 (or otherwise step 2 would be
skipped with a naive implementation).

However, thinking about it again, maybe `unset -v BASH_SOURCE_PATH'
would mean skipping step 1, and `BASH_SOURCE_PATH=' could just mean
the current working directory skipping steps 1 and 2.

Re: [PATCH v2 8/8] shell: restrict BASH_SOURCE_PATH when appropriate

[Koichi Murase]

2024年5月15日(水) 4:55 Chet Ramey :
> On 5/13/24 6:37 AM, Matheus Afonso Martins Moreira wrote:
> > Make the BASH_SOURCE_PATH variable read-only and unsettable
> > when the shell is operating in restricted mode. This variable
> > should be restricted for the same reasons why PATH is restricted.
>
> Or you could simply ignore BASH_SOURCE_PATH in restricted mode.

As I've already replied to the previous patch, I think we should
ignore BASH_SOURCE_PATH in the restricted mode. This will just become
a loophole if it's enabled. (Or, at least, this should be an opt-in
feature that the scripts to set up the restricted shells need to
explicitly turn on instead of being automatically enabled through the
existence of a variable BASH_SOURCE_PATH).  If it is automatically
turned on in the restricted shells, it implies that we change the list
of the items that the scripts to set up restricted shells need to care
about. All existing setups for the restricted shells would be spoiled.

Re: [PATCH v2 5/8] builtins/source: parse the -i option

[Koichi Murase]

2024年5月15日(水) 4:51 Chet Ramey :
> What do folks think about forcing an option to enable using
> BASH_SOURCE_PATH? Should it be required? Is it necessary? (I personally
> think it is not.) We discussed this briefly last week but without any
> conclusion.

Either works for me (suppose the `source' builtin considers
BASH_SOURCE_PATH even without specifying the option in non-POSIX
mode). I personally think both decisions to include and not to include
the suggested option are understandable.

To talk about its necessity, I actually agree with the others. I don't
think we need it.

However, at the same time, I think the new option could be justified
from the viewpoint of language design. If we have a new namespace
``BASH_SOURCE_PATH'' for the scripts to be sourced, it would be neat
to have an option to specify the namespace exclusively. It is similar
to `unset -v'. As far as the shell script is properly designed so that
the target variable is ensured to exist, `unset var' is sufficient
because the `unset' builtin first checks the variable name and falls
back to the function name only when the variable doesn't exist. In
this sense, `unset -v var' is not required. It's not necessary, yet I
think it can be a part of a language.

--
Koichi

Re: proposed BASH_SOURCE_PATH

[Koichi Murase]

2024年5月14日(火) 15:09 Martin D Kealey :
> 1. I therefore propose that where a relative path appears in
> BASH_SOURCE_PATH, it should be taken as relative to the directory
> containing $0 (after resolving symlinks), rather than relative to $PWD.
>
> As an interim step until that's implemented, please ignore any relative
> entries in BASH_SOURCE_PATH, so that users who really want the cwd in
> BASH_SOURCE_PATH get used to writing $PWD or /proc/self/cwd instead.

I totally agree that we should have a way to specify the files in
subdirectories by the path relative to the current script file.

* One concern with treating relative paths relative to the current
script location is that the user might put `.' in BASH_SOURCE_PATH to
mean the current directory. I don't think it is a good practice, but I
anticipate that users expect that. For example, one can observe the
default values of PATH [1] and BASH_LOADABLES_PATH [2] in the Bash
codebase. It explicitly contains `.' to mean the current working
directory. It is very confusing that `.' in BASH_SOURCE_PATH points to
the script location ($(dirname ${BASH_SOURCE[0]})), while `.' in PATH
and BASH_LOADABLES_PATH points to the current working directory
($PWD).

[1] 
https://git.savannah.gnu.org/cgit/bash.git/tree/config-top.h?h=devel&id=b3d8c8a4e7c5417d986f93f646ea740cb13c08d7#n61
[2] 
https://git.savannah.gnu.org/cgit/bash.git/tree/config-top.h?h=devel&id=b3d8c8a4e7c5417d986f93f646ea740cb13c08d7#n78

One solution might be to treat `.' as a special case to mean $PWD, and
the other relative paths would mean the path relative to the script
location. However, the problem is that it is confusing that the base
location changes depending on its form. In particular, the difference
in the behavior between `.' and `..' would be very confusing.

Another solution would be to introduce a special name to mean the
current script location. For example, we could introduce a tilde
expansion to mean the current script location since the only expansion
processed in `find_path_in()' and related functions is the tilde
expansion. We already have ~+ and ~+N (equivalently, ~N) to mean
${DIRSTACK[N]}. Similarly, we can have e.g. ~@  and ~@N to mean
$(dirname ${BASH_SOURCE[N]}). This can also be useful outside
BASH_SOURCE_PATH since determining the directory name of a path isn't
just ${path%/*} and one needs to care about edge cases to properly do
that.

* Another point to discuss is whether we should resolve the symbolic
links. Maybe it's just fine to resolve the symbolic links fully (as
suggested), but another possibility can be to search all the
directories appearing in the intermediate steps of the link
resolution.

> 2. Search BASH_SOURCE_PATH when any relative path is given, not just a path
> that lacks a '/', so that libraries can be organized into subdirectories.

I agree. I actually thought about raising the discussion about this
after the current patch set is merged.

> 3. To avoid accidentally loading a script rather than a library, while
> searching BASH_SOURCE_PATH, ignore any files that have exec permission,
> inverting the check normally made for executables in PATH. This would keep
> executables and libraries functionally separate, even if they're commingled
> in the same directories.

I disagree with this. Since the original intent of having
BASH_SOURCE_PATH would be to have the list of directories that do not
contain files not intended to be sourced, I don't see a reason to
perform additional filtering. I generally think that applications
shouldn't give a new meaning of the executable permission of files for
a purpose different from the original one. They may conflict.

> Yes I know that some folk think it's occasionally useful to have a single
> file that operates as both, but (a) new features should default to the
> "safest" mode of operation, (b) this isn't python and so that's pretty rare
> in practice,

How did you conclude that it's rare? Phi mentioned it. I also have a
script that operates as both. Furthermore, it's actually a recommended
practice by the bpkg manager [3].

[3] https://github.com/bpkg/bpkg?tab=readme-ov-file#packaging-best-practices

For a script file with both usages, I think one wants to put symbolic
links to the file in PATH and BASH_SOURCE_PATH. In such a situation,
the suggested non-executable requirement conflicts with the
requirement for executable files. Do we prohibit the existence of such
a script? Or do we request to make an actual copy or a hard link to
the file (which I think is a bit harder to manage the consistency)? I
think these are unnecessary requirements.

> 4. When using "source -i", if BASH_SOURCE_PATH is unset or empty, it's
> taken as equivalent to '.', so that it's useful to write "source -i
> ../lib/foo.bash" in a script at "$X/bin/bar" to load "$X/lib/foo.bash".

I currently have no particular opinion on this. Maybe I need to think
about it later.

Re: [PATCH v2 1/8] findcmd: parameterize path variable in functions

[Koichi Murase]

2024年5月13日(月) 19:38 Matheus Afonso Martins Moreira
:
> The PATH variable is hard coded in the user command finder function.
> Transforming it into an argument allows reusing the file finding logic
> with variables other than PATH, making the code more flexible.

I think what was suggested by Chet is to confine the changes in
builtins/source.def. The patches touch the interface of many
functions, but it seems essentially equivalent to just calling

  find_in_path (list->word->word, path_value ("BASH_SOURCE_PATH", 1),
FS_READABLE);

The cases where path_value() returns NULL or an empty string would be
anyway handled by find_in_path() [more specifically by
_find_user_command_internal() and find_user_command_in_path() called
by find_in_path()]. I doubt it's worth adding those changes in
`findcmd.c'. So are the changes in [PATCH v2 2/8].

Re: [PATCH v2 3/8] builtins/source: extract file executor function

[Koichi Murase]

2024年5月13日(月) 19:39 Matheus Afonso Martins Moreira
:
> Extract into a dedicated helper function the code which loads
> the contents of a file and executes it in the current shell.
> This separates this useful functionality from the path resolution
> mechanism used by the source builtin.

To other reviewers: I checked that this commit is still independent of
the other parts, so you can skip this commit if you focus on the
suggested feature.

Re: [PATCH v2 6/8] builtins/source: use source path in isolated mode

[Koichi Murase]

2024年5月13日(月) 19:40 Matheus Afonso Martins Moreira
:
> diff --git a/builtins/source.def b/builtins/source.def
> index 26040e70..7f870bfe 100644
> --- a/builtins/source.def
> +++ b/builtins/source.def
> @@ -177,6 +178,27 @@ search_for_file (WORD_LIST *list)
> [...]
> +static char *
> +isolated_search_for_file (WORD_LIST *list)
> [...]
> +  if (filename == 0)
> +{
> +  if (source_searches_cwd == 0)
> +return (NULL);
> +  else
> +filename = savestring (list->word->word);
> +}

* With this, `isolated_search_for_file()' falls back to local files in
the current working directory. I'm unsure about whether this is
intentional, but I think we should restrict the search in
BASH_SOURCE_PATH and shouldn't fall back to local files. In that case,
we need to make sure that `find_in_path_var()' returns NULL in the
case of empty BASH_SOURCE_PATH. In the current implementation,
`find_in_path_var()' seems to return `save_string (name)' for empty
BASH_SOURCE_PATH, which would induce the search in the current working
directory.

* Even if we intensionally fall back to local files, this part seems
just a repetition of `search_for_file()'. I think this common part
should be moved to `builtin_source()'.

Re: [PATCH 9/9] variables: define default BASH_LIBRARIES_PATH

[Koichi Murase]

2024年5月8日(水) 9:13 Matheus Afonso Martins Moreira :
> Please let me know if you agree to any of the following paths:
>
> ~/.local/share/bash:/usr/share/bash
> ~/.local/share/bash/lib:/usr/share/bash/lib

The former is absolutely no. If we choose that location to store
scripts, no place is left for the future addition of other types of
files. The latter style (but with `lib' being replaced by another word
if necessary as in an ongoing discussion about the name `library')
should be fine.

Even if it would finally be `library', I suggest that the directory
name `lib' should instead be `library' or `libraries' because
inexperienced users can be easily confused with the dynamic library
location `lib' when a script framework instructs the users to put
their script files in `share/bash/lib'. Even if we explicitly instruct
the users to put them in `/usr/share/bash/lib', some users still try
to put them in `/usr/lib' for some mysterious reason. It would
probably happen when `/usr/share/bash/lib' doesn't work for another
reason, and users try to put them directly in `/usr/lib` to "solve"
the problem. That happened with my framework. I named the directory of
the script files as `lib', which I regret now.

> Note that the default value of XDG_DATA_HOME
> is defined to be ~/.local/share.

In this case, I think the default value should actually honor the
value of XDG_DATA_HOME if present (instead of hard-coding it to be
~/.local/share).

  BASH_SOURCE_PATH=${XDG_DATA_HOME:-$HOME/.local/share}/:...

Then, there arise the repeated arguments of the chicken or the egg of
the XDG base directory initialization. Since XDG_DATA_HOME is supposed
to be set by the user in .bash_profile (or similar initialization
file), the prefered value of XDG_DATA_HOME is not known at the
initialization stage of login sessions of Bash.

> I'm not exactly invested in getting any XDG stuff
> into bash but it seems like a reasonable enough
> default for the user's personal libraries directory.

As mentioned above, it's useful but there is an issue of the chicken
or the egg. The user sets values to the variables in the shell, but
the shell wants to know the value of the variables to initialize it on
its startup. Some alternative ways are proposed, but a clean way
hasn't yet appeared.

I think it would be better to just confine ourselves in a similar
default as BASH_LOADABLES_PATH within this patch. The choices for the
location in the home directory can be discussed with the approach to
the XDG base directories.

Re: [PATCH 4/9] bashgetopt: define long option shortener function

[Koichi Murase]

2024年5月8日(水) 8:11 Matheus Afonso Martins Moreira :
> Some builtins do support long options, even though it seems
> to not be documented.

I didn't know the `type' builtin recognizes those forms.

> Also, I do realize that there's a big all caps warning
> in the file saying that "THIS SHOULD REALLY GO AWAY".
> I get the message, but I'm honestly not sure what
> that's referring to. The obsolescent options,
> the long options, the hack to handle them?
> All of this stuff? Not sure.

> The builtins seem to be quite consistent
> in their support for --help though.
> So there's at least some precedent
> for long option support.

I've been accepting the `--help' option because it is somewhat
special. When people do not know even whether the command supports a
long form of the options, they still want to get a help text. Then,
they typically attempt ` --help'.

> > If you consider the current form would be complete, you should provide
> > the reason that the other builtins should not have the long form of
> > the options, while only the source builtin should have a long option.
>
> I don't understand why they don't all have long options to begin with.
> Is there a reason for it?

I don't know the true reason, but at least it's tradition. It's been
this way for a long time.

> > I don't think those changes are allowed to slip in as a bonus
> > of another change on the source builtin.
>
> Well I'm not really trying to "slip in" anything.
> That implies I'm deceptively trying to get my way.
> I'm not. I just thought it would be a nice feature to have.

I wouldn't call it `deceptive', but by mixing the change in a larger
patch, the change has more chance to go into the codebase without
being focused by the reviewers and without being discussed
sufficiently as they would have if they were in a separate suggestion.

> If it's not acceptable, I can and will remove the long options.

This is my opinion: You can just remove the support for the long
option from the present patch set. If you are willing, you should
create another discussion in the bug-bash list (not in the help-bash
list), and then if accepted, complete the long options for all the
builtins, and re-submit the patch after that. The current introduction
of the long option is not in a complete form.

> > I'm not sure about what Chet thinks, but I don't think those are
> > really necessary, though it's a better design in general.
>
> I'll wait for his review.

> > I also have thoughts on the way how this was implemented, but I
> > wouldn't discuss it for now.
>
> Please do. If the implementation can be improved, I will do it.

I'm not sure if the full support for the long options would be
accepted by the community so wouldn't like to spend time to nicely
explain them at present. If you are fine with random sentences, it
doesn't work with `source -- --library' when one wants to source a
file of the actual name `--library'. This is a hypothetical case, but
still conflicts with the standard processing of the command-line
arguments. The hidden options of the `type' builtin have the same
issue. Next, there is already the `--help' option. If the long option
would be officially supported, the processing of `--help' and the
other long options should be consistent. We don't want to use two
different mechanisms to handle the long options at the same time.
Third, the `const char to;' should be `const int to;' to make it
consistent with the return type of `internal_getopt'. In particular,
this is not forward compatible when we want to add a long option that
doesn't have a corresponding short form, in which case one should
choose an integer that doesn't overlap with a char value (as seen in
the negative values of GETOPT_EOF and GETOPT_HELP). However, I have to
say there is also an issue in the existing choices of -1 and -99
because `char' is signed in typical C implementations and thus has the
range -128..127. We don't usually use the negative range (i.e., C1 and
GR in ASCII or the leading byte of a multibyte character) as an option
letter, but it's better to avoid.

Re: Re: [PATCH 1/9] builtins: extract file content executor function

[Koichi Murase]

2024年5月8日(水) 6:13 Matheus Afonso Martins Moreira :
> > By exposing this function in `common.h',
> > this effectively becomes a part of the public
> > interface for loadable builtins.
>
> [...]
>
> Are all external functions defined in all files inside the
> builtins directory exposed as part of a public API/ABI?

No.

> If that's the case, then it should also be made static
> as you suggested.

Even if it's not the case, I think one should mark it as `static' as
far as the function is not used by other translation units. Even if
it's not the interface exposed to the users, it's still an interface
for other translation units, and it's a premature exposition of
abstraction at present. No other translation units are using the
function.

> Can you break down for me which parts of the code base
> are public and which are private? This will allow me to
> avoid making such mistakes in the future.

You can check /examples/loadables/loadables.h in the tree. The headers
included from there (and further headers included thereby) are
explicitly the interface that the users can use to implement a
loadable builtin. For a wider public interface, you can look inside
the directory `$prefix/include/bash' (and subdirectories therein)
after running `make install'.

> I'd also like to suggest creating a private commons.c
> file where potentially reusable functionality can be
> moved to without exporting them.

I think such a feature should go into a related file in the main directory.

Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月8日(水) 7:12 Matheus Afonso Martins Moreira :
> > a language that has no support for namespaces, awkward scoping rules,
> > a problematic implementation of name references, and so on.
>
> I could at least attempt to address those issues.

I don't think these are requests to you. Those topics (in particular
the scoping and namerefs) are the ones that repeatedly pop up in the
bug-bash list. Each single topic could be a project that takes years.

Also, if you start to implement those (and the `load once' feature), I
need to go back to the arguments of the standard module manager. There
are existing module managers implemented by shell functions, which are
based on years of practices. Then, you cannot ignore them.

> Namespacing could be as simple as prepending the module name
> to the symbols that are being created.

I don't think that works consistently. The shell language is
inherently dynamic, so it isn't possible to resolve everything
statically at the loading time. What happens when a function is called
by its name stored in a variable passed through the caller's
parameter? What happens when a variable is referenced by its name
similarly? The behavior is unpredictable.

Also, if you consider implementing a namespace, I think you should
first check the ksh's implementation of namespace and learn what would
be the good points and what could be the practical problem there.

Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月7日(火) 23:17 Chet Ramey :
> On 5/6/24 1:21 AM, Phi Debian wrote:
> > Well even this is unclear 'BASH_SOURCE_PATH get searched before PATH' or
> > 'BASH_SOURCE_PATH get searched instead of' or even 'BASH_SOURCE_PATH get
> > searched after PATH'
>
> Because we haven't discussed the semantics. There shouldn't be any changes
> in posix mode, so that will continue to use PATH.

As an alternative option, is it possible that the `.' builtin follows
POSIX while the `source' builtin has the extension of
BASH_SOURCE_PATH? I don't have a particular preference here, but I'm
just curious about if there is a reason or motivation that the
`source' builtin should also follow POSIX.

Re: Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月7日(火) 21:40 Martin D Kealey :
> I wonder if it would be useful to add options to 'command':

Thanks for your suggestion. I still think those should be implemented
by shell functions, but that interface sounds interesting. That seems
to cover most of the cases I raised. However, those are merely
examples I know, and I still think it would be better to leave the
degrees of freedom for module managers.

> * '-o' would report only the first command found (when more than one is 
> given) (or could be '-1');
> * '-p' would skip builtins &  functions, and fail silently if no file can be 
> found;
> * '-x' would search for files that lack exec permission.

Maybe I'll attempt a shell-function implementation of this interface.

Re: Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月7日(火) 2:11 Matheus Afonso Martins Moreira :
> >> A native way to source libraries. Built into bash, available to all users.
> > That's the source builtin.
>
> It looks for scripts in directories meant for executables.
> It prefers files with the executable bit set.

Is this related to [1]?

[1] https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00132.html

I've checked the behavior, where the `source' builtin seems to still
load the non-executable one found first in the path.

  $
  GNU bash, version 5.3.0(1)-alpha (x86_64-pc-linux-gnu)
  $ ls -l t/bin{1,2}/file
  -rw-r--r-- 1 murase murase 15 2024-05-08 11:27:33 t/bin1/file
  -rwxr-xr-x 1 murase murase 15 2024-05-08 11:27:25 t/bin2/file
  $ head t/bin{1,2}/file
  ==> t/bin1/file <==
  echo bin1/file

  ==> t/bin2/file <==
  echo bin2/file
  $ (PATH=t/bin1:t/bin2 source file)
  bin1/file

> It's a native way to load bash scripts for sure.
> But it's not a native way to load libraries.

How do you define `the libraries' out of a wider category of Bash
scripts intended to be sourced? Even if they are distinct from other
types of Bash scripts, they are still a subcategory of Bash scripts.
Then, the source builtin is also a native way to load libraries. Maybe
you mean a dedicated way by "the native way", but I don't see the
necessity of having a dedicated way.

> > It's just your expectation of the ergonomics but not ours.
>
> What are your expectations?

The answer is found later in the same paragraph as the above sentence.
I mean the community by `ours' but not specifically mine. There are
existing implementations of module managers, which you don't seem to
have yet performed a survey, and which are not just simple as yours.
Of course, existing implementations are not designed by
anti-ergonomics, and there should be ergonomics by the respective
designers.

> Do you really think it's reasonable to suggest that
> users implement their own library importing function?

No. I just think it's unreasonable to force everyone to use a specific
way of module management that you are trying to invent without
referencing existing implementations.

> The fact such a function can be written in bash
> is evidence of how powerful bash is. It doesn't
> really follow though that everything that can be
> written in bash should be done in bash.

Right, but the reserve is not true either. I don't think everything
that is significantly complicated to show the power of Bash should be
a builtin.

> > He submitted patch seems unneededly large.
>
> The diffstat shows 9 files changed,
> 203 insertions, 92 deletions.

The number of lines being 203 is definitely large.

> Most of those are very simple
> refactorings of existing code.

I now know the reason. There are additional refactorings.

> > incomplete addition of the long option of builtins
>
> What do you mean? What is missing?
>
> > an incomplete support for the XDG base directories, which should
> > actually be discussed separately.
>
> XDG does not specify the ~/.local and ~/.local/lib directories. [...]

I've replied to them in respective patches.

> >  It does seem your personal hangup not giving a filename extension to
> > the script files, which ended up with mixed script files to be sourced
> > and executable files.
>
> Whether to use file extensions or not is the user's choice.
> Bash should work correctly regardless of what the file name is.

Right. At the same time, more or less, it is the user's responsibility
to avoid name conflicts between a script file and an executable file
just like it is between two script files to be sourced. But I agree
that the situation of the mixed script files and executable files can
be improved.

> > If you think that is the problem, your suggestion doesn't actually
> > solve the problem of module managers.
>
> I didn't think it was a problem at first. I just thought it was unnecessary.
> Bash already knows how to do it. We should make it do it for us.
>
> Now I do see it as a problem to be solved. Bash has a restricted shell mode
> which, among many other things, prohibits slashes in file names passed to
> the source builtin. If things are as you claim and the module managers really
> do resolve the paths themselves in order to pass an absolute path to source,
> that means they are all incompatible with bash's restricted shells.
> The proposed native implementation of the module loader is not.

I haven't thought about the restricted shells.

After thinking about it, I think the suggested feature itself should
probably be disabled under the restricted shell since similar features
such as `enable -f' and `command -p' are also disabled in the
restricted shell. If it would be enabled, BASH_SOURCE_PATH should be
at least readonly the same as PATH. Also, administrators who prepare a
restricted shell environment need to carefully set BASH_SOURCE_PATH to
be a restricted one. However, it would still be a loophole for old
setups for the restricted shell that don't take account of
BASH_S

Re: [PATCH 9/9] variables: define default BASH_LIBRARIES_PATH

[Koichi Murase]

2024年5月5日(日) 18:57 Matheus Afonso Martins Moreira
:
> +/* The default value of the BASH_LIBRARIES_PATH variable
> +   which is used by source -l instead of PATH. */
> +#ifndef DEFAULT_LIBRARIES_PATH
> +#define DEFAULT_LIBRARIES_PATH \
> +  "~/.local/lib/bash:/usr/local/lib/bash:/usr/lib/bash"

I don't think `$prefix/lib' would be the appropriate place to put the
script files. /lib directory is traditionally intended for the
architecture-dependent binary libraries loaded by ld. All the other
resources independent of the architecture (and thus can be "shared")
can just be put in /share/bash/scripts or something.

Furthermore, the location overlaps with the loadable builtins, which
are not the script files intended to be sourced. You are mixing the
script file with the loadable builtins while trying to separate the
script files from the executable files. The same issue will arise in
another place.

https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00090.html
2024年5月7日(火) 2:11 Matheus Afonso Martins Moreira :
> > an incomplete support for the XDG base directories, which should
> > actually be discussed separately.
>
> XDG does not specify the ~/.local and ~/.local/lib directories.
> Only the XDG Data Directory defaults to ~/.local/share if unset.
> That implies a ~/.local directory tree that mirrors that of /usr.

This is the same as the argument for the long option. Why only
BASH_LIBRARIES_PATH (among the other similar variables) should honor
~/.local indirectly implied by XDG_DATA_HOME? In the current state of
Bash, any of the XDG user directories haven't been hard-coded. Also,
as you already know, the introduction of any XDG user directories is
one of the sensitive topics in the bug-bash list, so I believe it
should be discussed separately. It shouldn't slip into the codebase as
a part of another change.

Re: [PATCH 8/9] builtins/source: search libraries in library mode

[Koichi Murase]

2024年5月5日(日) 18:57 Matheus Afonso Martins Moreira
:
> -  filename = search_for_file (list);
> +  if (!library_mode)
> +filename = search_for_file (list);
> +  else
> +filename = search_for_library (list);

I expected that the source builtin without the flag would also
consider BASH_LIBRARIES_PATH if present, which is more similar to
Chet's suggestion in the POSIX mode [1]. More specifically, the source
builtin first searches for BASH_LIBRARIES_PATH if present, or
otherwise falls back to PATH and the local files. When the flag is
specified, the fallbacks are disabled.

[1] https://lists.gnu.org/archive/html/help-bash/2024-04/msg00022.html

Re: [PATCH 4/9] bashgetopt: define long option shortener function

[Koichi Murase]

2024年5月5日(日) 18:57 Matheus Afonso Martins Moreira
:
> Define the shorten_long_options helper function that converts

Let's talk about whether we should support the long form of the option
in this patch set.

2024年5月7日(火) 2:11 Matheus Afonso Martins Moreira :
> > incomplete addition of the long option of builtins
>
> What do you mean? What is missing?

It's simply inconsistent in the present form. Why should only the
source builtin have the long form of the option? No other builtins
have the long form of options, which is inconsistent. Suppose you
would submit an independent patch to introduce the long option of
builtins in Bash. Would you do that only for the source builtin? If
you consider the current form would be complete, you should provide
the reason that the other builtins should not have the long form of
the options, while only the source builtin should have a long option.

If one tries to give the long options to all the existing builtins,
that should be discussed separately. I don't think those changes are
allowed to slip in as a bonus of another change on the source builtin.
I'm not sure about what Chet thinks, but I don't think those are
really necessary, though it's a better design in general.

I also have thoughts on the way how this was implemented, but I
wouldn't discuss it for now.

Re: [PATCH 2/9] findcmd: parameterize path variable in functions

[Koichi Murase]

2024年5月5日(日) 18:56 Matheus Afonso Martins Moreira
:
> The PATH variable is hard coded in the user command finder function.
> Transforming it into an argument allows reusing the file finding logic
> with variables other than PATH, making the code more flexible.

There is already a very similar feature of the `command' builtin,
which enforces PATH to be a specific path in searching the command yet
doesn't replace the actual value of PATH for the executed command.
This is implemented by the flag `#define CMDSRCH_STDPATH 0x02'
(findcmd.h:28 @ master). For consistency, shouldn't the suggested
feature to change the search path for the script files be implemented
by a similar flag CMDSRCH_XXX? This is sufficient because the
suggested new parameter `path_var' only becomes PATH or
BASH_LIBRARIES_PATH for now. I wouldn't expect a YAGNI argument about
the future possibility.

Re: [PATCH 1/9] builtins: extract file content executor function

[Koichi Murase]

The patches don't apply to the devel branch. You've made patches on
top of the master branch, but the master branch of Bash is just a
release branch, where each commit corresponds to a release. You should
normally work based on the devel branch.

2024年5月5日(日) 18:56 Matheus Afonso Martins Moreira
:
> Extract into a dedicated helper function the code which loads
> [...]

This patch is independent of the present change for the source option.
This is unrelated.

Also, you've factored out one function from `source_builtin(...)' and
moved another function and it to common.c. However, the new function
`execute_file_contents' is still called from only one place of the
original location.

I think you will not be satisfied by just this comment, so I'd explain
it in more detail. By exposing this function in `common.h', this
effectively becomes a part of the public interface for loadable
builtins. However, this abstraction is not motivated by the actual use
cases. Without any actual uses, we cannot make sure this specific way
of abstraction (such as the set of parameters that the function should
receive) would be fine. There is a possibility that we want to later
add other parameters after seeing the actual uses. However, once it
becomes a part of the public interface, we wouldn't be able to modify
it later easily. Otherwise, a loadable builtin makes the process
crash. Then, you'll have to prepare `execute_file_contents_v2(...)' or
something when the actual use cases reveal a necessity of new
parameters or a change of the interface. I think this is a premature
abstraction and should be avoided.

At least the new function should still be defined in
`builtins/source.def' with the static keyword, but even that way, the
change is completely unrelated to the addition of the new flag to the
source builtin.

You might try to claim the usefulness or the validity of this change,
but I don't have any objection to its usefulness. It's not the problem
of usefulness.

> Signed-off-by: Matheus Afonso Martins Moreira 

To what you are signing off... By putting this line, you imply that
you agree on the rules that the project defines for copyright, etc.,
but Bash doesn't specify it as far as I know. Then, you've signed off
what you haven't even checked the existence. Instead, FSF requires the
copyright assignment to FSF (which is more strict than just e.g. DCO
or anything), which shouldn't be exempted just by "Signed-off-by".
Once the patches are accepted with a non-trivial amount of codes, Chet
will explain it to you.

Re: Re: Re: [PATCH 0/9] Add library mode to source builtin

[Koichi Murase]

Honestly, I share the same feeling as the others about the name
`library'. The use of the ``library'' feels something wrong in this
context.

I still think the variable name should be BASH_SOURCE_PATH. I've
suggested `-l' as an *example*, but that's just because I couldn't
come up with another letter. It could probably be `-p' following the
similar option of `command -p'.

The reasons that using "library" in this context feels strange are as follows:

* The suggested feature just changes how the script file would be
searched. This doesn't define what would be the library in Bash and
what would not. The suggested feature can be used to load a library,
but the reverse is not true; a library doesn't need to be sourced
through the suggested feature. Even if a script file that contains a
set of utilities is sourced with an absolute/relative path with a
slash, I'd still call it a library. Even if it is found in PATH, I'd
still call it a library. Even if it is found in the current working
directory, I'd still call it a library.

The suggested command-line flag forces the search to be in
BASH_LIBRARIES_PATH and disables the search of libraries in PATH and
the current working directory. You seem to try to justify the naming
by explaining how the script files loaded by the suggested feature can
be considered libraries, but to justify the naming you would need to
explain also how the script files with utilities located in PATH or
the current working directory are NOT considered libraries. If you
would try that, I will contest against it.

* We haven't been using the term `library' in the Bash codebase to
mean the general scripts that contain a set of functions and are
intended to be sourced. These have been just called script files
without a specific name. When we introduce a new term, I'd expect a
clear difference from the existing one. If the new mechanism is
incompatible with the conventional script files, it would suffice as a
difference. Or if the suggested feature would introduce new mechanisms
dedicated to libraries (such as namespacing, dependency resolution,
etc.) and those mechanisms are not going to be provided/available with
the conventional `source' builtin, it would also suffice as a
difference. As far as the new term means essentially a similar thing
as existing script files, I don't see a reason to introduce a new
term.

* I feel not every script file that is intended to be sourced and
usefully placed in BASH_LIBRARIES_PATH would be actually a library. It
might define aliases and shell functions that the user can
interactively call from the command line. It can also be an
initialization code for the environment. It can also be just personal
configurations. For example, would you call a single line file « eval
"$(starship init bash)" » a library? Maybe you call it a library, but
I wouldn't.

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

2024年5月6日(月) 23:01 Koichi Murase :
> > The whole issue here is that in some circumstances bash defers that
> > notification too long, and doesn't do it before the user can run `jobs'.
> > You don't need to change `jobs' so it doesn't print foreground jobs
> > that were terminated by a signal, you need to figure out why those jobs
> > are still in the list when `jobs' runs.
>
> After figuring out, what should I do? If it is the only thing I can
> do, does that mean the jobs builtin continues to print the foreground
> jobs with funsubs/traps/etc? Is it impossible to differentiate the
> foreground jobs from the background ones in the output of the jobs
> builtin?

I've checked the latest devel push. I confirmed the new behaviors that
the signaled foreground jobs are immediately notified even in the middle of
trap/bindx/PROMPT_COMMAND, and therefore the `jobs' builtin wouldn't
print the foreground jobs anymore. This would solve a long-standing
issue of my framework about the ambiguous foreground and background
jobs. Thank you.

Also, I confirmed that a=${ (true) }; b=${ jobs; } would not report
the foreground job `(true)'.

Thank you for your consideration and discussion.

--
Koichi

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

Thank you.

2024年5月6日(月) 22:16 Chet Ramey :
> On 5/4/24 8:47 PM, Koichi Murase wrote:
> > If we take the theory that « the notification of the signaled
> > foreground jobs are required because it is not considered `reported'
> > by `$?' », given that the shells don't notify the normally terminated
> > foreground jobs, the jobs builtins are still required to print the
> > unnotified foreground jobs. This theory is not consistent with the
> > actual behavior in shells.
>
> That doesn't follow. The notification isn't performed by the jobs
> builtin (the shell doesn't run `jobs' internally).

Yes, of course. But in this thread, I have been talking about `sleep
20; jobs' [1] where the user explicitly calls the `jobs' builtin.

[1] Second last paragraph in
https://lists.gnu.org/archive/html/bug-bash/2024-05/msg0.html

"The jobs builtin" in my previous reply corresponds to `jobs' called
after `sleep 20' in `sleep 20; jobs'. It's not something internally
called somewhere.

> Since no one notifies
> about foreground jobs that aren't terminated by signals, we can omit
> them from the discussion -- they are removed from the jobs list as soon
> as they terminate.

I think this is my issue of interpreting the standard, but isn't the
`jobs' utility required by the standard to print "all jobs whose
status has changed and have not been reported by the shell"? If I
literally read it with the theory "$? isn't counted as `reporting'",
in the case `sleep 20; jobs' (where `sleep 20' is not terminated by
signals), the job `sleep 20' isn't notified at its termination, so the
subsequent `jobs' builtin is supposed to print the job "whose status
has not been reported". Am I missing something?

Hmm, maybe you mean that a shell is free to anytime remove the
terminated foreground jobs from the list regardless of whether it has
been reported or not, and "all jobs whose ... reported" do not count
the foreground jobs that the shell has removed from the list
regardless of reporting. Is this your interpretation?

> The whole issue here is that in some circumstances bash defers that
> notification too long, and doesn't do it before the user can run `jobs'.
> You don't need to change `jobs' so it doesn't print foreground jobs
> that were terminated by a signal, you need to figure out why those jobs
> are still in the list when `jobs' runs.

After figuring out, what should I do? If it is the only thing I can
do, does that mean the jobs builtin continues to print the foreground
jobs with funsubs/traps/etc? Is it impossible to differentiate the
foreground jobs from the background ones in the output of the jobs
builtin?

--
Koichi

Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月6日(月) 14:28 Matheus Afonso Martins Moreira :
> > I fail to see how this could possibly save "thousands and thousands
> > of lines of code".
>
> How many lines of code were needed to implement the module managers?
> Probably a lot of lines. I guessed it was in the "thousands of lines" 
> ballpark.

I thought that part "it potentially reduces thousands and thousands of
lines of code" in [1] was a part of describing "what Matheus thought
before submitting the patch", but now we shared that the situation in
module managers is not like that. Isn't it?

[1] https://lists.gnu.org/archive/html/bug-bash/2024-05/msg00053.html

Anyway, it's never true that thousands of lines are reduced by the
suggested feature.

> However many lines that is, this feature would allow me
> to reduce them all to zero by not actually needing a
> module manager.

It's trivial that your code reduces to zero because you designed your
builtin to satisfy your needs. That doesn't show anything about its
generality.

> At the very least, it will save users from having to
> implement an import function themselves.
> They could still do it but they won't have to.

That argument would apply to any other useful shell functions, but I
don't think we should make everything builtin.

> >  You keep talking about this proposal as if it would spark a huge
> >  sea change in script maintenance, but I just don't see it, at all.
>
> A native way to source libraries. Built into bash, available to all users.

That's the source builtin.

> With the ergonomics you expect it to have. With good defaults, no setup.

It's just your expectation of the ergonomics but not ours. The
suggestion would be more useful for sure, but it doesn't mean it's the
best. You haven't made any survey on existing implementations.

> That's an incredibly huge change.

If it were really "incredibly huge", I would try to block this change
to carefully think about the impact in the future, other
possibilities, and a possible better solution (e.g. with
shell-function implementations or with a possibly different interface
of a builtin). However, don't worry, I don't think the actual impact
is that large...

> Especially considering how little code
> was required to make it happen.

The submitted patch seems unneededly large. It also touches something
else including incomplete addition of the long option of builtins, and
an incomplete support for the XDG base directories, which should
actually be discussed separately.

> People see bash as a language that scales down instead of up.
> The shellcheck tool helps a lot with shell script development but
> managing complexity is still quite difficult.

What's wrong with it? Even on this list, I think people's attitude
about the future of Bash varies, but I don't think the community
thinks Bash should replace or compete with Python or other general
purpose scripting languages. The shell language is not a language
intended to scale up in general, though it is free to attempt that in
personal projects. I think Bash might acquire a necessary feature in
the future, but I don't think we should hastily add every unnecessary
feature just because that seems useful. Honestly, if we think about
the necessity, I don't think the suggested feature is indispensable.

> This could really change that.
>
> > AFAICT the only thing this proposal addresses is your personal
> > hangup about using PATH to find sourced files
>
> It's not a personal hangup.

It does seem your personal hangup not giving a filename extension to
the script files, which ended up with mixed script files to be sourced
and executable files.

> Alternatively, you reinvent part of the functionality of source
> by resolving the path yourself and giving it an absolute path.
> None of these solutions seem right to me.

If you think that is the problem, your suggestion doesn't actually
solve the problem of module managers. Module managers will have to
implement it by themselves to satisfy their needs (which is obviously
different from your ergonomics) even if your feature is added to Bash.

> > Much like the periodic requests for XDG-organized startup files
>
> I think I deserve at least a little more credit than that.
> [...] _the bash maintainer_
> replied to my email saying he thought it was a
> reasonable feature.

Is this your rebuttal to Lawrence's "The sourced files that can be
written with your feature available are exactly the same as the ones
that can be written right now. Much like the periodic requests for
XDG-organized startup files.". If you avoid discussion by showing the
relation with the authority, it seems a proof of lacking a
straightforward counter argument. In addition, as far as I understand,
Chet mentioned the additional variable for the locations to search,
but not a new builtin or a flag.

> I interpreted that as a strong
> signal that this would be a welcome contribution.

I don't think a welcome contribution means the merge without any
reviews and discussions. If the

Re: Re: Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月5日(日) 13:36 Matheus Afonso Martins Moreira :
> > If your purpose is just to solve a small inconvenience of the `source'
> > builtin mixing the namespaces of local scripts, "libraries", and
> > executables, I think the suggested `source -i' or `source -l' would be
> > fine. I think no additional reasoning is needed.
>
> That is my purpose!

OK.

> My solution to the file extensions problem for example is to
> just not have file extensions at all,

This is not normal. I think that is the source of your problem. Even
though it is not strict, executable files (including shell scripts
intended to be executed in a separate process) in PATH usually do not
have the filename extensions. On the other hand, the scripts files
intended to be sourced would typically have a filename extension of
.sh or .bash (or even double extensions such as `.theme.sh,'
`.plugin.sh', and `.alias.sh' depending on the type of the module. In
this way, the namespace of the script files to be sourced and
executable files are "roughly" separated. Of course, it is not strict
and users can still have an executable script with a filename
extension. In some cases, we would have a set of script files to be
sourced without extensions, but those script files are usually
intended to be managed by a specific module manager/loader that
resolves the path by itself (such as /examples/functions/autoload and
bash-completion _comp_load).

Nevertheless, this makes the proposed change less convincing more or
less..Do you think you submitted the patch if you would have had
script files with filename extensions from the beginning? If you
followed the typical arrangement of script files to be sourced, the
problem you faced shouldn't have been that serious.

> > However, in that case, I don't see much difference from
> > « PATH= source » (as far as the module just defines a set of 
> > functions).
>
> It's true that the code is more or less the same from the perspective
> of module managers. However, building this into bash does provide
> a subtle benefit: separation between executable and module paths.
>
> Chet Ramey outlined it in his email:
>
> No need to ensure that they'll not attempt to run external commands.
> The builtin is therefore less limited than a pure PATH solution.

Yes, but that's implied my phrase ``as far as the module just defines
a set of functions''. In case, I note that the functions defined in
the sourced script files can call external commands after the sourcing
completes because the shell function uses "the value of PATH of the
current dynamic context" but not "the value of PATH when the function
is defined".

Anyway, I agree it is a problem that the sourced script cannot call
the external commands if the script wants to do actual processing
(such as initializations) instead of just defining a set of functions
and variables. Nevertheless, it is technically possible to implement
the proper identification of the script file in a shell function with
a few dozens of lines, though it wouldn't be a single line
implementation.

> This would be most useful for a module manager that doesn't resolve
> the module path by itself.

I still don't think it is useful as I already wrote in the previous
reply. I don't think there is a demand to reduce a few lines or a few
dozens of lines from the module manager.

Also, the benefit in the module manager seems too specific. Not many
people try to implement the module manager. The language wouldn't be
designed solely for module managers. The module managers could be an
important piece of a language, but that's not all the important
aspects of a language.

If you were a developer of a module manager and faced a serious
problem in implementing it, which is impossible to solve with an
existing language feature set, the feature request would have been
understood. However, you just seem to raise this example as a
hypothetical project, which seems hardly convincing.

> I agree that a one line of code reduction is underwhelming
> but that's not the only reason to include it. The users are the
> main beneficiaries. This gives users a simple way to load
> modules without using any module managers at all.
> From the perspective of the user, it potentially reduces
> thousands and thousands of lines of code.

I'll reply to it in a corresponding thread by Lawrence.

> Still, I did and still do hope that it will make life easier
> for the maintainers of the module managers too.

If you still hope that, I believe your patches would be blocked until
we test the design by the actual implementation of module managers
based on it and show it's validity. Furthermore, I think it wouldn't
get into Bash because I don't think it's possible to show it. This
feature is simply unnecessary for module managers. Sure it might be
used to reduce a small amount of code, but it's not an indispensable
feature. It's not sufficient to justify the change.

> > Would you try to exclude the possibility
> > of a module manager in the future

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

2024年5月3日(金) 23:36 Chet Ramey :
> > To the email [2]. In particular, to the question in the second paragraph 
> > [2].
>
> OK. No, it doesn't make sense to do that.
>
> The better option is to change the notification strategy so that the jobs
> list doesn't contain terminated unnotified foreground jobs when run in a
> trap.

Thank you for your reply.

I agree with that. Actually, that should have been the intent of my
initial patch, though I don't remember how I did it exactly at that
time. Anyway, I guess you will have a better idea about how it should
be modified.

> > I think the description of the standard is still unclear, or the
> > standard doesn't try to specify this detail.
>
> Probably the latter.

I agree.

> > In that theory, what happens when `sleep 20' is successfully
> > terminated without any signals?
>
> Shells wouldn't, and don't, notify about that.

Then, after that, does `jobs' after the `sleep 20' print the
information about the foreground job `sleep 20'? That was my question.
If we take the theory that « the notification of the signaled
foreground jobs are required because it is not considered `reported'
by `$?' », given that the shells don't notify the normally terminated
foreground jobs, the jobs builtins are still required to print the
unnotified foreground jobs. This theory is not consistent with the
actual behavior in shells.

> > With this interpretation, the `jobs' builtin will never print
> > foreground jobs because, at the point when the `jobs' builtin is
> > running in the foreground, any other foreground jobs should have been
> > terminated (while setting $?) or turned into background jobs as no
> > more than one jobs can run in the foreground at the same time.
>
> Sure. In addition, I want to preserve the behavior of notifying about
> terminated foreground jobs that were killed by a signal we don't ignore.

I think it is better to preserve the behavior of notifying about
signaled foreground jobs, yet I wouldn't expect the jobs builtin's
behavior of reporting foreground jobs in funsubs to be preserved. The
standard only describes the job status report of the background jobs
in Issue 8 [XCU 2.11]. Even if the notification of the foreground jobs
is not mentioned in the standard, I think the foreground-job
notification is not forbidden.



I think it is simpler to update the standard wording of [XCU 3 jobs -
OPTIONS] from "all jobs whose ..." to "all background jobs whose "
instead of trying to interpret `setting $?' as a way of `job status
reporting' and letting the readers deduce the fact that the jobs
builtin will not print the foreground jobs. Draft [XCU 3 jobs -
DESCRIPTION / paragraph 1] says "the jobs utility shall display the
status of background jobs that ...", so I think the jobs builtin is
intended for printing the background jobs.

--
Koichi

Re: Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

If your purpose is just to solve a small inconvenience of the `source'
builtin mixing the namespaces of local scripts, "libraries", and
executables, I think the suggested `source -i' or `source -l' would be
fine. I think no additional reasoning is needed. In this case, the
"#pragma once" feature wouldn't be needed or should be a separate
flag. You can of course add extra reasoning about the necessity of
this small improvement, but the problem is that I don't think those
arguments are valid.

If your goal is to facilitate the primitive parts needed for library
management, I don't think the current suggestion would actually help.
The source builtin is already the primitive feature that can be used
to develop a module manager on top of it. I don't think any additions
to the source builtin is useful for developing a module manager
because the typical use cases by the module managers are different
from your assumptions (as will be discussed later in this reply). This
is a YAGNI feature, which hasn't been tested by real use cases of the
intended usage. Then, you don't have a motivation to modify the source
builtin or add a new builtin at present.

The question is whether you would still value `source -i' if it were
not useful for module managers at all. To me, you seem to raise the
bar by yourself. Where is your bar?

2024年5月4日(土) 11:44 Matheus Afonso Martins Moreira :
> I think we were unable to reach an understanding because we were both
> using the same words to talk about different things.

Can I call what you describe the "module loader"? Then,

> Those are problems better
> solved by Linux distribution packagers or a custom package manager for bash.
> I may be wrong but I believe that is the role many of those bash frameworks
> are fulfilling.

Let me call the above the "module manager".

I understand that the purpose of the *present* patch is the module
loader, but I thought you tried to prepare the primitive useful for a
unique module manager. This is because you mention that a language
should have only one module system so that the library developers do
not need to care about the variation. The variations in the module
managers is a problem since the library itself needs to be adjusted
under the rule of the module manager (for dependency resolution, etc).
This is a general problem for a programming language, and recent
programming languages have respective standard ways of module
management. But maybe you've been discussing something else.

> So at least to me this is all about the "import" primitive.
> Which particular form it takes is up to debate, it's something
> that's worth discussing and getting right before it's released
> due to the difficulty in changing such core APIs after the fact.
> However, the fact is that I think many of those other considerations
> are actually out of scope!

I understand you are not going to develop the module manager but just
want to focus on the module-loader part. Nevertheless, I think you can
agree that we want to discuss the actual use cases of the "import"
primitive in order to shape the needed core API in a usable form
(Without mentioning the use cases, what would be discussed?). Besides,
you seem to assume the future uses of the "import" primitive in module
managers. Then, we cannot avoid the discussion about the use cases in
the module managers.

> Those are problems better
> solved by Linux distribution packagers or a custom package manager for bash.
> I may be wrong but I believe that is the role many of those bash frameworks
> are fulfilling. In order to do that, they _had_ to create an import primitive
> for themselves because bash lacked such facilities. My intention is to fix
> the lack of the import facilities, not to replace those frameworks.

Yes, we had to create the "import" command by ourselves, but the
situation doesn't change at all even after the suggested modification.
.

> Ideally, they'll be using bash's native import under the hood!

Yes, module managers still need to implement their own "import"
command while using the proposed "import" primitive under the hood,
and it's simply interchangeable with the source builtin we already
have.

* Module managers typically try to identify the file under more
detailed rules, which cannot be implemented by just the `source -i'
builtin. For example, module managers allow dropping the filename
extension from the specified name; the "import moduleA" command would
search the file "moduleA", "moduleA.sh", or "moduleA.bash". Or some
other managers identify the version of a module based on the requested
range. Some frameworks try to first identify the system version of the
module and fall back to the bundled version depending on some
conditions. Those rules depend on the module manager. Also, some
module managers want to manage the actual paths of the modules that
are loaded. For these reasons, most module managers actually resolve
the path by itself using their rules and specify *the absolute path*
to the so

Re: Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

I think we can separate the discussion about the module system and the
source builtin. What is the primary purpose or the reasoning behind
the suggestion?

If the purpose is to introduce a standard module system in Bash, I'll
have to be opposed to the change because I think the current
suggestion for `source -i' wouldn't help satisfy that purpose at all.
At least, it is immature or insufficient to be used as a module
system. There are many things to consider, including the include guard
I already mentioned, a central repository to avoid conflicts in the
library names, etc. You also need to fight against existing module
frameworks to convince them about introducing a mechanism that might
beat them. If one has to implement a module system using a set of
shell functions based on `source -i', the situation doesn't change
from the current one with many module systems based on the source
builtin.

However, if the purpose is solely to solve the ambiguous search paths
of the `source' builtin, I think a flag to the source builtin is a
reasonable suggestion. Currently, the command `source ' might
hit three different types of files:

1. a file in the current working directory
2. a script file (that is supposed to be sourced) in PATH
3. this is not intended, but an executable file that is not supposed
to be sourced.

To make the source builtin find type 1, one can use `source ./'.
However, there is no way to force type 2 without being affected by
type 1 or 3 when the library is not found and/or there is a
conflicting name. Then, we can prepare a separate variable
BASH_IMPORT_PATH for the search paths for script files to source, and
add a flag of the source builtin that forces searching only
BASH_IMPORT_PATH. It's similar to `unset -v/-f/-n' where the command
`unset ' can ambiguously remove a variable or a function, but
`one can specify a type through -v/-f/-n.

This is my personal preference for the name, but the current simple
feature of restricting the search paths doesn't feel `import'. I'd
think it can simply be BASH_SOURCE_PATH. The flag can also be
something other than `-i' or `--import', like `source -l ' to
mean the "library".

2024年5月4日(土) 3:00 Matheus Afonso Martins Moreira :
> >  No, those files need to be distributed by the Bash package of
> >  distributions, and the path needs to be properly set up.
>
> Then I don't think that would be the ideal solution.

I feel we should ask the package maintainers to include those files.
As for the loadable builtins, even though the recent versions of Bash
install loadable builtins in $prefix/lib/bash when installed by «
./configure && make install », most package maintainers do not include
them in the (main) package for some reason. As a result, even if Bash
provides useful loadable builtins, the script writers cannot assume
their existence, which is a problem. Currently, the files under
/examples/functions are not installed even by « ./configure & make
install », but I think useful shell functions can also be installed in
e.g. $prefix/share/bash/functions, and also included in the packages
of the distributions.

> It's embarrassing but the truth is I was not even aware of these projects.

Don't worry. You are here to ask the community about that.

> My intention was simply to add a feature I wanted to bash.
> I experienced difficulty with maintaining a growing collection
> of personal scripts, had this idea, coded it up and sent patches.

If it were the personal purposes, you could start it with a shell
function implementation. There is no need to modify the upstream Bash
for personal uses. It's not too late to ask for standardization when
you need to share your framework with others.

> >  I also think the variation is actually needed because the assumed
> >  search paths are different for different libraries. We cannot simply
> >  merge all those implementations into a single command because each
> >  library wants to limit BASH_IMPORT_PATH to the locations they provide.
>
> Why is this required?
>
> >  Then, it's identical to `PATH= source'. I also think the
> >  variation is not a problem. The separation of the domain is useful.
>
> Please elaborate on this point. How is it useful?

I assume the case where different module systems coexist. In
particular, interactive frameworks and applications need to be loaded
in a single instance of Bash even when they are not integrated with
each other. The different module systems do not need to cooperate with
each other but should be allowed to coexist in a session and work
without interfering with each other.

For example, when module X managed by system A requests another module
Y, we want to find module Y provided by system A and exclude the
modules provided by system B (even when no module is provided with the
conflicting name Y in system B). The idea is the same as yours to
separate the namespace of the executable files and the script files to
source.

> > If desired, the developer can choose one module framework
> > an

Re: Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月3日(金) 14:45 Matheus Afonso Martins Moreira :
> This is suboptimal. In case the library script cannot found
> in the intended directories, it will needlessly search every
> other entry in PATH for it, despite the fact they will contain
> only unsuitable executables and programs.

There is no doubt that it is more optimal to skip the unnecessary
search, but it doesn't need to be implemented as a new builtin. The
question is whether such a micro-optimization deserves a new builtin.
As far as the scripts are properly set up, the search failure
shouldn't happen. This micro-optimization would only have an effect in
the situation where the shell program does not work. Usually, we do
not care about the performance in a situation where we need to report
an error.

> It also raises the possibility of false positives:
> finding a different executable with the same name
> and sourcing it, causing hard to debug problems.

Yes, that is a problem in directly using the source builtin, but I
think the original intention for the design of `.' and `source' is
that the library shouldn't have the same name as exiting commands.
Again, this is related to the improper setup of the library. Then,
there is always the possibility of a name conflict between different
libraries that share their names even if it doesn't conflict with
executable files. Yeah, but sourcing an executable file can be more
problematic than sourcing a different library.

> >  [...], but you can/should put the files without adding the
> >  executable permission to avoid the confusion you mention.
>
> That's true and a good point. It was my intention to
> reduce the number of candidate directory entries as
> much as possible. Would you say that it would never
> be appropriate for a shell script library to be executable?

As far as it is not intended to be an executable file, yes. By "an
executable file", I mean a file that will be executed in a separate
process.

However, I shall note that some shell programs are intended to be used
as both a library and a separate program. This is also pointed out by
Phi. For example, `modernish' from the modernish framework is
basically intended to be a script to source but can also be used as a
separate program. When it is used as a separate program, for example,
it can be used to retrieve information about the library and setups
and can also be used to run tests. The main file of my framework,
`ble.sh', also behaves in a similar manner, though I do not give the
executable permission to it by default and instead instruct the user
to use `bash /path/to/ble.sh [options]'. Nevertheless, I know that
some users put a symbolic link to `ble.sh' in ~/bin or somwhere and
adds the executable permission to `ble.sh'.

> It might be useful to reverse that condition: make it so
> only non-executable files can be imported instead.
> What do you think?

For the above reason, I think you still need to search for both types of files.

> What if instead of a new builtin, we added options to source?
> This:
>
> import some-file
>
> Could become this:
>
> source -i some-file
> source --import some-file

I think this is more acceptable compared to the new `import' builtin,
but I still think it can be a shell function (or an alias to work
around libraries not using `declare -g' as Chet mentioned).

> >  If this builtin would be included, I think this should be a loadable
> >  builtin that the user can enable as an option.
>
> That's a good idea. When I studied the source code,
> there were comments saying the builtins could be
> turned off and even programatically detected
> via the enable builtin. I made use of those
> facilities so the import builtin should already
> have these properties.

Yes, all the builtins can be enabled/disabled by the `enable' builtin.
But what I wanted to say is that the `import' builtin should be an
opt-in feature, i.e., it should be disabled by default to avoid
conflicts with existing `import'.

> I'm not sure if it's a dynamically loadable module, but it probably could be.

It's not in the current patch but can be modified to be a loadable builtin.

> > it should better be a script file in /examples/functions
>
> In this case, would every script automatically
> have access to the import function?

No, those files need to be distributed by the Bash package of
distributions, and the path needs to be properly set up.

> >  so simple that I doubt it should be defined as a separate library
> >  file. It can be copied directly into the script file.
>
> It could, but that opens the door to variation.

>From your perspective, I understand you are trying to invent the
standard. However, it seems to me that you are trying to add even
another variation (which is less functional than the existing
implementations) to the market.

https://xkcd.com/927/

You are overwriting `import' of the Bash Infinity framework [1]. As
other implementations, Bash Line Editor has `ble-import' [2]. The
Basher framework has `inclu

Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月3日(金) 11:17 Lawrence Velázquez :
> If "import" were qualitatively different from "source" (say it
> integrated some sort of namespace functionality), then it would be
> more compelling.

That sounds interesting. It's a bit unrelated to the main discussion,
but I was wondering if it is possible to allow expansions in the
function name  in the function definition "function 
". Zsh allows it:

$ zsh -c 'ns=mynamespace; function $ns/util1 { echo This is util1; };
mynamespace/util1'
This is util1

This is just a hypothetical feature, but if we have a PATH-like
variable (e.g. BASH_FUNCTION_PATH or BASH_FUNCTION_NAMESPACE) for the
function name search, one can set « local
BASH_FUNCTION_PATH=mynamespace » to make `mynamespace/util1' visible
with the name `util1'.

But a problem of such a variable-based approach is that the namespace
is resolved by a dynamic condition (i.e. the variable in the context
when it is called) instead of the lexical context (i.e. where the
caller is defined or to which namespace the caller belongs). Maybe `.'
in BASH_FUNCTION_PATH can be treated as any namespace the caller
belongs to.



By the name "import", I expect also something like an include guard,
i.e., it loads the same file only once at most. I have a
shell-function implementation,`ble-import', in my framework. It
doesn't support namespace, but it supports the include guard. It also
accepts a setting for the paths to search. It also has a feature like
`with-eval-after-load' of elisp (or the "onload" event).

--
Koichi

Re: [PATCH 0/4] Add import builtin

[Koichi Murase]

2024年5月3日(金) 9:56 Matheus Afonso Martins Moreira
:
> The source builtin uses the PATH variable for resolving file names
> which means they would have to be placed alongside normal executables
> which could cause confusion.

You can prepare a separate directory for the script files to source
and put the directory name at the beginning of PATH, e.g.,
PATH=~/.config/bash/functions:$PATH. As Lawrence has pointed out, you
seem to assume the script files to source have the executable
permissions, but you can/should put the files without adding the
executable permission to avoid the confusion you mention. The source
builtin finds the file regardless of the executable permission. In
this way, it is clear to both the users and Bash that those files are
not executable.

In addition, BASH_LIBRARY_PATH (or BASH_SOURCE_PATH) for the source
builtin seems already in the ``list'' [1]. The new builtin would
become unnecessary If it is implemented.

[1] https://lists.gnu.org/archive/html/help-bash/2024-04/msg00022.html

> This patch set introduces a new "import" builtin which behaves just like
> the source builtin but only looks up executables in the BASH_IMPORT_PATH
> variable.

We need to be careful in adding a new builtin name since existing
users might already have an executable file "import", which would be
hidden by the suggested new builtin. The name "import" is not listed
in POSIX XCU 2.9.1 (1b), the list of command names that a shell
implementation might process specially.

If this builtin would be included, I think this should be a loadable
builtin that the user can enable as an option. The loadable builtins
are included in /examples/loadables in the source tree of Bash. The
location of the loadable builtin files can be specified by
BASH_LOADABLES_PATH and loaded by the `enable' builtin.

However, as pointed out by Dale, what is done by this new feature can
just be implemented by a shell function. In this case, it is generally
recommended to implement it as a shell function. If you want to
include it in Bash, it should better be a script file in
/examples/functions (instead of a loadable module in
/examples/loadables). But considering the simplicity of the shell
function implementation by Dale and the fact that the source builtin
essentially covers the feature, I'm not sure if it's worth providing.



I've checked the discussion in help-bash [2]. The above arguments seem
to be already mentioned in the original discussion. You say you've
included everyone's thoughts [3,4], but no one seems to have
recommended adding a new builtin.

[2] https://lists.gnu.org/archive/html/help-bash/2024-04/threads.html#00016
[3] https://lists.gnu.org/archive/html/help-bash/2024-05/msg0.html
[4] https://lists.gnu.org/archive/html/help-bash/2024-05/msg1.html



Reply to [4]

2024年5月3日(金) 9:56 Matheus Afonso Martins Moreira
:
> Bash is powerful enough to do it with a function, no doubt about it.
> The thing is that approach begs the question of how to source that
> function which is also a reusable library function. This chicken and
> egg problem is why I believe a library facility is something that's
> better off as a native feature of the language.

The function « import() { PATH=$BASH_IMPORT_PATH source "$@"; } » is
so simple that I doubt it should be defined as a separate library
file. It can be copied directly into the script file. Even if it would
be defined in a library script file, you need to anyway specify a path
in BASH_IMPORT_PATH to use that function. I don't feel much difference
between specifying two paths (set BASH_IMPORT_PATH and source
/path/to/import.sh) and specifying only the former (set
BASH_IMPORT_PATH and source import.sh).

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

2024年5月1日(水) 22:58 Chet Ramey :
> Yes, and I said I could work around this case.

Thank you for your consideration.

> > [1] https://lists.gnu.org/archive/html/bug-bash/2022-10/threads.html#00054
> > [2] https://lists.gnu.org/archive/html/bug-bash/2022-11/threads.html#00049
> > (continued. I haven't received a reply)
>
> To what?

To the email [2]. In particular, to the question in the second paragraph [2].

> > I'm not sure about the answer, but kre's opinion seems reasonable to
> > me.
>
> OK. The standard does say "reporting the exit status with the special
> parameter '?'" in
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_02

I think the description of the standard is still unclear, or the
standard doesn't try to specify this detail. I'm not sure whether we
can relate "the exit status reported by $?" and "the job status that
has been reported'' just from the description of the standard.
However, when we try to define "the job status that has been reported"
consistently with both the standard description and the actual
behaviors of shells, I think it is *one* reasonable interpretation to
regard the job status is reported when the exit status is stored in
$?.

> Correct. Is setting $? sufficient, or does the standard mean the reporting
> on foreground jobs that are killed by a signal -- one that shells don't
> ignore -- before the next command, which is what all shells do?
>
> For instance, if you run
>
> sleep 20; jobs

As far as I test it, yash and zsh don't print the information of the
signaled foreground jobs in this case, though ash/ksh families and
bash/posh print it as you tell. If we interpret $? can be used to
``report'' the foreground job status, the standard technically
wouldn't require printing the job information to stdout in this case,
but it doesn't prohibit printing either.

> and hit the `sleep' with SIGUSR1 from another process, all shells will
> print some message about the process exiting due to SIGUSR1 before `jobs'
> is executed, and `jobs' prints nothing, because the reporting has already
> happened.

In that theory, what happens when `sleep 20' is successfully
terminated without any signals? If the `reporting' must be the actual
text printed to stdout, « sleep 20; jobs » should print the job
information of the foreground terminated job `sleep 20', but not a
single does that.



To summarize my current thinking, I feel the standard didn't intend to
specify the detailed behavior for the foreground jobs. I don't expect
the `jobs' builtin to print any foreground jobs for consistency with
the behavior in the existing shells (except for Bash
trap/bindx/PROMPT_COMMAND handlers and 5.3 funsubs). An
``interpretation'' of the POSIX standard consistent with the existing
shell behavior would be to count $? as `reporting'.

With this interpretation, the `jobs' builtin will never print
foreground jobs because, at the point when the `jobs' builtin is
running in the foreground, any other foreground jobs should have been
terminated (while setting $?) or turned into background jobs as no
more than one jobs can run in the foreground at the same time. In a
case, the `jobs' builtin is inside a background job, the foreground
jobs that started after forking the background jobs are not visible to
the `jobs' builtin anyway.

--
Koichi

Re: 5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

2024年4月30日(火) 5:07 Chet Ramey :
> OK, let's explore this again.

I haven't yet been convinced by the previous discussion [1,2] about
the reporting of the foreground dead jobs in the trap handler, but
this time, the situation is slightly different. The `jobs' builtin
reports the foreground dead jobs even in a normal context of the
commands executed through the command line when the new function
substitutions are used.

[1] https://lists.gnu.org/archive/html/bug-bash/2022-10/threads.html#00054
[2] https://lists.gnu.org/archive/html/bug-bash/2022-11/threads.html#00049
(continued. I haven't received a reply)

> POSIX says `jobs' should report on
>
> "the status [...] not been reported by the shell"
>
> So the question is what exactly constitutes a foreground job whose status
> has "not been reported."

I'm not sure about the answer, but kre's opinion seems reasonable to
me. If we need to report all the foreground jobs executed in a normal
way, jobs will have to remember all the foreground jobs executed in
the past since the last call of jobs, but I don't think that is a
traditional behavior of most shells. POSIX Draft (202x/D4.1) XCU 2.11
mentions the reporting of the ``background'' jobs ``immediately prior
to writing the next prompt'', but the reporting of the foreground jobs
does not seem to be described.

With this report, I'm not just talking about the implementation of the
`jobs' builtin for a given list of jobs. I guess this should also be
related to the timing when the entries of foreground jobs in the
internal table should be cleared (or marked as ``notified''), though I
haven't checked the actual internal processing in the current
codebase. In particular, since the foreground jobs are not reported by
« (true); b=${ jobs; } » (where only « a=${ (true) } » is changed to «
(true) » and the invocation of the `jobs' builtin is unchanged), it is
more likely to be related to the management of the foreground jobs in
the internal job table rather than the implementation of the `jobs'
builtin, I guess.

--
Koichi

5.3-alpha: value substitutions remove unrelated REPLY when its REPLY is removed

[Koichi Murase]

When REPLY that is prepared by a value substitution ${ ; } is
removed by , the value substitution removes the previous-scope
REPLY, which is not prepared by the value substitution, after the
execution of .

$ bash-5.3-alpha --norc
$ unlocal() { unset -v "$@"; }
$ f2() { unlocal "$1"; echo "f2:$1=${!1}"; }
$ f1() { local REPLY=1; a=${| f2 REPLY; }; echo "f1:REPLY=$REPLY"; }
$ f1
f2:REPLY=1
f1:REPLY=

Removing REPLY inside a value substitution is admittedly arguable.
Nevertheless, it is still more consistent for value substitutions to
remove only the REPLY they prepared, if present, and to avoid removing
unrelated REPLY.

--
Koichi

5.3-alpha: the `jobs' builtin prints foreground dead jobs with function substitutions

[Koichi Murase]

I'm currently testing my framework with 5.3-alpha. In 5.3-alpha and
the devel branch, foreground dead jobs are reported by the `jobs'
builtin with function substitutions:

$ bash-5.3-alpha --norc
$ a=${ (true) }; b=${ jobs; }; declare -p b
declare -- b="[1]   Done( true )"

I expect the jobs builtin to exclude the foreground dead jobs in the
above command. This doesn't happen with slightly different versions
like

$ a=${ (true) }; b=${ (jobs); }; declare -p b
$ (true); b=${ jobs; }; declare -p b
$ a=${ (true) }; b=$(jobs); declare -p b
$ a=${ (true) }; jobs
$ a=${ (true) }; (jobs)

I would like to get the list of the background dead jobs using the
`jobs' builtin, but this behavior contaminates it with the foreground
ones as far as I use the function substitutions. If the current
behavior would be the intended one, I would request an option of the
`jobs' builtin that excludes the foreground dead jobs.

--
Koichi

Re: Bug tracking

[Koichi Murase]

2024年4月1日(月) 9:35 Martin D Kealey :
> On Mon, 11 Dec 2023, 05:19 Chet Ramey,  wrote:
> > On 11/30/23 5:18 AM, Martin D Kealey wrote:
> > > If there's a bug tracking system beyond "threads in a mailing list", I'd
> > > like to know how I can get access to it.
> >
> > https://savannah.gnu.org/support/?group=bash
>
> That's a good start, but it seems incomplete, and there's little -- perhaps
> no -- overlap with bug reports in this list.
>
> Has bashbug always sent email to bug-bash@gnu.org, or was it previously fed
> into Savannah?

I have been also wondering how to subscribe to the bug-tracking system
of Bash on Savannah (through e-mail notifications, etc.). I created an
account on savannah.gnu.org in the past but could not find a way.

> Perhaps bug tracking could be migrated to a more modern system?

I think a more realistic solution for GNU is to improve Savane, which
is also an open-source project:
https://savannah.nongnu.org/projects/administration/

Re: exec redirection is undone if fd (>9) has O_CLOEXEC

[Koichi Murase]

2024年2月24日(土) 15:32 Emanuele Torre :
> On Sat, Feb 24, 2024 at 02:11:25PM +0900, Koichi Murase wrote:
> > I have a question.  Maybe it's not technically a bug as fd > 9 is
> > involved, but the resulting behavior appears to be strange.  I
> > received some reports from users in my project and tried to find out
> > what is happening.  This is a reduced case:
>
> Thank you for bringing this up; I have also noticed weird behaviour with
> redirections of cloexec file descriptors a couple months ago.

Thank you for your additional information.

> Bash seems to check for the CLOEXEC flag in the source file descriptor,
> and preserves it in the destination file descriptor when performing a
> n<&m redirection.

Ah, I have to confess that I'm already relying on this specific
behavior of copying CLOEXEC, so if it is changed now, I might have to
consider removing some features from my project.

You might expect the redirections would be essentially implemented by
a single syscall, but in reality, the shell redirections are more
complicated.  For example, we need to save and restore original
streams associated with the file descriptor.  In that sense, it's not
surprising to me that the shell performs extra operations (even if it
is not so obvious to the users), but I'm neutral about whether the
current behavior is better or worse.



Anyway, as far as I look at the source code, the behavior of copying
CLOEXEC is an intentional design and is not a random side effect of
some other processing.  In this sense, the strange behavior you
observe is a design issue.

Meanwhile, the behavior that `exec 50>2.txt' does not have an effect
does not seem to be the intentional design.  It seems to be just a
side effect of the manipulation of the saved fds.  In this sense, the
strange behavior I observe is the one that Chet did not primarily
intend, I guess.  But it's just my naive guess.

--
Koichi

exec redirection is undone if fd (>9) has O_CLOEXEC

[Koichi Murase]

I have a question.  Maybe it's not technically a bug as fd > 9 is
involved, but the resulting behavior appears to be strange.  I
received some reports from users in my project and tried to find out
what is happening.  This is a reduced case:

  #!/usr/bin/env bash
  enable -f /path/to/lib/bash/fdflags fdflags
  exec 50>1.txt
  fdflags -s +cloexec 50
  exec 50>2.txt
  echo hello >&50
  ls -l 1.txt 2.txt

The result is

  $ ./reduced.sh
  -rw-r--r-- 1 murase murase 6 2024-02-24 13:53:01 1.txt
  -rw-r--r-- 1 murase murase 0 2024-02-24 13:53:01 2.txt

I expect that the string `hello' is saved in 2.txt, but it actually
goes into 1.txt.  It turned out that the redirection "50>2.txt" is
actually performed, but it's immediately undone at the end of `exec'
command.  The undo redirection is set up in the if statement starting
on redir.c:1288 (devel 43ecbeb3).  The code comment says:

  /* experimental: if we're saving a redirection to undo for a file
  descriptor above SHELL_FD_BASE, add a redirection to be undone if
  the exec builtin causes redirections to be discarded.  There needs
  to be a difference between fds that are used to save other fds and
  then are the target of user redirections and fds that are just the
  target of user redirections.  We use the close-on-exec flag to tell
  the difference; fds > SHELL_FD_BASE that have the close-on-exec flag
  set are assumed to be fds used internally to save others. */

This piece of the code seems to be introduced in commit cac4cdbf
(ChangeLog says the change to redir.c was made on 2011-10-09):

https://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel&id=cac4cdbf5


I'm just curious about the background of this different treatment for
"fds that are used to save other fds and then are the target of user
redirections" (quoted from the code comment).  Also, it says
"experimental", but is there any alternative implementation, where the
user-supplied O_CLOEXEC fds are not undone?

Even if there is a technical background, the current behavior observed
by the users is strange, and this behavior doesn't seem to be
specified in the Bash documentation.  The documentation just says

  Redirections using file descriptors greater than 9 should be used
  with care, as they may conflict with file descriptors the shell uses
  internally.

but this behavior is non-trivial even if the user treats the fds with
"care".  Note that, in the actual code, the number 50 in the above
reduced case is chosen to be a number that is not used (which can be
tested by e.g. « ! : >&50 ». There can be false-negatives with this
test, but that's fine).

--
Koichi

Re: It is possible to remove the readonly attribute from {BASH, SHELL}OPTS

[Koichi Murase]

2024年2月20日(火) 18:12 Martin D Kealey :
> So yes please, I'd like "local" to push a new variable definition that
> inherits nothing from any outer one: not name-ref, not read-only, not array
> (of any kind), not assignment modifiers (integer, upper-case, lower-case),
> and above all, not any previous values.

In my understanding, Bash behaves as you hope from the beginning
except for the global readonly variables (2.04+) and the export
attribute. I also checked the behavior of old versions now, but Bash
seems to behave in that way at least in Bash 2.0. If I'd raise some
exceptions, I thought in the past there was a bug that some array
attribute or behavior were inherited to a local variable (but don't
remember exactly), but that's just a bug found in a very limited
situation. Also, the recently added option `shopt -s localvar_inherit'
turns on the behavior you hate, but it's not enabled by default.

> Ideally each function invocation would have its own variable namespace,
> only using the global namespace as a fall-back,

That breaks so many existing scripts.

> Maybe this would be twisting 'local' too much, and it warrants creating a
> new keyword such as 'var'?

Perl's `local' is dynamic scoping, but Perl later added `my' for the
lexical scoping. Eric Blake suggested `local -s' [1]. Alex xmb
suggested `local -L' [2].

[1] https://lists.gnu.org/archive/html/bug-bash/2010-11/msg00036.html
[2] https://lists.gnu.org/archive/html/help-bash/2021-05/msg00094.html

--
Koichi

Re: [PATCH] tests/array.tests: using grep -E instead of egrep

[Koichi Murase]

2024年2月19日(月) 18:01 Lawrence Velázquez :
> On Mon, Feb 19, 2024, at 4:08 AM, Xiangyu Chen wrote:
> > From: Xiangyu Chen 
> >
> > The console ouput lots of egrep warning message, those message causes
> > test case fail in diff stage.
>
> On what system does this happen?  The proposed changes might break the test 
> suite on some older systems.

GNU grep started to output error messages in v3.8 (2022-09) for egrep,
so the test failure will be happening in most GNU/Linux distributions
soon.

Re: declare -A +A

[Koichi Murase]

2024年2月13日(火) 6:13 Chet Ramey :
> Only for indexed and associative arrays, since those are the attributes
> that cause changes in the underlying value storage format. That's different
> than turning the integer attribute on and off, for instance.
>
> Should it be an actual error, or should the shell just cancel out the
> attribute change requests and go on? What do folks think?

I think it can be a usage error; it doesn't do anything on the
variable (i.e. doesn't make it an associative array) and outputs an
error message.

Re: About `M-C-e` expand result `'` failed

[Koichi Murase]

Thank you for the discussion.

2024年2月4日(日) 12:59 Martin D Kealey :
> I am generally concerned about breaking changes that affect existing
> scripts, but I see changes to readline as less problematic,

I also assume shell scripts, but shell scripts for interactive
settings.  Interactive settings can also be a large-scale shell
script.  There are many frameworks---such as fzf, atuin, autojump,
zoxide, sbp, liquidprompt, etc.

> since an interactive user gets the chance to check the replacement
> before hitting enter.

I don't think we can ensure that the users *always* have a chance to
check it.  It is possible to set up the readline macro so that it
immediately runs the expanded command (as I implied in my previous
reply by including \r at the end of an example macro string).  If it
is possible, then there are always some users utilizing the behavior
in the world no matter how one person considers "this is almost never
useful" or "an interactive user gets the chance".

In fact, I think there is a motivation to do so: To force
re-evaluation of PS1 to update the prompt (e.g. after changing the
current directory), a command needs to be executed through the
readline bindable function `accept-line'.  For example, one can
generate « cd '...' » by "\M-\C-e" and execute it by C-m in a single
macro string.

> Yes users have to learn new behaviours, which isn't ideal, but if my
> suggested "unquote" ("dequote"? "strip-quoting"?) bindable function
> was also added, the impact on users would be minimal: press M-C-e
> and then a second keypress to remove quotes, bringing the input
> buffer to the same state as would occur under the current
> arrangement. (By choosing a default binding for "unquote" that
> currently does nothing, people could then use the same keyboard
> arpeggio on both old and new bash.)

That's right, but I'm not talking about the cases where the user
directly inputs "\M-\C-e" on the keyboard.  I'm talking about macros
set up using « bind '"...": "..."' » by shell frameworks.  To me, the
above paragraph seems to be irrelevant to the present discussion.

> > If the requoting behavior would be desired, I strongly suggest
> > keeping the existing behavior of shell-expand-line but adding a
> > separate new bindable function (like
> > shell-expand-and-requote-line) to perform the expansion and
> > requoting.
>
> On reflection, this would be a fair compromise, at least in the
> short term.

Does it need to be short-term?  Do we need to remove the feature?

> Might we offer guidance that distros include a new binding for C-M-e
> in their supplied /etc/skel which would only affect new users, not
> existing users?

I have a native question.  Why do people on this thread discuss
changing the behavior of "\M-\C-e" even though there is still a large
space of key combinations?  "\M-\C-e" is already used in existing
shell frameworks for interactive uses, e.g. an old version of fzf
keybindings used "\M-\C-e" in macros for every Bash version (yet
currently only for bash <= 3.2).  There would be still customized
versions of fzf that are not updated, or other personal configurations
inspired by the fzf keybindings.  I don't see a reason to introduce
unnecessary conflicts when we can just pick another key combination
(e.g. "\M-\C-x") for `shell-expand-and-requote-line'.

--
Koichi

Re: About `M-C-e` expand result `'` failed

[Koichi Murase]

> Before Bash 3.2

P.S. Sorry, I meant "In Bash 3.2 and before,", where 3.2 is included
in the statement.

Re: About `M-C-e` expand result `'` failed

[Koichi Murase]

I was just watching the discussion because I thought the existing
behavior would never be changed as it behaved this way for a long
time. However, I now think I should leave a comment because even
Martin (who I believed was one of the careful people about backward
compatibility as seen in [1,2]) seems to suggest a breaking change.

https://lists.gnu.org/archive/html/bug-bash/2022-11/msg00017.html
https://lists.gnu.org/archive/html/bug-bash/2023-11/msg00150.html

2024年2月3日(土) 21:45 Martin D Kealey :
> If that's how the documentation is interpreted, then clearly there's a bug
> in the documentation, because this is almost never *useful* behaviour.

This is not true. Before Bash 3.2, READLINE_LINE and READLINE_POINT
were not supported, where Bash configurations used readline macros
(something like « bind '"xxx": "\C-u\C-k$(...)\C-\M-e\r"' ») to
replace the command line with a dynamic content. I.e., the macro first
inserts a text « $(some-code-generator) » and expands it by C-M-e.
Please note that $(some-code-generator) may want to include in a
generated command characters that should be treated as shell special
characters. If one wants to support a wide range of versions from Bash
3.2 (shipped with macOS) to the latest one, this is still one of the
used techniques. If the behavior of C-M-e or shell-expand-line is
going to be changed, such types of Bash configurations are all broken.
For example, if $(some-code-generator) generates « echo "Hello,
world!" », we have been expecting the output « Hello, world! ». If the
quoting were going to be applied in the next version, the output would
become « "Hello, world!" », which is not an expected behavior.

If the requoting behavior would be desired, I strongly suggest keeping
the existing behavior of shell-expand-line but adding a separate new
bindable function (like shell-expand-and-requote-line) to perform the
expansion and requoting.

--
Koichi

Re: bash aborts line-editing on tab-expansion with find pattern

[Koichi Murase]

2024年1月15日(月) 23:02 :
> Description:
> Found while editing history line with a find pattern.
> Apparently bash doesn't like + characters very much.
>
> Repeat-By:
> find . -type f -ctime +30 ! -ctime +40 -exec ls -la --full-time {} +
> Move cursor to any point after find and before -exec, press TAB, and 
> bash will print
> -bash: COMP_POINT: substring expression < 0

That is an issue of the "bash-completion" project, which provides the
configurations for the Bash programmable completion, but is not Bash's
issue. This is actually reported at [1], which was already fixed in
[2]. The fix is not yet in the release version of bash-completion, so
you need to use the master branch of bash-completion on GitHub if you
want a copy of bash-completion free from this issue.

[1] https://github.com/scop/bash-completion/issues/1080
[2] https://github.com/scop/bash-completion/pull/860

--
Koichi

Re: issue with debug trap

[Koichi Murase]

2023年12月16日(土) 6:22 Giacomo Comes :

The original post asks the reason for the difference in the status
between the following two cases:

if ((0)); then :; else echo "here \$? is 1"; fi
if ((0)); then :; fi; echo "but here \$? is 0"

and the answer is that the former returns the status of ((0)) (which
is $? = 1), but the latter returns the status of the entire
if-statement (which is $? = 0 because no branch is selected. If any
branch was selected, it would've been the exit status of the last
command in the selected branch).

--
Koichi

Re: $((expr)) allows the hexadecimal constant "0x"

[Koichi Murase]

2023年12月10日(日) 15:58 Koichi Murase :
> 2023年12月10日(日) 14:13 Martin D Kealey :
> > But even if you still thought this was worth doing, it wasn't necessary to
> > make $((10#)) completely illegal: Bash could look ahead and only intervene
> > if the following character is '-' (affecting $((10#-digits)) but not
> > $((10#))).
>
> I'm not a big fan of `10#[-+]digits' and invalidating `10#' either
> since the Bash change also broke mine [1,2], but I'd say the above
> lookahead switching is worse than the current behavior. [...]

I checked the behaviors of different shells because I was interested
in them.  They seem to vary more than I expected.

  | $((10#)) | $((2*10#-1)) | $((2*10#- 1))
--|--|--|---
bash-5.0  | 0| -1   | -1
bash-5.1  | error| error| error
ksh93u| 0| -2   | -1
ksh93v| error| error| error
mksh  | 0| -1   | -1
zsh   | 0| -2   | error

ksh93u is tested with "Version AJM 93u+m/1.0.3 2022-08-25" and
"Version AJM 93u+ 2012-08-01". ksh93v is tested with "Version A
2020.0.0" (strictly speaking, this is not 93v but developed based on
93v).  $((2*10# -1)) was the same as $((2*10#-1)). $((2*10# - 1)) was
the same as $((2*10#- 1)).

bash-5.0 behaves in the same way as mksh. bash-5.1 behaves in the same
way as ksh93v. ksh93u and zsh behave in a similar way, but zsh
produces a syntax error with `10#- 1'.  I think the behavior of zsh is
similar to the lookahead switching suggested by Martin.  ksh93u also
performs a kind of lookahead switching, but it's different from what
Martin suggests.

I guess bash-5.1 referenced the behavior of ksh93v, but this is
actually not an actively maintained branch.  Recently, ksh under an
active development seems to be ksh93u+m derived from ksh93u+.

--
Koichi

Re: $((expr)) allows the hexadecimal constant "0x"

[Koichi Murase]

2023年12月11日(月) 23:37 Chet Ramey :
> On 12/10/23 1:58 AM, Koichi Murase wrote:
> > It's too late, but a better way could have been introducing the
> > function-call syntax and supporting signed n-adic numbers through e.g.
> > « int([-+]digits, n) ».
>
> You're 40 years too late. The [base#]number syntax was in ksh-83.

Sorry for my confusing writing, but I meant that instead of dropping
the support for `10#' in Bash 5.1, we could have supported something
like « int([-+]digits, n) » or « strtol([-+]digits, n) ». Maybe I
misunderstand it, but AFAIU, `10#' was dropped in the following
thread, which discusses `10#$X' where e.g. X='-10':

https://lists.gnu.org/archive/html/bug-bash/2019-06/threads.html#00039

I thought the motivation for dropping the support for `10#' was to
make `10#-10' unambiguous when ksh's treatment of « 10#[-+]digits »
would be added to Bash in the future. However, if « int([-+]digits, n)
» is added instead, « 10#[-+]digits » does not need to be added to
Bash, the ambiguity in « 10#[-+]digits » does not arise, and `10#' can
continue to mean 0 for backward compatibility.

Anyway, it's too late; `10#' is already removed.

--
Koichi

Re: $((expr)) allows the hexadecimal constant "0x"

[Koichi Murase]

2023年12月10日(日) 14:13 Martin D Kealey :
> But even if you still thought this was worth doing, it wasn't necessary to
> make $((10#)) completely illegal: Bash could look ahead and only intervene
> if the following character is '-' (affecting $((10#-digits)) but not
> $((10#))).

I'm not a big fan of `10#[-+]digits' and invalidating `10#' either
since the Bash change also broke mine [1,2], but I'd say the above
lookahead switching is worse than the current behavior. In the above
way, there is still a backward compatibility problem with
$((10#$X-digits)) with an empty X. One might suggest that the
arithmetic lexer can reference the original string before the shell
expansions, but the arithmetic evaluation does not work in that way.
For example, there is the same problem with « let "10#$X-digits" »
where the shell expansion and the arithmetic evaluation are clearly
separated.

It's too late, but a better way could have been introducing the
function-call syntax and supporting signed n-adic numbers through e.g.
« int([-+]digits, n) ».

[1] 
https://github.com/akinomyoga/ble.sh/commit/2ea48d7cd4fe2a808b20e3c4c81f662f5a5f5451
[2] 
https://github.com/akinomyoga/ble.sh/commit/7545ea31bda2e34e25b53c6550270397ccc77361

Re: bash should consult .config/bash/...

[Koichi Murase]

2023年12月10日(日) 13:23 Oğuz :
> Trends come and go. If there is demand for this feature bash should do it
> by honoring an environment variable like BASH_HOME when looking for rc
> files instead of exploring a fixed path.

In that case, a question would be what would be the standard way to
specify the Bash-specific variable, BASH_HOME. We cannot set it in
~/.bashrc because it should be specified before Bash decides the
location of .bashrc.

- Maybe the package managers of distributions can specify the
  configure option -DSYS_BASHRC and set BASH_HOME there. But, in that
  case, ~/${XDG_CONFIG_HOME:-~/.config/bash}/bashrc could be directly
  sourced from SYS_BASHRC, which is already possible with the current
  Bash.  Another problem is that if it is not set up by the package or
  the system administrators, the user cannot use it.

- Besides, Zsh seems to support ZDOTDIR, which can be used to specify
  the location of user configurations.  The Zsh users seem to specify
  ZDOTDIR in /etc/zshenv or /etc/zsh/zshenv. This approach has the
  same problem that a user needs to ask a system administrator to
  configure /etc/zshenv as the user likes.

By the way, even if we forget about `.config/bash', it may be more
consistent to have a variable like the suggested one. There are
already similar variables, ENV and BASH_ENV. However, the former is
only used in the interactive posix mode, and the latter is only used
in the non-interactive non-posix mode.  There does not seem to be a
similar variable that can be used in the interactive non-posix mode
and the non-interactive posix mode.

- interactive posix -> $ENV
- interactive non-posix -> N/A (hardcoded to be ~/.bashrc)
- non-interactive posix -> N/A
- non-interactive non-posix -> $BASH_ENV

--
Koichi

Re: bash should consult .config/bash/...

[Koichi Murase]

2023年12月10日(日) 12:10 jidanni :
> Maybe on the list of config files bash looks at,
> there should be also .config/bash/... as that is

There was a discussion:

https://lists.gnu.org/archive/html/bug-bash/2021-05/threads.html#00024
https://lists.gnu.org/archive/html/bug-bash/2021-11/threads.html#00036

Re: "read -s -n999 -t0.01 key" hangs until keypress

[Koichi Murase]

2023年10月18日(水) 0:01 Chet Ramey :
> He claims to have tested on bash-5.2. I just can't reproduce it.

Oh..., I somehow thought they were both 5.1 (5.1.16 and 5.1.15). I
didn't notice my misreading even when I extracted the quoted part.
Thanks for pointing it out.

--
Koichi

Re: "read -s -n999 -t0.01 key" hangs until keypress

[Koichi Murase]

2023年10月12日(木) 19:18 Bodo Thiesen :
> Tested versions:
> bash-5.1.16 with readline 8.1.2
> bash-5.2.15 with readline 8.2.1

You are using Bash 5.1, but I think it should have been already fixed
in Bash 5.2. There were similar reports [1-3] (you may compare the
test cases found there with yours):

[1] https://lists.gnu.org/archive/html/bug-bash/2020-10/msg00095.html
[2] https://lists.gnu.org/archive/html/bug-bash/2021-02/msg00059.html
[3] https://lists.gnu.org/archive/html/bug-bash/2021-02/msg00084.html

Bash 5.1 and 5.2 use totally different implementations for the
timeouts. The (default) implementation has been switched by [2] from
SIGALARM to select/pselect in Bash 5.2.


Koichi

Re: 5.3: job error caused by funsub + bind + loadable builtin

[Koichi Murase]

2023年10月2日(月) 23:15 Chet Ramey :
> On 9/15/23 12:47 AM, Koichi Murase wrote:
> > Description:
> >
> >DEBUG warnings are produced for inconsistent numbers of dead jobs
> >when a function substitution is used with the loadable builtin
> >`sleep' inside `bind -x'.  I guess this is related to the
> >implementation of function substitutions newly introduced in 5.3,
> >but I have not tried to pin down the problem in the codebase.
>
> Thanks, this is harmless. The child forked to run the async function call
> needed to reset the number of dead jobs in the jobs list after clearing it.

Thank you for taking the time to look into that.

--
Koichi

5.3: job error caused by funsub + bind + loadable builtin

[Koichi Murase]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -march=native -O3
uname output: Linux chatoyancy 6.4.10-100.fc37.x86_64 #1 SMP
PREEMPT_DYNAMIC Fri Aug 11 15:18:39 UTC 2023 x86_64 x86_64 x86_64
GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.3
Patch Level: 0
Release Status: devel

Description:

  DEBUG warnings are produced for inconsistent numbers of dead jobs
  when a function substitution is used with the loadable builtin
  `sleep' inside `bind -x'.  I guess this is related to the
  implementation of function substitutions newly introduced in 5.3,
  but I have not tried to pin down the problem in the codebase.

  Note: It does not seem to be reflected in the above auto-generated
  "Configuration Information", but I specified `--with-bash-malloc=no'
  as a ./configure option.

Repeat-By:

  $ PS1='\$ ' LANG=C bash-dev --norc
  $ enable -f /path/to/lib/bash/sleep sleep
  $ function f1 { for a in 1; do builtin sleep 0.1; done }
  $ builtin bind -x '"\C-t": v=${ f1 & }'
  $ <--- press  several times
  bash-dev: DEBUG warning: mark_dead_jobs_as_notified: ndead (0) !=
js.j_ndead (1)
  bash-dev: DEBUG warning: mark_dead_jobs_as_notified: ndead (0) !=
js.j_ndead (2)
  bash-dev: DEBUG warning: mark_dead_jobs_as_notified: ndead (0) !=
js.j_ndead (3)
  bash-dev: DEBUG warning: mark_dead_jobs_as_notified: ndead (0) !=
js.j_ndead (4)
  bash-dev: DEBUG warning: mark_dead_jobs_as_notified: ndead (0) !=
js.j_ndead (5)

  The value of `js.j_ndead' increases unlimitedly as typing C-t.

--
Koichi

Re: nofork command substitution

[Koichi Murase]

2023年5月25日(木) 21:13 Robert Elz :
>   | Does that mean ${(command)} would be explicitly unsupported even
>   | though {(command)} is allowed for the brace grouping?
>
> The two are mostly unrelated.

Yes, they are independent, but there is still a "choice" to make the
new grammar similar to the existing one for the Principle Of Least
Astonishment or like that. I admit that what would be the POLA design
could depend on the person, so we can discuss it.

> What Chet is planning on removing is allowing the 'C' in the opening ${C
> that indicates that this is a command substitution, and not some other kind
> of expansion, will no longer have the case where C=='('.

Ah, OK. This comment solved my question. I was misunderstanding it as
if the trailing semicolon would become mandatory.

Re: nofork command substitution

[Koichi Murase]

2023年5月24日(水) 22:27 Chet Ramey :
> > [...]
>
> Thanks for testing it.

Thank you for all the explanations and clarifications.

> Both are valid, but don't get too fond of ${(command)} -- I'm going to
> remove it in favor of people using ${ (command); } if they want a subshell.

Does that mean ${(command)} would be explicitly unsupported even
though {(command)} is allowed for the brace grouping? Would other
cases without semicolons, such as ${ if ...; then ...; fi }, also be
going to be removed?

> > Then, can I understand the grammar in the following way?  First, there
> > are two types of nofork command substitutions:
> >
> >${ compound_list }
> >${| compound_list }
> >
> > where `compound_list' is what is defined by EBNF in POSIX XCU 2.10.2.
>
> You could always look at the shell grammar:
>
> [...]
>
> funsub: DOLBRACE compound_list '}'
>  {
>$$ = $2;
>  }
>  |   DOLBRACE newline_list '}'
>  {
>$$ = (COMMAND *)NULL;
>  }

OK, I haven't checked the grammar definition in the source code. I now
looked into parse.y, but it doesn't seem to be so clear to me what
would be the distinction among the variations of C = , ,
, `(', and `|' for `${C command; }' just by looking at the
definition. It seems to be processed on subst.c:6880 semantically.
Anyway, I'm satisfied now.

Thank you.

--
Koichi

Re: nofork command substitution

[Koichi Murase]

2023年5月16日(火) 2:35 Chet Ramey :
> The latest devel branch push has the initial implementation of `nofork'
> command substitution. The excerpt from the info manual describing it is
> appended.
>
> Please test it out, and find the places I missed. Thanks.

I really appreciate that the feature ${ command; } is finally
implemented.  I have a function that mimics the behavior of the nofork
command substitution in my project.  I changed the function to switch
to use the new nofork command substitutions in new Bash versions [1]
and tested it in the devel branch of Bash.  The initial push of the
nofork command substitution (commit e44e3d50) did not work at all
because of the same error as reported by Oguz [2].  Now with the
latest push (commit 782af562), it seems to work perfectly so far.

[1] 
https://github.com/akinomyoga/ble.sh/blob/0906fd959e6c3f08a63ca1eab815ed6acb9244d3/src/util.sh#L2203-L2205
[2] https://lists.gnu.org/archive/html/bug-bash/2023-05/msg00045.html



1. Question about the grammar

>If the first character following the open brace is a '(', COMMAND
> is executed in a subshell, and COMMAND must be terminated by a ')'.

I guess this is just a minor wording issue, but if I literally read
the above description, the nofork command substitution starting with
`${(' needs to have the form « ${(COMMAND);} ».  However, as far as I
test it, « ${(COMMAND)} » is also allowed.  Also, «
${(COMMAND);COMMAND;} », which does not end with `)', seems to be also
allowed as far as I test it.  I guess `${(' is actually not really
different from the other `${', `${', and `${',
but is just a version where the COMMAND starts with a subshell (...).

The description reads like there are three [ i.e., ${, ${(, and ${| ]
or five [ i.e., ${, ${, ${, ${(, and ${| ]
distinct types of nofork command substitutions.  However, in my
understanding after testing it, there are actually only two conceptual
variants of the command substitutions: one starts with '${|' and the
others.

Then, can I understand the grammar in the following way?  First, there
are two types of nofork command substitutions:

  ${ compound_list }
  ${| compound_list }

where `compound_list' is what is defined by EBNF in POSIX XCU 2.10.2.
The lookup for the ending `}' is performed in a similar way as the
brace grouping

  { compound_list }

or as specified in POSIX XCU 2.4 so that, e.g., a semicolon is needed
for a simple command such as ${ echo; }.  Of course, the semicolon is
not mandatory when it is not mandatory in the case of the brace
grouping, e.g., ${ if true; then echo true; fi } is a well-formed
nofork command substitution.  The current implementation seems to be
consistent with this understanding.

If we understand it in this way, it is natural to include  as an
introducer to the nofork command substitutions in addition to 
and  because it is the case in the brace grouping.  The
opening paren `(' is also the same.  There seems to be a suggestion to
exclude , but I think it is strange and inconsistent to exclude
.  By the way, if we would be more strictly consistent with the
grammar in the brace grouping, the delimiters `<' and `>' should also
introduce nofork command substitutions, such as `${< file.txt;}'
(which would be a synonym of `$(< file.txt)', I guess) or `${<
file.txt sed s/a/b/g;}'.



2. About the ending brace

There seems to be a suggestion to allow « } » in an arbitrary position
to terminate the nofork command substitution, but I'm actually opposed
to the suggestion even if it is different from the undocumented
behaviors of ksh and mksh.

In my thinking, the nofork command substitution can be mentally
understood as we first have a brace grouping `{ compound_list }' and
then turn it into a substitution by prefixing `$', though it might not
be the strict explanation of the grammar.  This relation is just the
same as the case for subshell `( compound_list )' and the command
substitution `$( compound_list )'.  Then, I expect that any commands
that are grammatically valid in the brace grouping are allowed in the
nofork command substitution.

If we allow `}' of « ${ echo } ... » to end the nofork command
substitution, it means that the syntax inside the nofork command
substitutions ${ ... } is slightly different from that in any other
context, i.e., we invent a variant of the shell language only valid in
the nofork command substitution.  For example, we cannot put a valid
POSIX command « echo } » inside the nofork command substitutions
without modifications such as « echo '}' ».

I prefer the current implementation for the lookup of the ending `}',
which I feel is much more consistent with the shell language.



3. ${(...)} vs $(...)

There seems to be a doubt in introducing `${( compound_list )}' as a
construct distinct from the normal command substitution `$(
compound_list )', but I do need `${( compound_list )}' because the
normal command substitution doesn't create a process group while the
subshell (...) in the nofork comman

Re: [PATCH] compgen -V option (store output in array)

[Koichi Murase]

2023年4月18日(火) 0:59 Chet Ramey :
> Once you have -V, isn't -z superfluous? It's just
>
> compgen -V varname options word
> printf '%s\0' "${varname[@]}"
>
> right? What am I missing?

You are right. Actually, when I implemented -z, I also initially
thought about the possibility to add an option to store results in a
specified array. Then, after looking at the codebase of
bash-completion and thinking about the actual use cases, I decided to
switch to the flag -z, but I actually don't remember the exact reason
I finally switched to -z. (Maybe it was because when I submit a patch
to bash-completion, it is easier to explain the code using mapfile,
which is already used in bash-completion, than using printf '%s\0',
which the maintainer might not be comfortable with).

I thought about it again and I now feel -V is better if one needs to
choose one from -V and -z.

--
Koichi

Re: [PATCH] compgen -V option (store output in array)

[Koichi Murase]

2023年4月13日(木) 17:57 Koichi Murase :
> The first patch `r0034.compgen-update-desc.patch' update the
> description of existing options in manuals before adding the
> description of the new option. The patch `r0034.compgen-zflag.patch'
> adds the option -z.

Sorry, the second patch is broken. I failed to rebase it. Please
replace it with the fixed version `r0034.compgen-zflag.v2.patch'
attached to this email.

--
Koichi


r0034.compgen-zflag.v2.patch
Description: Binary data

Re: [PATCH] compgen -V option (store output in array)

[Koichi Murase]

2023年4月13日(木) 17:31 Grisha Levit :
> Since the predominant use case for compgen is  generating output that then
> gets split back up into an array, it seems like it would be nice to have an
> option that avoids the extra steps (and their associated pitfalls)

I have related but unsubmitted patches that I made before. I attach
them. This adds a new flag option `-z` that makes the output of
`compgen' separated by NUL This is useful when we want to process the
candidates of compgen (that may contain newlines as a part of a
candidate) using external commands such as `sort -z'. Also, now we can
load a NUL-separated stream in an array by « mapfile -d '' ».
Currently, there is no way to distinguish the newlines as a part of
candidates from the newlines that are used to separate candidates. The
option name `-z' could be alternatively `-0' or any other as far as it
works.

The first patch `r0034.compgen-update-desc.patch' update the
description of existing options in manuals before adding the
description of the new option. The patch `r0034.compgen-zflag.patch'
adds the option -z.

These are related discussions in bash-completion and ohmybash:

https://github.com/scop/bash-completion/issues/704#issuecomment-1091631732
https://github.com/ohmybash/oh-my-bash/issues/341#issuecomment-1168073038
https://github.com/scop/bash-completion/pull/765#issuecomment-1173097965

I'd be glad if both `-z' and `-V' could be added in `compgen'.

--
Koichi


r0034.compgen-update-desc.patch
Description: Binary data


r0034.compgen-zflag.patch
Description: Binary data

Re: [PATCH] Work around GNU/Linux timestamp glitch

[Koichi Murase]

2023年3月24日(金) 9:25 Paul Eggert :
> Without this patch, Bash can hand out user-visible timestamps
> that are out of order, because on GNU/Linux the 'time'
> function uses a different clock than file timestamps
> and the 'gettimeofday' function.  See
> .

FYI, there was a discussion in 2020-04. It was originally reported by
Felix Hauri.
https://lists.gnu.org/archive/html/bug-bash/2020-04/msg00072.html

By the way, you should prepare patches based on the devel branch of Bash.

Re: [PATCH] Save more readline state when running compgen

[Koichi Murase]

2023年3月22日(水) 22:51 Chet Ramey :
> > I wouldn't say my usage is typical, but I do that in fact. I have a
> > shell function to do that for an arbitrary command (which I once
> > posted in help-bash [1] but now it's old. The latest is [2]) and use
> > it everywhere.
>
> So is your function suite dependent on the compgen invocations changing
> the readline completion state?

I mean the function itself does not rely on the compgen, but I call
compgen through such a function so the compgen would be executed in
the main shell. If I understand this report correctly, it changes the
state of Readline. But I'm not sure if it affects any observable
behavior in my specific setup because I re-implement the entire
completion so these `compgen's are not called from inside the readline
bindable function `complete'. Do you think calling `compgen', which
leaves the state of `compopt -o filenames', etc., outside the bindable
function `compete' would affect any behaviors?

--
Koichi

Re: [PATCH] Save more readline state when running compgen

[Koichi Murase]

2023年3月22日(水) 6:00 Greg Wooledge :
> I don't use programmable completion, but I could *imagine* someone using
> a temp file to store the results, then using mapfile to read them back
> in, to avoid the fork() that the command substitution uses.

I wouldn't say my usage is typical, but I do that in fact. I have a
shell function to do that for an arbitrary command (which I once
posted in help-bash [1] but now it's old. The latest is [2]) and use
it everywhere.

[1] https://lists.gnu.org/archive/html/help-bash/2020-05/msg00057.html
[2] https://github.com/akinomyoga/ble.sh/blob/2eadcd5b/src/util.sh#L2173-L2225

> You'd
> need a non-forking way to create the temp file, and to remove it, though,
> or you'll lose all your gains in those steps.

Yes, there is no gain for the temp file If we are going to do that
only once. But if we are going to use it repeatedly in one interactive
session or in a single execution of a Bash program, we can obtain
gains because we can create the temporary file only once at the
beginning of the session/program (of course with appropriate
permissions, etc.). Of course, the temporary file should be cleaned up
later by using the EXIT trap or another way.

[3] https://github.com/akinomyoga/ble.sh/blob/2eadcd5b/ble.pp#L1346-L1486
(creation of a tempdir)
[4] https://github.com/akinomyoga/ble.sh/blob/2eadcd5b/ble.pp#L1501-L1551
(cleanup)



But I also sometimes utilize `compgen' for purposes unrelated to
completions. For example, when I want to check the list of Bash
versions installed in my system, I would directly run

$ compgen -c bash

in the command line. Another usage is to filter array elements by «
compgen -W '"${arr[@]}"' -X '...' » (assuming the array elements do
not contain newlines), which can be combined with the temporary-file
the approach mentioned above.

--
Koichi

Re: Behaviour of an empty conditional expression is undefined?

[Koichi Murase]

2023年2月23日(木) 17:58 Qingsheng Huang :
> The variable $A has never been defined.
> Double brackets surrounding 1) a few spaces or 2) an empty variable even
> quoted behave illogically.  I've also try different lengths of spaces, and
> also an empty variable without quotes, the same illogical.
>
> I am curious if I am the first guy encounter such a situation.

This is already reported [1] and fixed in devel [2].

[1] https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00098.html
[2] 
https://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel&id=6711d2dc7793ed3d35e91b2bd4ecbdfccfcf469f

Re: On the behaviour of the checkwinsize shopt from within a subshell

[Koichi Murase]

2023年2月22日(水) 4:25 Chet Ramey :
> I think we could probably allow it during programmable completion, too. I
> don't know if it will provide any user benefit, though.

I see. Maybe it is more consistent, and the code would be simpler.

I'm not sure either if there is really a use case for checkwinsize in
the programmable-completion context, but I recall that Fzf introduces
a TUI selection menu triggered by completions for a certain word like
**, for which the terminal window size is needed (but Fzf is
implemented in a different language, Go lang).

--
Koichi

Re: On the behaviour of the checkwinsize shopt from within a subshell

[Koichi Murase]

2023年2月21日(火) 20:28 Koichi Murase :
> I recently noticed that `checkwinsize' stopped working in the trap
> string and the `bind -x' context from Bash 5.2, which broke my script.

The behavioral change in the trap handlers and `bind -x' seems to have
been introduced in the following commit.

https://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel&id=b6a567e7f13406952cbb1d1adb2f00b2260a871e

This is the related report.

https://lists.gnu.org/archive/html/bug-bash/2022-03/msg00018.html

This is the related change at that time:

> diff --git a/CWRU/CWRU.chlog b/CWRU/CWRU.chlog
> index ef39921f..f86f0cbe 100644
> --- a/CWRU/CWRU.chlog
> +++ b/CWRU/CWRU.chlog
> @@ -3341,3 +3341,13 @@ parse.y
> `make visible' flag or through sh_strvis if we're not running the
> prompt string through word expansions. Fixes issue reported by
> Josh Harcome  back in mid-January
> +
> +  3/11
> +  
> +jobs.c
> + - wait_for: don't call get_tty_state() if readline is dispatching
> +   (RL_STATE_DISPATCHING) with the terminal settings changed
> +   (RL_STATE_TERMPREPPED), the same way we don't if we are running a
> +   command for programmable completion. Fixes bug with SIGINT reverting
> +   to the saved readline terminal settings reported by
> +   Markus Napierkowski 
> diff --git a/jobs.c b/jobs.c
> index 25289f4a..77d9dc35 100644
> --- a/jobs.c
> +++ b/jobs.c
> @@ -3117,7 +3117,7 @@ if (job == NO_JOB)
> else
>  #if defined (READLINE)
>   /* We don't want to do this if we are running a process during
> -programmable completion. */
> - if (RL_ISSTATE (RL_STATE_COMPLETING) == 0)
> +programmable completion or a command bound to `bind -x'. */
> + if (RL_ISSTATE 
> (RL_STATE_COMPLETING|RL_STATE_DISPATCHING|RL_STATE_TERMPREPPED) == 0)
>  #endif
>   get_tty_state ();

`checkwinsize' had been processed in `get_tty_state ()' in bash-5.1
and before.  As explained in the change log, we do not want to call
`get_tty_state ()' in this context, but we can instead directly call
`get_new_window_size ()' to only process `checkwinsize'.  I attach a
possible patch [r0039.checkwinsizeA.patch.txt] for this.  Or another
option might be to directly call `ioctl (fd, TIOCGWINSZ, &ws)' there
[r0039.checkwinsizeB.patch.txt].

Note: The patch just tries to fix the regression in 5.2 but does not
change the behavior within subshells.
--
Koichi
From be2cfd1b30b1a78b6a55a6b283b6f9c57250314e Mon Sep 17 00:00:00 2001
From: Koichi Murase 
Date: Tue, 21 Feb 2023 21:21:13 +0900
Subject: [PATCH 1/2] process checkwinsize in trap handlers and "bind -x"

---
 jobs.c | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/jobs.c b/jobs.c
index d6552a3e..0526c189 100644
--- a/jobs.c
+++ b/jobs.c
@@ -3058,8 +3058,17 @@ if (job == NO_JOB)
  else
 #if defined (READLINE)
/* We don't want to do this if we are running a process during
-  programmable completion or a command bound to `bind -x'. */
-   if (RL_ISSTATE 
(RL_STATE_COMPLETING|RL_STATE_DISPATCHING|RL_STATE_TERMPREPPED) == 0)
+  programmable completion. */
+   if (RL_ISSTATE (RL_STATE_COMPLETING) == 0)
+ if (RL_ISSTATE (RL_STATE_DISPATCHING|RL_STATE_TERMPREPPED) != 0)
+   {
+ /* If we are running a process during a trap handler or a
+command bound to `bind -x', we do not call `get_tty_state'
+but only update the window size. */
+ if (check_window_size)
+   get_new_window_size (0, (int *)0, (int *)0);
+   }
+ else
 #endif
get_tty_state ();
 
-- 
2.39.0

From e17a482c75df53dc4b6b7208404d75324e92cab0 Mon Sep 17 00:00:00 2001
From: Koichi Murase 
Date: Tue, 21 Feb 2023 21:21:13 +0900
Subject: [PATCH 1/2] process checkwinsize in trap handlers and "bind -x"
 (option B)

---
 jobs.c | 22 --
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/jobs.c b/jobs.c
index d6552a3e..e77ee065 100644
--- a/jobs.c
+++ b/jobs.c
@@ -3058,8 +3058,26 @@ if (job == NO_JOB)
  else
 #if defined (READLINE)
/* We don't want to do this if we are running a process during
-  programmable completion or a command bound to `bind -x'. */
-   if (RL_ISSTATE 
(RL_STATE_COMPLETING|RL_STATE_DISPATCHING|RL_STATE_TERMPREPPED) == 0)
+  programmable completion. */
+   if (RL_ISSTATE (RL_STATE_COMPLETING) == 0)
+ if (RL_ISSTATE (RL_STATE_DISPATCHING|RL_STATE_TERMPREPPED) != 0)
+   {
+ /* If we are running a process during a trap handler or a
+command bound to `bind -x', we do 

Re: On the behaviour of the checkwinsize shopt from within a subshell

[Koichi Murase]

2023年2月21日(火) 18:19 Kerin Millar :
> In this case, the values of the variables were not updated from within the 
> subshell, only the initial shell. I consider this to be surprising because 
> the manual does not mention any such limitation. Consequently, the utility of 
> the checkwinsize feature suffers in terms of being a potential alternative to 
> the invocation of stty(1) with the "size" operand (an operand which will 
> eventually be standard, but isn't yet).

I recently noticed that `checkwinsize' stopped working in the trap
string and the `bind -x' context from Bash 5.2, which broke my script.
Here's an example:

$ bash-5.1 --norc -O checkwinsize
$ trap 'echo $COLUMNS:$LINES; sleep 3; /bin/true; echo $COLUMNS:$LINES' WINCH
$ 215:45/* <-- change the terminal size twice. Note: the latter
WINCH received while processing the first WINCH is discarded. */
239:75/* <-- after /bin/true, it is updated. */

exit
$ bash-5.2 --norc -O checkwinsize
$ trap 'echo $COLUMNS:$LINES; sleep 3; /bin/true; echo $COLUMNS:$LINES' WINCH
$ 215:45/* <-- change the terminal size twice. */
215:45/* <-- now it's not updated after /bin/true. */

exit

It seems the behavior in the subshells didn't change in Bash 5.2
(i.e., checkwinsize didn't work in the subshells in 5.1 either), but I
also hope that `checkwinsize' would be changed to work in any context
(including subshells, trap handlers, bind -x, etc.).

--
Koichi

Re: unset does not remove functions like a[b] unless -f is specified

[Koichi Murase]

2023年2月7日(火) 0:59 Koichi Murase :
> By the way, I got a reply from the chair, Andrew, yesterday and could
> get my account in the Austing Group Issue Tracker. I'll later submit
> an issue about the implementation extension of function names with
> slashes.

Sorry for the delay about an issue in Austin Group Issue Tracker. I
haven't submitted it yet. I'll try to find time this weekend.

P.S. The history in [1] is also interesting. Thanks.

[1] https://lists.gnu.org/archive/html/bug-bash/2023-02/msg00109.html