Re: \! and \# in PS1 vs PS2 vs PS4, PS0 and ${var@P}

[Grisha Levit]

On Sat, Mar 18, 2017 at 3:21 PM, Chet Ramey  wrote:
> How would you suggest capturing this in the documentation in a way that
> would be clearer?  The text from your previous message or something else?

I'm finding it hard to come up with a concise and accurate description
of the behavior since there's kind of a lot going on..

I took a stab at instead changing the behavior to make \! and \#
consistent for all of a command's prompt strings (patch attached);
perhaps this is a reasonable approach?


consistent-hist-and-cmnd-nums-in-prompts.patch
Description: Binary data

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Chet Ramey]

On 3/21/17 4:27 PM, Peter & Kelly Passchier wrote:
> On 22/03/2560 01:32, Martijn Dekker wrote:
>> A workaround for the original poster's problem could be:
>>
>>  (unset -f jobs; unalias jobs; eval 'jobs -l') | wc
> 
> 'unset -f jobs' is not guaranteed to unset jobs, it might set jobs to a
> function definition in an earlier scope..!

There are no local functions.  All functions exist at the global scope.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Peter & Kelly Passchier]

On 22/03/2560 01:32, Martijn Dekker wrote:
> A workaround for the original poster's problem could be:
> 
>   (unset -f jobs; unalias jobs; eval 'jobs -l') | wc

'unset -f jobs' is not guaranteed to unset jobs, it might set jobs to a
function definition in an earlier scope..!

Re: optimized_assignment and READLINE_LINE

[Chet Ramey]

On 3/21/17 11:51 AM, Grisha Levit wrote:
> The new optimized assignment in the devel branch that is used when
> appending to a string has a bug with the way READLINE_LINE is checked
> in bash_execute_unix_command:

Thanks for the report.  Please keep looking for other instances of this.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: fc -l history specification out of range with empty history

[Grisha Levit]

On Tue, Mar 21, 2017 at 11:59 AM, Chet Ramey  wrote:
>> As a special case, `fc -l -0' usually prints itself but not in the
>> empty history (or HISTSIZE=1) case.
>
> Yeah, that's a special case, but not when there are no commands in
> the history list.

Thanks for the explanation. FWIW my less-degenerate use case was
getting the just-entered command in PS0 but `history 1' should
probably do just as well in this case.

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Chet Ramey]

On 3/21/17 2:38 PM, Greg Wooledge wrote:
> On Tue, Mar 21, 2017 at 07:32:27PM +0100, Martijn Dekker wrote:
>> A workaround for the original poster's problem could be:
>>
>>  (unset -f jobs; unalias jobs; eval 'jobs -l') | wc
>>
>> The 'eval' is to stop the alias from being expanded at parse time before
>> you have a chance to unalias it. This should be about as robust as
>> running 'builtin jobs -l', except it still doesn't check whether the
>> builtin might have been disabled.
> 
> Does this still trigger the "jobs hack"?  Given that the shell is allowed
> to use simple lexical analysis to determine whether we're just running
> one jobs (or trap) command in the subshell, my first guess would *not* be
> that this would work.  (Not least because we are definitely *not* just
> running one jobs command; there are at least three commands.)

It doesn't, but the text Eric quoted doesn't really apply here anyway,
since it only mentions command substitution.  Arbitrary subshells'
behavior is up to the implementation, and bash does what it does only
to allow the output of jobs to be piped (e.g., `(jobs)' doesn't work
the same way).

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Greg Wooledge]

On Tue, Mar 21, 2017 at 07:32:27PM +0100, Martijn Dekker wrote:
> A workaround for the original poster's problem could be:
> 
>   (unset -f jobs; unalias jobs; eval 'jobs -l') | wc
> 
> The 'eval' is to stop the alias from being expanded at parse time before
> you have a chance to unalias it. This should be about as robust as
> running 'builtin jobs -l', except it still doesn't check whether the
> builtin might have been disabled.

Does this still trigger the "jobs hack"?  Given that the shell is allowed
to use simple lexical analysis to determine whether we're just running
one jobs (or trap) command in the subshell, my first guess would *not* be
that this would work.  (Not least because we are definitely *not* just
running one jobs command; there are at least three commands.)

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Martijn Dekker]

Op 21-03-17 om 18:08 schreef Greg Wooledge:
> A builtin is always used by preference over an external command of the
> same name.  You don't need to specify "builtin jobs" to be sure you're
> using the builtin.  Just use "jobs".
> 
> The only time you need to use the "builtin" command is when you're
> defining a function by the same name, and you want bash to use its
> builtin instead of your function.

You may also need to use it if you're writing a function or script used
by another program you don't control, or in another user's interactive
shell environment. In that case you have no way to be sure whether
'jobs' might already be used as an alias or function name.

A workaround for the original poster's problem could be:

(unset -f jobs; unalias jobs; eval 'jobs -l') | wc

The 'eval' is to stop the alias from being expanded at parse time before
you have a chance to unalias it. This should be about as robust as
running 'builtin jobs -l', except it still doesn't check whether the
builtin might have been disabled.

- M.

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Eric Blake]

On 03/21/2017 12:08 PM, Greg Wooledge wrote:

> The code that makes bash behave differently when "jobs" is one of the
> commands in a pipeline/subshell is kind of a hack.  It's probably not
> extremely well known outside of this mailing list, but I suspect many
> people have used it without realizing what it is.  It's a fairly intuitive
> hack.

And it's for more than just 'jobs' - at least 'trap' relies on it, too.

> 
> I've got no strong opinions about whether the "jobs hack" should be
> documented.

POSIX documents the 'jobs'/'trap' hack thus:

When a subshell is entered, traps that are not being ignored shall be
set to the default actions,
except in the case of a command substitution containing only a single
trap command, when the
traps need not be altered. Implementations may check for this case using
only lexical analysis;
for example, if `trap` and $( trap -- ) do not alter the traps in the
subshell, cases such as
assigning var=trap and then using $($var) may still alter them.

For more background on this, see:
http://austingroupbugs.net/view.php?id=53

although that bug specifically mentions 'trap', and concludes that a
separate bug may need to be opened (which I could not find) giving the
same treatment to 'jobs'.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Hengyang Zhao]

Hi Greg,

Thanks for your explanation! Now it solved my problem. I did misuse the
"builtin" builtin when using "jobs". Now I only need to enforce that "jobs"
is "enabled", which is not a walkaround :-)

Except one thing that bothers me a tiny bit is that, how about I wrote a
"jobs" function and still want to pipe "builtin jobs" somewhere? But I
believe I won't make such a use in the near future. Anyways, let me read
the code first :-)

Thanks for your time!


Sincerely,

Hengyang

On Tue, Mar 21, 2017 at 10:08 AM Greg Wooledge  wrote:

> On Tue, Mar 21, 2017 at 04:53:59PM +, Hengyang Zhao wrote:
> > But back to the user's perspective, as I looked up "help jobs" or "help
> > builtin", the sematics of "builtin" is only for forcing the shell to use
> > the builtin version, right? Actually, I was writing a script that needs
> to
> > secure the use of the builtin jobs, but now I need to seek for a reliable
> > walkaround instead of using "builtin".
>
> A builtin is always used by preference over an external command of the
> same name.  You don't need to specify "builtin jobs" to be sure you're
> using the builtin.  Just use "jobs".
>
> The only time you need to use the "builtin" command is when you're
> defining a function by the same name, and you want bash to use its
> builtin instead of your function.  In practice, this should be quite rare.
> You'd really only use it if you are creating a wrapper function.  For
> example:
>
> cd() {
> local where=${1-$HOME}
> # set xterm title bar
> printf '\e]2;%s\a' "$where"
> builtin cd "$where"
> }
>
> The "builtin cd" at the end prevents bash from recursively calling the
> function.
>
> > So if we don't treat it as a bug, is
> > it still a good suggestion that we write a caveat info the "builtin" help
> > info?
>
> The code that makes bash behave differently when "jobs" is one of the
> commands in a pipeline/subshell is kind of a hack.  It's probably not
> extremely well known outside of this mailing list, but I suspect many
> people have used it without realizing what it is.  It's a fairly intuitive
> hack.
>
> I've got no strong opinions about whether the "jobs hack" should be
> documented.  I don't think the "builtin" command needs any further
> explanation, though.  The "help builtin" text already contains a terse
> version of the explanation I gave up above.
>
-- 
Hengyang Zhao

Ph.D. Candidate, Electrical Engineering
University of California, Riverside

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Greg Wooledge]

On Tue, Mar 21, 2017 at 04:53:59PM +, Hengyang Zhao wrote:
> But back to the user's perspective, as I looked up "help jobs" or "help
> builtin", the sematics of "builtin" is only for forcing the shell to use
> the builtin version, right? Actually, I was writing a script that needs to
> secure the use of the builtin jobs, but now I need to seek for a reliable
> walkaround instead of using "builtin".

A builtin is always used by preference over an external command of the
same name.  You don't need to specify "builtin jobs" to be sure you're
using the builtin.  Just use "jobs".

The only time you need to use the "builtin" command is when you're
defining a function by the same name, and you want bash to use its
builtin instead of your function.  In practice, this should be quite rare.
You'd really only use it if you are creating a wrapper function.  For
example:

cd() {
local where=${1-$HOME}
# set xterm title bar
printf '\e]2;%s\a' "$where"
builtin cd "$where"
}

The "builtin cd" at the end prevents bash from recursively calling the
function.

> So if we don't treat it as a bug, is
> it still a good suggestion that we write a caveat info the "builtin" help
> info?

The code that makes bash behave differently when "jobs" is one of the
commands in a pipeline/subshell is kind of a hack.  It's probably not
extremely well known outside of this mailing list, but I suspect many
people have used it without realizing what it is.  It's a fairly intuitive
hack.

I've got no strong opinions about whether the "jobs hack" should be
documented.  I don't think the "builtin" command needs any further
explanation, though.  The "help builtin" text already contains a terse
version of the explanation I gave up above.

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Hengyang Zhao]

Dear Chet,

Thanks for your reply and I apologyze for the mistake I made in the "buggy
behavior" section.

I didn't realize that a builtin is forced to run in a subshell until you
told me. Yes, as I tried, "jobs" consumes no PID, but even "jobs | true"
consumes two. I'll look into the code following the clue you gave me.

But back to the user's perspective, as I looked up "help jobs" or "help
builtin", the sematics of "builtin" is only for forcing the shell to use
the builtin version, right? Actually, I was writing a script that needs to
secure the use of the builtin jobs, but now I need to seek for a reliable
walkaround instead of using "builtin". So if we don't treat it as a bug, is
it still a good suggestion that we write a caveat info the "builtin" help
info?

Thanks again for your detailed reply! It helps me a lot :-)



Sincerely,

Hengyang

On Tue, Mar 21, 2017 at 8:13 AM Chet Ramey  wrote:

> On 3/20/17 7:47 PM, Hengyang Zhao wrote:
> > Hello,
> >
> > I am experiencing an unexpected behavior when using "builtin jobs -l". It
> > seems that it's output fd is not as same as "jobs -l" without the builtin
> > prefix. So when I piped it into wc, I got different results (as seen in
> the
> > code I pasted below).
> >
> > Bash version: GNU bash, version 4.3.43(1)-release
> (x86_64-redhat-linux-gnu)
> > OS: Fedora 25, kernel 4.9.13-200.fc25.x86_64
> >
> > Buggy behavior:
> > when we have one or more background processes:
> > jobs -l | wc# produces 0 0 0
> > builtin -l | wc  # produces non-zero non-zero non-zero
>
> (This is actually the opposite of what happens, as shown by your examples
> below.)
>
> > I tried to fix the bug, but after I looked at the code, the jobs printing
> > procedure clearly prints to stdout. So I think it's not a easy work to
> > track down the bug.
>
> It's not a bug.  The first part of a pipeline is always run in a subshell.
> That subshell doesn't really have any jobs, since none of the parent's
> jobs are children of that shell -- you can't wait for them, for instance.
>
> There is special-case code that attempts to detect when the `jobs' builtin
> is running (execute_cmd.c:execute_subshell_builtin_or_function) and doesn't
> remove the jobs from the jobs table in that one case, specifically to allow
> the output of `jobs' to be piped.  Running the `builtin' builtin defeats
> that.
>
> Chet
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.edu
> http://cnswww.cns.cwru.edu/~chet/
>
-- 
Hengyang Zhao

Ph.D. Candidate, Electrical Engineering
University of California, Riverside

Re: fc -l history specification out of range with empty history

[Chet Ramey]

On 3/21/17 11:35 AM, Grisha Levit wrote:
> If the history is empty (or if HISTSIZE is 1) running `fc -l' produces an 
> error:
> 
> $ history -c
> $ fc -l
> bash: fc: history specification out of range

fc deals with the commands in the history list before the `fc' (that's
what the `previous command' in the Posix description means; it's equivalent
to -1).  So saying fc -l, which is equivalent to fc -l -16 (or
fc -l -16 -1), is out of range.  Even fc -l -1 is out of range.

ksh93 and mksh both print (different) errors; dash doesn't.

> I'm not sure this is necessary, since other instances of `first' and
> `last' being outside of the saved history range don't produce an
> error.  

Posix says explicitly that they don't when there are commands in the
history list, but you have to specify them.

Also the `history' command doesn't have this behavior:
> 
> $ history -c
> $ history
> 1  history

Because `history' doesn't skip to the previous command.

> As a special case, `fc -l -0' usually prints itself but not in the
> empty history (or HISTSIZE=1) case.

Yeah, that's a special case, but not when there are no commands in
the history list.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

optimized_assignment and READLINE_LINE

[Grisha Levit]

The new optimized assignment in the devel branch that is used when
appending to a string has a bug with the way READLINE_LINE is checked
in bash_execute_unix_command:

v = bind_variable ("READLINE_LINE", rl_line_buffer, 0);
l = v ? value_cell (v) : 0;

# parse_and_execute ...

v = find_variable ("READLINE_LINE");
l1 = v ? value_cell (v) : 0;
if (l1 != l)
maybe_make_readline_line (value_cell (v));

For example:

$ bind -x '"\C-g": f'
$ f() { READLINE_LINE+=x; }
$ ^G
(nothing happens)

Not sure if any other places have the same issue; I couldn't
immediately find any.

Re: "unset var" pops var off variable stack instead of unsetting it

[Stephane Chazelas]

2017-03-20 16:32:10 -0400, Chet Ramey:
[...]
> > See also:
> > 
> > $ bash -c 'f() { unset a; echo "$a";}; a=1; a=2 f'
> > 1
> > 
> > already mentioned.
> 
> A distinction without a difference; the behavior is explicitly the same.
[...]

One I haven't mentioned yet is:

$ bash -c 'f() { local a; unset a; echo "$a";}; a=1; a=2 f'
1

IOW, the work around I was mentioning earlier (of using "local"
before "unset" to make sure "unset" unsets) doesn't work in that
case. You'd need to use the same work around as for mksh/yash
(call unset in a loop until the variable is really unset (with
the nasty side effect of unsetting the variable in a scope
you're need meant to tamper with) so you'd want to do it in a
subshell).

-- 
Stephane

Re: lockup in bgp_delete()

[Luke Shumaker]

On Mon, 20 Mar 2017 18:29:07 -0400,
Eduardo Bustamante wrote:
> 
> This was reported a month ago:
> http://lists.gnu.org/archive/html/bug-bash/2017-02/msg00025.html

Oh! Thank you, I tried searching the list archives, but nothing came
up.  Odd; it shows up in searches now.

-- 
Happy hacking,
~ Luke Shumaker

fc -l history specification out of range with empty history

[Grisha Levit]

If the history is empty (or if HISTSIZE is 1) running `fc -l' produces an error:

$ history -c
$ fc -l
bash: fc: history specification out of range

I'm not sure this is necessary, since other instances of `first' and
`last' being outside of the saved history range don't produce an
error.  Also the `history' command doesn't have this behavior:

$ history -c
$ history
1  history

As a special case, `fc -l -0' usually prints itself but not in the
empty history (or HISTSIZE=1) case.

Re: Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Chet Ramey]

On 3/20/17 7:47 PM, Hengyang Zhao wrote:
> Hello,
> 
> I am experiencing an unexpected behavior when using "builtin jobs -l". It
> seems that it's output fd is not as same as "jobs -l" without the builtin
> prefix. So when I piped it into wc, I got different results (as seen in the
> code I pasted below).
> 
> Bash version: GNU bash, version 4.3.43(1)-release (x86_64-redhat-linux-gnu)
> OS: Fedora 25, kernel 4.9.13-200.fc25.x86_64
> 
> Buggy behavior:
> when we have one or more background processes:
> jobs -l | wc# produces 0 0 0
> builtin -l | wc  # produces non-zero non-zero non-zero

(This is actually the opposite of what happens, as shown by your examples
below.)

> I tried to fix the bug, but after I looked at the code, the jobs printing
> procedure clearly prints to stdout. So I think it's not a easy work to
> track down the bug.

It's not a bug.  The first part of a pipeline is always run in a subshell.
That subshell doesn't really have any jobs, since none of the parent's
jobs are children of that shell -- you can't wait for them, for instance.

There is special-case code that attempts to detect when the `jobs' builtin
is running (execute_cmd.c:execute_subshell_builtin_or_function) and doesn't
remove the jobs from the jobs table in that one case, specifically to allow
the output of `jobs' to be piped.  Running the `builtin' builtin defeats
that.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Bug: different behavior between "jobs -l" and "builtin jobs -l"

[Hengyang Zhao]

Hello,

I am experiencing an unexpected behavior when using "builtin jobs -l". It
seems that it's output fd is not as same as "jobs -l" without the builtin
prefix. So when I piped it into wc, I got different results (as seen in the
code I pasted below).

Bash version: GNU bash, version 4.3.43(1)-release (x86_64-redhat-linux-gnu)
OS: Fedora 25, kernel 4.9.13-200.fc25.x86_64

Buggy behavior:
when we have one or more background processes:
jobs -l | wc# produces 0 0 0
builtin -l | wc  # produces non-zero non-zero non-zero


I tried to fix the bug, but after I looked at the code, the jobs printing
procedure clearly prints to stdout. So I think it's not a easy work to
track down the bug.


Way to reproduce:

bash-4.3$
bash-4.3$ enable -a | grep jobs
enable jobs
bash-4.3$ cat
^Z
[1]+  Stopped cat
bash-4.3$ jobs -l
[1]+ 14030 Stopped cat
bash-4.3$ jobs -l | wc
  1   4  39
bash-4.3$ builtin jobs -l | wc
  0   0   0
bash-4.3$ type jobs
jobs is a shell builtin
bash-4.3$ type -a jobs
jobs is a shell builtin
jobs is /usr/bin/jobs
jobs is /bin/jobs
bash-4.3$

Thank you!

Sincerely,
Hengyang
-- 
Hengyang Zhao

Ph.D. Candidate, Electrical Engineering
University of California, Riverside

Re: arithmetic syntax error loop after declaration of exported integer variable

[Chet Ramey]

On 3/20/17 6:52 PM, D630 wrote:
> I get an infinite error loop in B, and I suppose that's not because of my
> settings.

It's not.  This will be fixed in the next devel branch push.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: "unset var" pops var off variable stack instead of unsetting it

[Dan Douglas]

On 03/18/2017 12:19 PM, Chet Ramey wrote:
> On 3/17/17 6:35 PM, Dan Douglas wrote:
> 
>> The problem is the non-obvious nature of unset's interaction with scope,
>> (and the lack of documentation). Not much can be done about the former,
>> as it is with so many things.
> 
> How would you suggest improving the documentation? I can see highlighting
> the fact that unset applied to a local variable at the same scope
> preserves the local attribute. What else?
> 

The effect of unset on a local was what I had in mind, but really the
manual says very little about scope. All it says right now is:

"Variables local to the function may be declared with the local builtin
command.  Ordinarily, variables and their values are shared between the
function and its caller."

Which doesn't exactly describe dynamic scope even for those that know
what that means.

Also not documented is how a variable declared with declare/typeset is
distinct from an unset variable.