Re: Assignment of $* to a var removes spaces on unset IFS.

[Robert Elz]

Date:Mon, 13 Aug 2018 11:51:08 -0400
From:Bize Ma 
Message-ID:  


  | Bash since 4.3 fails to follow what the documentation describes as that on
  | an assignment values do not undergo splitting or globing.

This is already fixed.

What's more, reporting bugs in bash 4.3 is just a waste of everyone's time.
Upgrade to (at least) the current released version, and use that instead.

kre

Re: Assignment of $* to a var removes spaces on unset IFS.

[Eduardo Bustamante]

On Mon, Aug 13, 2018 at 5:25 PM Robert Elz  wrote:
[...]
> What's more, reporting bugs in bash 4.3 is just a waste of everyone's time.
> Upgrade to (at least) the current released version, and use that instead.

Adding to this point:

* bash 4.3 was released 4 years ago (2014-02-26)
* bash 4.4 is the current release of bash (2016-09-15)
* bash 5.0 is in alpha state (2018-05-22)

I know that some stable Linux distributions might still be
distributing 4.3, but that just means that you have to report the bug
against these distros, not upstream.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Bize Ma]

2018-08-13 21:16 GMT-04:00 Eduardo Bustamante :

> On Mon, Aug 13, 2018 at 5:25 PM Robert Elz  wrote:
> [...]
> > What's more, reporting bugs in bash 4.3 is just a waste of everyone's
> time.
> > Upgrade to (at least) the current released version, and use that instead.
>
> Adding to this point:
>
> * bash 4.3 was released 4 years ago (2014-02-26)
> * bash 4.4 is the current release of bash (2016-09-15)
> * bash 5.0 is in alpha state (2018-05-22)
>
>
You both are confused. The original table included

> b43sh   : [foo bar baz quux][  foo bar  baz quux  ]
> b44sh   : [foo bar baz quux][  foo bar  baz quux  ]

That's   4.3.48(2)-release   and4.4.19(2)-release


Both are failing.

Note that 4.4.19 is newer than what is available in
https://www.gnu.org/software/bash/

The report is valid for present releases.
The bug is still present (since 4.3).

Please, do not waste our time in incorrect claims.

Do your homework and test !


> I know that some stable Linux distributions might still be
> distributing 4.3, but that just means that you have to report the bug
> against these distros, not upstream.


That is not the problem, don't jump to conclusions.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Eduardo A . Bustamante López]

On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
(...)
> Please, do not waste our time in incorrect claims.
> 
> Do your homework and test !
(...)

Did you test Bash 5.0? Because that's where the current bug fixes are going to.
I don't see Chet releasing a new version of 4.4 to fix something that's already
fixed there.

dualbus@ubuntu:~/src/gnu/bash$ cat /tmp/script 
echo $BASH_VERSION
set -- "  foo  ""  bar  baz  "  "  quux  "
unset IFS
a=$*
b="$*"
printf '[%s]' "$a" "$b"; echo

dualbus@ubuntu:~/src/gnu/bash$ bash /tmp/script 
4.4.19(1)-release
[foo bar baz quux][  foo bar  baz quux  ]

dualbus@ubuntu:~/src/gnu/bash$ ./bash /tmp/script 
5.0.0(1)-alpha
[  foo bar  baz quux  ][  foo bar  baz quux  ]

Please read this thread:

* http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html

Re: Assignment of $* to a var removes spaces on unset IFS.

[Bize Ma]

2018-08-13 22:09 GMT-04:00 Eduardo A. Bustamante López :

> On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
> (...)
> > Please, do not waste our time in incorrect claims.
> >
> > Do your homework and test !
> (...)
>
>



> dualbus@ubuntu:~/src/gnu/bash$ bash /tmp/script
> 4.4.19(1)-release
> [foo bar baz quux][  foo bar  baz quux  ]
>
>
So, it is confirmed that the bug exists in the present release, Is it not?

> Did you test Bash 5.0? Because that's where the current bug fixes are
going to.

So, is it a "wont fix" for 4.4 (present release) ?


> I don't see Chet releasing a new version of 4.4 to fix something that's
already
> fixed there.

That version is not even beta, it is still alpha, are you asking that
everyone should use
non-released (and not yet tested as beta) alpha release ?


> dualbus@ubuntu:~/src/gnu/bash$ cat /tmp/script
> echo $BASH_VERSION
> set -- "  foo  ""  bar  baz  "  "  quux  "
> unset IFS
> a=$*
> b="$*"
> printf '[%s]' "$a" "$b"; echo

Thanks for copying the supplied script.



> dualbus@ubuntu:~/src/gnu/bash$ ./bash /tmp/script
> 5.0.0(1)-alpha
> [  foo bar  baz quux  ][  foo bar  baz quux  ]
>
>
Thanks for testing. !!


> Please read this thread:
>
> * http://lists.nongnu.org/archive/html/bug-bash/2017-09/msg00058.html
>

So? It is not solved in that thread, it is even said that:

 > This is a bug in bash and it should be fixed, not excused.

To which I agree. After a year, nothing else have been said about it.

It seems about time to get it solved. Or say that it won't be.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Eduardo A . Bustamante López]

On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote:
(...)
> That version is not even beta, it is still alpha, are you asking that
> everyone should use
> non-released (and not yet tested as beta) alpha release ?
(...)

I didn't say that. My point is that, in the context of bug reports, it's
important that you always test against the *unreleased* version of the code,
since that's where most bug fixes are queued up (or at least the ones that break
backwards compatibility).

> So? It is not solved in that thread, it is even said that:
> 
>  > This is a bug in bash and it should be fixed, not excused.
> 
> To which I agree. After a year, nothing else have been said about it.
> 

For some reason the threading broke in the archive browser. Here's the response
from Chet:

http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html

The fix for this issue is already in the Git repository (devel branch) and in
Bash 5.0. 

I cannot answer for Chet, but please consider this:

The fix for this bug breaks backwards compatibility, so that means it cannot be
a patch-level fix for 4.4. It should either be a minor version increase (4.5) or
go into the next major release, which is 5.0 (which was already declared alpha
state, and includes the fix).

> It seems about time to get it solved. Or say that it won't be.

Again, it's fixed already and the fix is scheduled to go out in the next
release (5.0).

Given that, do you think this bug is severe enough to have to issue a new minor
version just for it? (4.5),

or can we just wait for 5.0 to come out which fixes this and a bunch of other
bugs? (see the CHANGES file, in particular, entry `oo',
http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b)

Re: Assignment of $* to a var removes spaces on unset IFS.

[Robert Elz]

Date:Mon, 13 Aug 2018 21:36:23 -0400
From:Bize Ma 
Message-ID:  


  | You both are confused. The original table included

You misinterpreted my message.

I said two independant things ... first that the bug is already fixed.
I didn't say that the fixed version was released yet.

Second, that you should stop reporting bugs in 4.3 (which is
relatively ancient).   If you had reported it as a bug in 4.4
(which means that you had indicated that you had verified
it there) the second sentence would not have been in my
reply.

  | The report is valid for present releases.

That's true.   This is an insignificant issue really and not
even really worthy of a patch.   But it will be fixed in the next
release.


Eduardo A. Bustamante López  said:
  | My point is that, in the context of bug reports, it's important that you
  | always test against the *unreleased* version of the code,

(first, sorry if your name gets messed up ... it looks correct in my
mail composition window, but the message is likely to be marked
as "charset=us-ascii" or something, and the accented character
is unlikely to survive correctly...)

I think that is going too far - if a bug is in the latest released
version, it is perfectly reasonable for people to report it.

Asking people to even know where the development version is,
let alone how to fetch it, build it, etc, is a bit too much to expect.
It is nice if that can be done, but certainly should not be a
requirement.

Sometimes (for any bug report) the submitter will just be told
"already fixed" which means that a later release will have the fix
- which is the best result that could be achieved if it was not
already fixed.

   | The fix for this bug breaks backwards compatibility, 

All bug fixes break backwards compat, but only for scripts
that incorrectly assumed that the broken behaviour was
correct - such scripts are also broken. 

I mean, what would you say to someone who said "xxx always
dumped core when I did ..., and I was relying on that to detect
the error by the presence of the core file, in the latest revision
it no longer dumps core, but instead gives an error message,
please put the core dump back."

There is no reason the fix for this could not be in a 4.4 patch. 
On the other hand this is such an obscure issue that getting
the fix distributed in a hurry is also not all that important.

kre

Re: Assignment of $* to a var removes spaces on unset IFS.

[Pierre Gaston]

On Tue, Aug 14, 2018 at 6:18 AM Eduardo A. Bustamante López <
dual...@gmail.com> wrote:

> On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote:
> (...)
> > That version is not even beta, it is still alpha, are you asking that
> > everyone should use
> > non-released (and not yet tested as beta) alpha release ?
> (...)
>
> I didn't say that. My point is that, in the context of bug reports, it's
> important that you always test against the *unreleased* version of the
> code,
> since that's where most bug fixes are queued up (or at least the ones that
> break
> backwards compatibility).
>
> > So? It is not solved in that thread, it is even said that:
> >
> >  > This is a bug in bash and it should be fixed, not excused.
> >
> > To which I agree. After a year, nothing else have been said about it.
> >
>
> For some reason the threading broke in the archive browser. Here's the
> response
> from Chet:
>
> http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html
>
> The fix for this issue is already in the Git repository (devel branch) and
> in
> Bash 5.0.
>
> I cannot answer for Chet, but please consider this:
>
> The fix for this bug breaks backwards compatibility, so that means it
> cannot be
> a patch-level fix for 4.4. It should either be a minor version increase
> (4.5) or
> go into the next major release, which is 5.0 (which was already declared
> alpha
> state, and includes the fix).
>
> > It seems about time to get it solved. Or say that it won't be.
>
> Again, it's fixed already and the fix is scheduled to go out in the next
> release (5.0).
>
> Given that, do you think this bug is severe enough to have to issue a new
> minor
> version just for it? (4.5),
>
> or can we just wait for 5.0 to come out which fixes this and a bunch of
> other
> bugs? (see the CHANGES file, in particular, entry `oo',
>
> http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b
> )
>
>

Re: Assignment of $* to a var removes spaces on unset IFS.

[Greg Wooledge]

On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
> Note that 4.4.19 is newer than what is available in
> https://www.gnu.org/software/bash/

The current patch level of bash can be obtained by downloading the most
recent tarball from  and then applying the
subsequent patches from ,
or whichever directory is appropriate for the current version.

The patches for 4.4 currently go up to bash44-023, which would correspond
to bash version 4.4.23 once compiled.

I will also repeat, once more, my advice that one should NEVER write
a script containing an unquoted $* or $@ expansion.  It breaks in all
kinds of ways in more than one shell.  Just don't do it, and these
problems go away.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Bize Ma]

2018-08-13 23:17 GMT-04:00 Eduardo A. Bustamante López :

> On Mon, Aug 13, 2018 at 10:52:20PM -0400, Bize Ma wrote:
> (...) My point is that, in the context of bug reports, it's
> important that you always test against the *unreleased* version of the
> code,
>

Not if the version to test is the released one.


> (...)Here's the response from Chet:
>
> http://lists.nongnu.org/archive/html/bug-bash/2017-10/msg00023.html
>
> The fix for this issue is already in the Git repository (devel branch) and
> in
> Bash 5.0.
>

Excellent, that is good to know. There is a solution.

Chet explains it clearly.

Just that it is not applied to the present release version 4.4.
Maybe it should be put into a "stack of pending patches" that all will be
applied when a new number release (4.5) has to be done for whatever
other reason.

When that happens, we can say that it "is solved for current release".

I cannot answer for Chet, but please consider this:
>
> The fix for this bug breaks backwards compatibility


In what sense? Other shells don't have this issue and the change got
introduced in 4.3, so, this change is backward compatible with several
versions up to 4.3 (backward compatible).


> so that means it cannot be
> a patch-level fix for 4.4. It should either be a minor version increase
> (4.5) or
> go into the next major release, which is 5.0 (which was already declared
> alpha
> state, and includes the fix).
>

I am not who must make that call. Someone else must.

If this is not going to be solved in 4.4, then it is a "wontfix" for that
release.

> It seems about time to get it solved. Or say that it won't be.
>
> Again, it's fixed already


No, it is not "fixed already" on 4.4 release.
Let's call things as they are, not as anyone "hope" they can be.

and the fix is scheduled to go out in the next
> release (5.0).
>

Excellent, great, version 5.0 has one issue less to solve. Yay!!


> Given that, do you think this bug is severe enough to have to issue a new
> minor
> version just for it? (4.5),
>

Don't ask me, it is not a call that I can make, I am just a random user.


> or can we just wait for 5.0 to come out which fixes this and a bunch of
> other
> bugs? (see the CHANGES file, in particular, entry `oo',
> http://git.savannah.gnu.org/cgit/bash.git/diff/CHANGES?h=
> bash-5.0-testing&id=9a51695bed07d37086c352372ac69d0a30039a6b)
>

Yes, there it says that some changes were made to match POSIX
interpretation of $* $@.
I believe that the issue presented in the OP is one of those changes for
5.0.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Greg Wooledge]

This reply was sent to me without Cc-ing the list.  I have added the Cc.

On Tue, Aug 14, 2018 at 11:39:20PM -0400, Bize Ma wrote:
> On Tue, 14 Aug 2018 12:34:31, Greg Wooledge said:
> 
> > I will also repeat, once more, my advice that one should NEVER write
> > a script containing an unquoted $* or $@ expansion.
> 
> That is plainly INCORRECT, Greg.

You are incorrect.

> > It breaks in all kinds of ways in more than one shell.
> 
> That several shells do different things is a bug on those shells, not bash.

Agreed.  And the fact that IN REAL LIFE, THOSE SHELLS EXIST AND HAVE
THOSE BUGS, which are triggered by incorrect code, is a reason to write
code correctly so as not to trigger those bugs.

Or maybe you're one of those people who doesn't care about reality.

> > Just don't do it, and these problems go away.
> 
> If I want the split+glob to take effect I can do:
> 
>echo $*
> 
> 
> There is nothing wrong with that (don't claim that it change in different
> shells, that is a different issue than using split+glob in bash, go back to
> the point above about other shells if you wish).

There is absolutely definitely positively 100% certainly something wrong
with that.

Let's break your script, shall we?

Here's your script, except I'm going to represent it as a function.  Doing
it as a script would have the same effect.

glob() {
# "Return" (write to stdout, one per line) the expansions of all
# arguments as globs against the current working directory.
printf %s\\n $*
}

Will you at least agree that this is your intent, and a fair
representation of your proposed solution?  I'll take that as a "yes".

So now, you can pass SOME globs to it (properly quoted), and it will
appear to work for those globs:

wooledg:~$ glob '*.pdf'
0400_0001.pdf
11412687.pdf
11412859.pdf
[...]
Epic Web Service - PatientLookup.pdf
[...]

At this point, the naive script writer will say "Yay, it worked!"

The experienced script writer knows that it did not, in fact, "work".
It only "worked" for this one degenerate kindergarten-level example.

Let's test it again.

wooledg:~$ glob '*Web Service*'
*Web
Service*

But... but... but... the PREVIOUS glob worked!  Why didn't this one
work?

Because the script (function) does not ACTUALLY work.  It is broken.
You can't solve this problem using an unquoted $* expansion.  Not in
reality.

Some of us care about reality.

Other languages have no problem with this.  Tcl, for example, has a [glob]
command that takes a list of glob-patterns and returns a list of files
that match them.

wooledg:~$ tclsh
% puts [join [glob {*Web Service*}] \n]
Epic Web Service - PatientLookup.pdf
% puts [join [glob *.pdf] \n]
Infoblox_DNS_mgmt.pdf
prtest.pdf
PMCNoteCoverPage.pdf
[...]

Of course, it doesn't sort them, because I didn't call [lsort].  Adding
that is trivial.

Now, you tried to implement Tcl's [glob] in bash, but you did it naively
and the naive version does not work.  You took a shortcut.  That shortcut
falls off a cliff.

I'll leave the non-broken implementation as an exercise for the reader.

Now, back to the original points:

1) Certain shells have bugs in them.  These shells are in widespread use
   on real systems in real life.

2) One of those shells is bash, which makes it relevant to bug-bash.

3) Some of those bugs involve the unquoted expansions of $* and $@.

4) Even in the ABSENCE of such bugs, the use of an unquoted $* or $@
   expansion does not actually solve the problems you claim it solves.

5) Therefore, for TWO reasons, you should not use unquoted $* or $@
   in your shell scripts.

   Reason 1: it doesn't solve the problem.
   Reason 2: it sometimes breaks even worse due to shell bugs.

Re: Assignment of $* to a var removes spaces on unset IFS.

[Ilkka Virta]

On 15.8. 15:44, Greg Wooledge wrote:

glob() {
 # "Return" (write to stdout, one per line) the expansions of all
 # arguments as globs against the current working directory.
 printf %s\\n $*
}

But... but... but... the PREVIOUS glob worked!  Why didn't this one
work?


I'm sure you know what word splitting is.


I'll leave the non-broken implementation as an exercise for the reader.


$ glob() { local IFS=; printf '%s\n' $*; }
$ touch "foo bar.txt" "foo bar.pdf"
$ glob "foo bar*"
foo bar.pdf
foo bar.txt

(Well, you'd probably want 'nullglob' too, and there's the minor issue
that printf '%s\n'  prints at least one line even if there are no
arguments but I'll ignore that for now.)


Of course, in most cases, and unquoted expansion is not what one wants,
but if there's need to glob in the shell, then an unquoted expansion is
what has to be used. How IFS affects word splitting isn't just about
$* , the issue is the same even if you only have one glob in a regular
variable.

--
Ilkka Virta / itvi...@iki.fi

Re: Assignment of $* to a var removes spaces on unset IFS.

[Chet Ramey]

On 8/13/18 10:52 PM, Bize Ma wrote:

>  > This is a bug in bash and it should be fixed, not excused.
> 
> To which I agree. After a year, nothing else have been said about it.
> 
> It seems about time to get it solved. Or say that it won't be.

It's fixed in the current development versions, and that fix will be in
the next release. It looks like it's been fixed since April, 2017 as part
of a code cleanup to implement Posix interp 888:

http://lists.nongnu.org/archive/html/bug-bash/2017-04/msg1.html

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Assignment of $* to a var removes spaces on unset IFS.

[Chet Ramey]

On 8/14/18 8:34 AM, Greg Wooledge wrote:
> On Mon, Aug 13, 2018 at 09:36:23PM -0400, Bize Ma wrote:
>> Note that 4.4.19 is newer than what is available in
>> https://www.gnu.org/software/bash/
> 
> The current patch level of bash can be obtained by downloading the most
> recent tarball from  and then applying the
> subsequent patches from ,
> or whichever directory is appropriate for the current version.

It's always easier to download

http://git.savannah.gnu.org/cgit/bash.git/snapshot/bash-master.tar.gz

That is the head of the master branch, which is always the current
released version plus all released patches.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Assignment of $* to a var removes spaces on unset IFS.

[Bize Ma]

2018-08-15 8:44 GMT-04:00 Greg Wooledge :

> This reply was sent to me without Cc-ing the list.  I have added the Cc.
>
> On Tue, Aug 14, 2018 at 11:39:20PM -0400, Bize Ma wrote:
> > On Tue, 14 Aug 2018 12:34:31, Greg Wooledge said:
> >
> > > I will also repeat, once more, my advice that one should NEVER write
> > > a script containing an unquoted $* or $@ expansion.
> >
> > That is plainly INCORRECT, Greg.
>
> You are incorrect.


I am always incorrect, I like to be so.
But  we are discussing an issue, not my personal problems.


> > > It breaks in all kinds of ways in more than one shell.
> >
> > That several shells do different things is a bug on those shells, not
> bash.
>
> Agreed.


Excellent, we agree.


> And the fact that IN REAL LIFE, THOSE SHELLS EXIST AND HAVE
> THOSE BUGS, which are triggered by incorrect code


No, sorry, but bugs are triggered by perfectly good code, that's why they
are called bugs.


> , is a reason to write
> code correctly so as not to trigger those bugs.
>

Writing code to *work around* bugs is *not* the correct solution.
It is only a way to *perpetuate* those bugs.
The correct solution is to resolve the bugs.

But this is just a smoke curtain. There are no bugs about $@ and $* in Bash.

Or maybe you're one of those people who doesn't care about reality.
>

Sometimes I don't , most of the time when I sleep, sometimes when I day
dream.


> > > Just don't do it, and these problems go away.
> >
> > If I want the split+glob to take effect I can do:
> >
> >echo $*
> >
> >
> > There is nothing wrong with that (don't claim that it change in different
> > shells, that is a different issue than using split+glob in bash, go back
> to
> > the point above about other shells if you wish).
>
> There is absolutely definitely positively 100% certainly something wrong
> with that.
>

Ah, yes, that's correct, thanks, I should have used printf, not echo.
But you corrected it, thanks again.

Let's break your script, shall we?
>
> Here's your script, except I'm going to represent it as a function.  Doing
> it as a script would have the same effect.
>
> glob() {
> # "Return" (write to stdout, one per line) the expansions of all
> # arguments as globs against the current working directory.
> printf %s\\n $*
> }
>
> Will you at least agree that this is your intent,


No, that is *not* my intent.
You can not implement a glob function when *both: split and glob* are in
effect.

If you want a glob function, stop the split, set IFS to null:

$ cat script
#!/bin/bash
glob() ( IFS=; printf %s\\n $* )
glob '*Web Service*'

$ ./script
Epic Web Service - PatientLookup.pdf


and a fair
> representation of your proposed solution?  I'll take that as a "yes".
>

No it is not a fair representation of anything I said.
Only of a twist that you want to present here.

So now, you can pass SOME globs to it (properly quoted), and it will
> appear to work for those globs:
>

But it will fail to work on the next example as you want to demonstrate,
therefore you have not coded correctly to workaround the shells bugs
(your own words).

Some of us care about reality.
>

Yes, when I am awake.

Other languages have no problem with this.  Tcl,

(...)

But we are in bash, are we not?


Now, you tried to implement Tcl's [glob] in bash, but you did it naively
> and the naive version does not work.  You took a shortcut.  That shortcut
> falls off a cliff.
>

No, *you* did. You implemented a flawed glob function.


> I'll leave the non-broken implementation as an exercise for the reader.
>

You are not able to do it?


> Now, back to the original points:
>
> 1) Certain shells have bugs in them.  These shells are in widespread use
>on real systems in real life.
>
> 2) One of those shells is bash, which makes it relevant to bug-bash.
>
> 3) Some of those bugs involve the unquoted expansions of $* and $@.
>

Not in bash.


> 4) Even in the ABSENCE of such bugs, the use of an unquoted $* or $@
>expansion does not actually solve the problems you claim it solves.
>

I claimed nothing. I presented a valid use. Don't put words in my mouth.


> 5) Therefore, for TWO reasons, you should not use unquoted $* or $@
>in your shell scripts.
>
>Reason 1: it doesn't solve the problem.
>Reason 2: it sometimes breaks even worse due to shell bugs.
>

There is no therefore if the premises are wrong.