Re: IFS field splitting doesn't conform with POSIX

[Kerin Millar]

On Sun, 02 Apr 2023 09:32:20 +0700
Robert Elz  wrote:

> Date:Sat, 1 Apr 2023 19:44:10 -0400
> From:Saint Michael 
> Message-ID:  
> 
> 
>   | The compelling reason is: I may not know how many values are stored in the
>   | comma-separated list.
> 
> Others have told you you're wrong, but this is not any kind of compelling
> reason - you simply give one more variable name than you expected to need
> (than you would have used otherwise) and then all the extra fields that
> you wanted the shell to ignore will be assigned to it - which you are free
> to ignore if you like, or you can test to see if anything is there, and
> issue an error message (or something) if more fields were given than you
> were expecting.   Much better behaviour than the shell simply ignoring
> data (silently).

I would add to this that bash affords one the luxury of using read -a before 
proceeding to check the size of the resulting array.

-- 
Kerin Millar

Re: IFS field splitting doesn't conform with POSIX

[Robert Elz]

Date:Sat, 1 Apr 2023 18:49:56 -0600
From:Felipe Contreras 
Message-ID:  


  | Fortunately kre did listen.

Not really.I agree that what POSIX currently says is not correct,
which is why the defect report got filed (you may have noticed that there
was no new wording proposed there - and still isn't - which is because
this is very hard to get correct, other than possibly by simply giving
the code that should be executed (no, that won't happen)).

But the others are correct, POSIX (in general) standardises what shells
actually do - you can see this if you read a few pages, all kinds of
things lead to unspecified (or worse, undefined) behaviour.   That's
because different implementations do different things in those cases.
Not because some specific behaviour could not be required, not even
that doing so might not be better all around.   But implementations
don't do the same thing in those cases, and so users cannot rely upon
anything particular happening (sometimes behaviour is unspecified,
but only between a limited number of choices).

The standard has two purposes - one is to allow application writers
(users) work out what they can expect to work, and what they should not
do if they expect code to be portable.   The other is so implementors
of new implementations (of the shell, or anything else included)
know what to implement (and where they can do things differently).

You're right, when the standard uses "shall" it is being prescriptive,
and implementations must do that if they want to claim to conform.
But the standard only does that when the existing (at least major, and
intending to conform) implementations, at the time the standard is
written, actually do what is proposed to be required by a "shall".

There are odd occasions (such as the read errors in scripts) where something
that (almost all) implementations do is so obviously the wrong thing to
do, that the standard requires implementations to change, but if you
looked at that issue, and I believe you did, that was only done after
checking with implementors to see if they were willing to make the
change.

In this case, the standard will certainly end up saying that IFS
characters (both white space and others - there are differences in
how they work, but not in this regard) terminate fields, and
if there is nothing after the final IFS character (or characters,
in the case of IFS whitespace), then there is no additional field,
and if there is something there, then that makes an additional field,
even if there is no IFS terminator following it.   That's because that's
what all (or essentially all) shells do, and always (for almost 45
years now) have done so.

That is, if we have "IFS=," then both a,b,c and a,b,c,
produce 3 fields "a" "b" and "c".

On the other hand, the standard is likely to say that whether
characters other than space/tab/newline which are white space according
to the definition of that term in the standard, can be IFS white
space, is unspecified - because shell implementations are split
about that (about 60/40 for "no" - even though the standard currently
seems to say "yes").   That is unless shell implementers can be persuaded
to change their implementations, which in this case is probably unlikely
(as no-one can be sure that there aren't scripts around which rely
upon their current behaviour - no-one wants to break backward compat).
The effect will probably be that using any white space char in IFS, other
than the blessed 3, will make a script non-portable (might work with one
shell, and not another).

kre

Re: IFS field splitting doesn't conform with POSIX

[Robert Elz]

Date:Sat, 1 Apr 2023 19:44:10 -0400
From:Saint Michael 
Message-ID:  


  | The compelling reason is: I may not know how many values are stored in the
  | comma-separated list.

Others have told you you're wrong, but this is not any kind of compelling
reason - you simply give one more variable name than you expected to need
(than you would have used otherwise) and then all the extra fields that
you wanted the shell to ignore will be assigned to it - which you are free
to ignore if you like, or you can test to see if anything is there, and
issue an error message (or something) if more fields were given than you
were expecting.   Much better behaviour than the shell simply ignoring
data (silently).

  | GNU AWK, for instance, acts responsibly in the same exact situation:
  | line="a,b,c,d";awk -F, '{print $1}' <<< $line
  | a

awk is a different language with different rules, used in a different way.
Further, it isn't all that different really - you're only using $1, but
awk doesn't simply discard the other fields, they're there, called $2 $3 ...
There's even NF which tells you how many are there.

kre

[PATCH] fix bcopy params

[Grisha Levit]

---
 lib/sh/oslib.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/sh/oslib.c b/lib/sh/oslib.c
index ab7924df..85f7b491 100644 --- a/lib/sh/oslib.c
+++ b/lib/sh/oslib.c @@ -161,7 +161,7 @@ getdtablesize (void) # undef bcopy
 # endif void
-bcopy (void *s, *d, size_t n)
+bcopy (void *s, void *d, size_t n) {
   FASTCOPY (s, d, n); }
--
2.40.0

Re: IFS field splitting doesn't conform with POSIX

[Lawrence Velázquez]

On Sat, Apr 1, 2023, at 9:27 PM, Kerin Millar wrote:
> On Sat, 1 Apr 2023 19:44:10 -0400
> Saint Michael  wrote:
>
>> There is an additional problem with IFS and the command read
>> 
>> Suppose I have variable  $line with a string "a,b,c,d"
>> IFS=',' read -r x1 <<< $line
>> Bash will assign the whole line to x1
>>  echo $x1
>> line="a,b,c,d";IFS=',' read -r x1 <<< $line;echo $x1;
>> a,b,c,d
>> but if I use two variables
>> line="a,b,c,d";IFS=',' read -r x1 x2 <<< $line;echo "$x1 ---> $x2";
>> a ---> b,c,d
>> this is incorrect. If IFS=",", then a read -r statement must assign the
>
> No it isn't.
>
>> first value to the single variable, and disregard the rest.
>
> No it musn't. Read 
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html 
> and pay particular attention to the definition of what must happen 
> where there are fewer vars (names) than fields encountered.

Also, observe the behavior of other shells:

% cat foo.sh
echo a,b,c,d | {
IFS=, read x1
printf '%s\n' "$x1"
}

echo a,b,c,d | {
IFS=, read x1 x2
printf '%s ---> %s\n' "$x1" "$x2"
}

% bash foo.sh
a,b,c,d
a ---> b,c,d
% dash foo.sh
a,b,c,d
a ---> b,c,d
% ksh foo.sh
a,b,c,d
a ---> b,c,d
% mksh foo.sh
a,b,c,d
a ---> b,c,d
% yash foo.sh
a,b,c,d
a ---> b,c,d
% zsh foo.sh
a,b,c,d
a ---> b,c,d

And the Heirloom Bourne shell:

b# echo a,b,c,d | { IFS=, read x1; printf '%s\n' "$x1"; 
}; echo a,b,c,d | { IFS=, read x1 x2; printf '%s ---> %s\n' "$x1" "$x2"; }
 a,b,c,d
 a ---> b,c,d

-- 
vq

Re: IFS field splitting doesn't conform with POSIX

[Kerin Millar]

On Sat, 1 Apr 2023 19:44:10 -0400
Saint Michael  wrote:

> There is an additional problem with IFS and the command read
> 
> Suppose I have variable  $line with a string "a,b,c,d"
> IFS=',' read -r x1 <<< $line
> Bash will assign the whole line to x1
>  echo $x1
> line="a,b,c,d";IFS=',' read -r x1 <<< $line;echo $x1;
> a,b,c,d
> but if I use two variables
> line="a,b,c,d";IFS=',' read -r x1 x2 <<< $line;echo "$x1 ---> $x2";
> a ---> b,c,d
> this is incorrect. If IFS=",", then a read -r statement must assign the

No it isn't.

> first value to the single variable, and disregard the rest.

No it musn't. Read 
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html and pay 
particular attention to the definition of what must happen where there are 
fewer vars (names) than fields encountered.

-- 
Kerin Millar

Re: IFS field splitting doesn't conform with POSIX

[Lawrence Velázquez]

> On Apr 1, 2023, at 8:49 PM, Felipe Contreras  
> wrote:
> 
> On Sat, Apr 1, 2023 at 6:35 PM Lawrence Velázquez  wrote:
>> 
>> On Sat, Apr 1, 2023, at 8:02 PM, Felipe Contreras wrote:
>>> In that example they are discussing whether or not to make that
>>> behavior a *requirement*. That is prescriptive.
>> 
>> You're so busy pretending this is debate club that you're completely
>> missing everyone's point, which is that the Austin Group by and
>> large aims to standardize existing behavior.
> 
> I did not miss your point, you are missing mine


Begin forwarded message:

> From: Emanuele Torre 
> Subject: Re: IFS field splitting doesn't conform with POSIX
> Date: March 30, 2023 at 1:48:54 PM EDT
> To: Felipe Contreras 
> Cc: bug-bash@gnu.org
> 
> On Thu, Mar 30, 2023 at 11:35:08AM -0600, Felipe Contreras wrote:
>>> How can you say that the current implementation that bash, dash, etc.
>>> use is not compliant to the POSIX specification?
>> 
>> I have never said that.
> 
> The title of this thread is "IFS field splitting doesn't conform with
> POSIX".


-- 
vq

Re: IFS field splitting doesn't conform with POSIX

[Felipe Contreras]

On Sat, Apr 1, 2023 at 6:35 PM Lawrence Velázquez  wrote:
>
> On Sat, Apr 1, 2023, at 8:02 PM, Felipe Contreras wrote:
> > In that example they are discussing whether or not to make that
> > behavior a *requirement*. That is prescriptive.
>
> You're so busy pretending this is debate club that you're completely
> missing everyone's point, which is that the Austin Group by and
> large aims to standardize existing behavior.

I did not miss your point, you are missing mine, which you are clearly
not interested in listening to.

Fortunately kre did listen.

Cheers.

-- 
Felipe Contreras

Re: IFS field splitting doesn't conform with POSIX

[Lawrence Velázquez]

On Sat, Apr 1, 2023, at 8:02 PM, Felipe Contreras wrote:
> In that example they are discussing whether or not to make that
> behavior a *requirement*. That is prescriptive.

You're so busy pretending this is debate club that you're completely
missing everyone's point, which is that the Austin Group by and
large aims to standardize existing behavior.  If a situation arises
where many implementations are not conformant, then the standard
is flawed, and it is wrongheaded to blame the implementations.

-- 
vq

Re: IFS field splitting doesn't conform with POSIX

[Felipe Contreras]

On Thu, Mar 30, 2023 at 1:20 PM Lawrence Velázquez  wrote:
> On Thu, Mar 30, 2023, at 2:25 PM, Felipe Contreras wrote:

> > The challenge is in deciding what they *should* do, which is not
> > descriptive, but prescriptive.
>
> The Austin Group does not see its role as prescriptive, although
> during discussions implementers are often open to modifying their
> implementations to achieve consensus.  If many implementers agree
> to make a change, the result may appear prescriptive.  (A recent
> example is .)

In that example they are discussing whether or not to make that
behavior a *requirement*. That is prescriptive.

> >> If what it says differs from what the majority of shells do, then it's
> >> POSIX that is wrong.
> >
> > Then there is no point in looking at the standard, since we know what
> > it should say
>
> The standard is a reference that describes a set of broadly common
> behaviors.  Not everyone is interested in researching and testing
> an assortment of implementations whenever they want to determine
> whether a behavior is portable.
>
> (Also: bash, dash, ksh, and zsh are not the only shells that exist.)

Precisely because they are not the only shells that exist, an
agreement between current implementers--which they themselves might
see as descriptive of their implementations--results in text that says
"the shell shall", which is prescriptive.

If I write a new shell (which I am seriously considering) which aims
to be called POSIX-compatible, that "shall" is 100% prescriptive.

-- 
Felipe Contreras

Re: IFS field splitting doesn't conform with POSIX

[Greg Wooledge]

On Sat, Apr 01, 2023 at 07:44:10PM -0400, Saint Michael wrote:
> There is an additional problem with IFS and the command read
> 
> Suppose I have variable  $line with a string "a,b,c,d"
> IFS=',' read -r x1 <<< $line
[...]

https://mywiki.wooledge.org/BashPitfalls#pf47

Re: IFS field splitting doesn't conform with POSIX

[Saint Michael]

There is an additional problem with IFS and the command read

Suppose I have variable  $line with a string "a,b,c,d"
IFS=',' read -r x1 <<< $line
Bash will assign the whole line to x1
 echo $x1
line="a,b,c,d";IFS=',' read -r x1 <<< $line;echo $x1;
a,b,c,d
but if I use two variables
line="a,b,c,d";IFS=',' read -r x1 x2 <<< $line;echo "$x1 ---> $x2";
a ---> b,c,d
this is incorrect. If IFS=",", then a read -r statement must assign the
first value to the single variable, and disregard the rest.
and so on, with (n) variables.
The compelling reason is: I may not know how many values are stored in the
comma-separated list.
GNU AWK, for instance, acts responsibly in the same exact situation:
line="a,b,c,d";awk -F, '{print $1}' <<< $line
a
We need to fix this





On Sat, Apr 1, 2023, 6:11 PM Mike Jonkmans  wrote:

> On Sat, Apr 01, 2023 at 03:27:47PM -0400, Lawrence Velázquez wrote:
> > On Fri, Mar 31, 2023, at 2:10 PM, Chet Ramey wrote:
> > > kre filed an interpretation request to get the language cleaned up.
> >
> > For those who might be interested:
> >
> > https://austingroupbugs.net/view.php?id=1649
>
> Thanks for the link.
>
> And well done, kre!
>
> --
> Regards, Mike Jonkmans
>
>

Re: IFS field splitting doesn't conform with POSIX

[Mike Jonkmans]

On Sat, Apr 01, 2023 at 03:27:47PM -0400, Lawrence Velázquez wrote:
> On Fri, Mar 31, 2023, at 2:10 PM, Chet Ramey wrote:
> > kre filed an interpretation request to get the language cleaned up.
> 
> For those who might be interested:
> 
> https://austingroupbugs.net/view.php?id=1649

Thanks for the link.

And well done, kre!

-- 
Regards, Mike Jonkmans

Re: IFS field splitting doesn't conform with POSIX

[Lawrence Velázquez]

On Fri, Mar 31, 2023, at 2:10 PM, Chet Ramey wrote:
> kre filed an interpretation request to get the language cleaned up.

For those who might be interested:

https://austingroupbugs.net/view.php?id=1649

-- 
vq

Failed $(

[Zev Weiss]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -g -O2
uname output: Linux hatter 5.18.19_1 #1 SMP PREEMPT_DYNAMIC Thu Aug 25 14:36:55 
UTC 2022 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu

Bash Version: 5.2
Patch Level: !PATCHLEVEL!
Release Status: release

Description:

When 'set -e' is enabled, file-content substitutions of the form $(<...) 
cause an immediate exit even when the subsitution is in a conditional 
context such as the condition of an 'if' statement or the left operand 
of the '||' operator.


I'm guessing this probably has something to do with the fork-suppression 
mentioned in the release notes for bash 5.2, since the problem seems to 
have appeared with that release (5.1.16 does not exhibit the bug).


Repeat-By:

  $ bash -e -c 'x=$(In prior releases this produced an error message for the nonexistent 
file but then continued to print 'hello'; in bash 5.2 and later it exits 
with a non-zero exit status immediately after issuing the error message 
without proceeding to the 'echo' command.


The same behavior can be seen with an 'if' statement instead:

  $ bash -e -c 'if ! x=$(