Re: Trailing newlines disappear
On 14/06/2560 03:52, Chet Ramey wrote: > You mean command substitution cutting off the input it reads at the > occurrence of a NUL? What I am really after is a shell option for command substitution not discarding trailing newlines. Peter
Re: Trailing newlines disappear
On 6/12/17 8:55 PM, Peter & Kelly Passchier wrote: > On 13/06/2560 02:54, Chet Ramey wrote: >> If you want to effectively change it to a newline, specify NUL as the >> line delimiter using the -d option (new in bash-4.4). > > Thanks, that sounds like a clean solution! > I only reverted to mapfile because $(...) command substitution could not > be made to work. (I still have to 'convert' the array to a normal > variable in the next step...). Would a shell option for command > substitution using NUL be a desirable feature to bash users?? You mean command substitution cutting off the input it reads at the occurrence of a NUL? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Trailing newlines disappear
On 13/06/2560 02:54, Chet Ramey wrote: > If you want to effectively change it to a newline, specify NUL as the > line delimiter using the -d option (new in bash-4.4). Thanks, that sounds like a clean solution! I only reverted to mapfile because $(...) command substitution could not be made to work. (I still have to 'convert' the array to a normal variable in the next step...). Would a shell option for command substitution using NUL be a desirable feature to bash users?? Peter
Re: Trailing newlines disappear
On 6/9/17 1:40 PM, Peter & Kelly Passchier wrote: > On 09/06/2560 23:38, L A Walsh wrote: >> Chet Ramey wrote: >>> >>> Should mapfile silently drop the NULs? >> >> Maybe add a flag to ignore NUL bytes that could be used in the 'read' >> statement as well? If not specified, keep same behavior? > > That sounds like it might be useful. > It might be more desirable to change it to a newline instead of dropping > it? (Or both, with different flags??) If you want to effectively change it to a newline, specify NUL as the line delimiter using the -d option (new in bash-4.4). -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 08:58:05PM +0300, Pierre Gaston wrote: > Even if they don't realize it, few people would expect: > > var=$(wc -l file);echo "$var" > > to print 2 lines. imadev:~$ file=$'a\nb\nc\nd\ne'; touch "$file" imadev:~$ var=$(wc -l "$file"); echo "$var" 0 a b c d e > Trailing newlines are often not that interesting. But embedded newlines certainly are! (Really, though, I know what you meant. Command substitution strips trailing newlines because that's how it has always worked, and because 95% of the time, that's what you want.)
Re: Trailing newlines disappear
On Fri, Jun 9, 2017 at 8:40 PM, Peter & Kelly Passchier < peterke...@passchier.net> wrote: > On 09/06/2560 23:38, L A Walsh wrote: > > Chet Ramey wrote: > >> > >> Should mapfile silently drop the NULs? > > > > Maybe add a flag to ignore NUL bytes that could be used in the 'read' > > statement as well? If not specified, keep same behavior? > > That sounds like it might be useful. > It might be more desirable to change it to a newline instead of dropping > it? (Or both, with different flags??) > I feel this kind of magic behavior would result in hackish scripts or fill a somewhat rare niche at best. I'd rather have bash to fully handle arrays of byte, or nothing. > And how about a shell option to not omit trailing newlines in command > substitutions?? I find that very undesirable and unnecessary behaviour. > > Peter > > I would argue that dropping the trailing newlines is what most use and want. Even if they don't realize it, few people would expect: var=$(wc -l file);echo "$var" to print 2 lines. Trailing newlines are often not that interesting.
Re: Trailing newlines disappear
On 09/06/2560 23:38, L A Walsh wrote: > Chet Ramey wrote: >> >> Should mapfile silently drop the NULs? > > Maybe add a flag to ignore NUL bytes that could be used in the 'read' > statement as well? If not specified, keep same behavior? That sounds like it might be useful. It might be more desirable to change it to a newline instead of dropping it? (Or both, with different flags??) And how about a shell option to not omit trailing newlines in command substitutions?? I find that very undesirable and unnecessary behaviour. Peter
Re: Trailing newlines disappear
On 09/06/2560 19:06, Greg Wooledge wrote:
> imadev:~$ a=$(printf 'foo\0bar\nbaz\nquux\n'; printf x) a=${a%x}
> bash: warning: command substitution: ignored null byte in input
> imadev:~$ declare -p a
> declare -- a="foobar
> baz
> quux
> "
>
> imadev:~$ IFS= read -rd '' a < <(printf 'foo\0bar\nbaz\nquux\n')
> imadev:~$ declare -p a
> declare -- a="foo"
>
> imadev:~$ unset a
> imadev:~$ mapfile -t a < <(printf 'foo\0bar\nbaz\nquux\n')
> imadev:~$ declare -p a
> declare -a a=([0]="foo" [1]="baz" [2]="quux")
That is good to realize. I guess languages that strive for improvements
and yet backward comparibility over many years just end op being arcane!
Peter
Re: Trailing newlines disappear
Chet Ramey wrote: Should mapfile silently drop the NULs? Maybe add a flag to ignore NUL bytes that could be used in the 'read' statement as well? If not specified, keep same behavior?
Re: Trailing newlines disappear
On 6/9/17 9:36 AM, Greg Wooledge wrote: > If anything, the fact that $() drops NUL bytes is the surprising outlier. > But it's definitely too late to change that. Remember how many people > bitched and complained when 4.4 added a warning? How could I forget? But the complaining came because they wanted the NUL- dropping behavior (which had been in place forever) and objected to the noise. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 09:22:58AM -0400, Chet Ramey wrote: > On 6/9/17 8:06 AM, Greg Wooledge wrote: > > imadev:~$ mapfile -t a < <(printf 'foo\0bar\nbaz\nquux\n') > > imadev:~$ declare -p a > > declare -a a=([0]="foo" [1]="baz" [2]="quux") > > There's no mystery here. Mapfile reads lines delimited by newlines -- > read(2) is happy to do that job. Shell variable values can't contain NULs; > they never have been able to. The entire line read gets stored as the > value, but since C strings are NULL-terminated, you just see the contents > preceding the first NUL. Should mapfile silently drop the NULs? I'm not requesting any changes. I'm simply trying to show all of the existing behaviors, and specifically the three different results that one gets by using three different techniques to read a file into a variable. (Either you get the entire file with all NULs removed; or you get the file with each line truncated at the first NUL within that line; or you get the file only up to the first NUL.) That way, people can choose the appropriate technique for the result they want. If anything, the fact that $() drops NUL bytes is the surprising outlier. But it's definitely too late to change that. Remember how many people bitched and complained when 4.4 added a warning?
Re: Trailing newlines disappear
On 6/9/17 8:06 AM, Greg Wooledge wrote: > Actually, it looks like the mapfile variant also has its own... let's > call it "idiosyncratic" NUL handling: > > imadev:~$ unset a > imadev:~$ mapfile -t a < <(printf 'foo\0bar\nbaz\nquux\n') > imadev:~$ declare -p a > declare -a a=([0]="foo" [1]="baz" [2]="quux") > > Where did "bar" go? Looks like mapfile loses the contents after a NUL > byte within each line, but then subsequent lines are fine. (E.g. what > you would get if you blindly called strcpy() for each line, with the > source buffer possibly having multiple NULs.) There's no mystery here. Mapfile reads lines delimited by newlines -- read(2) is happy to do that job. Shell variable values can't contain NULs; they never have been able to. The entire line read gets stored as the value, but since C strings are NULL-terminated, you just see the contents preceding the first NUL. Should mapfile silently drop the NULs? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 08:06:28AM +0700, Peter & Kelly Passchier wrote:
> On 09/06/2560 05:26, Eduardo Bustamante wrote:
> > What's wrong with:
> >
> > IFS= read -rd '' foo < "$file"
>
> I think that's the winner!
So long as you know the file doesn't contain any NUL bytes. The other
variants will silently or loudly strip them out, while this one stops
reading at the first one it sees.
imadev:~$ unset a
imadev:~$ a=$(printf 'foo\0bar\nbaz\nquux\n'; printf x) a=${a%x}
bash: warning: command substitution: ignored null byte in input
imadev:~$ declare -p a
declare -- a="foobar
baz
quux
"
imadev:~$ unset a
imadev:~$ IFS= read -rd '' a < <(printf 'foo\0bar\nbaz\nquux\n')
imadev:~$ declare -p a
declare -- a="foo"
Actually, it looks like the mapfile variant also has its own... let's
call it "idiosyncratic" NUL handling:
imadev:~$ unset a
imadev:~$ mapfile -t a < <(printf 'foo\0bar\nbaz\nquux\n')
imadev:~$ declare -p a
declare -a a=([0]="foo" [1]="baz" [2]="quux")
Where did "bar" go? Looks like mapfile loses the contents after a NUL
byte within each line, but then subsequent lines are fine. (E.g. what
you would get if you blindly called strcpy() for each line, with the
source buffer possibly having multiple NULs.)
Re: Trailing newlines disappear
On 09/06/2560 05:26, Eduardo Bustamante wrote: > What's wrong with: > > IFS= read -rd '' foo < "$file" I think that's the winner!
Re: Trailing newlines disappear
On Thu, Jun 8, 2017 at 3:56 PM, Greg Wooledge wrote:
> On Thu, Jun 08, 2017 at 10:44:29PM +0200, Geir Hauge wrote:
>> You can pick one of these instead:
>>
>> mapfile < "$file"; IFS= foo="${MAPFILE[*]}"; unset -v IFS
>>
>> or
>>
>> mapfile < "$file"; printf -v foo %s "${MAPFILE[@]}"
>
> or
>
> lambda() { local MAPFILE IFS=; mapfile < "$file"; foo="${MAPFILE[*]}"; }
> lambda
>
What's wrong with:
IFS= read -rd '' foo < "$file"
Re: Trailing newlines disappear
On Thu, Jun 08, 2017 at 10:44:29PM +0200, Geir Hauge wrote:
> You can pick one of these instead:
>
> mapfile < "$file"; IFS= foo="${MAPFILE[*]}"; unset -v IFS
>
> or
>
> mapfile < "$file"; printf -v foo %s "${MAPFILE[@]}"
or
lambda() { local MAPFILE IFS=; mapfile < "$file"; foo="${MAPFILE[*]}"; }
lambda
Re: Trailing newlines disappear
On 09/06/2560 03:44, Geir Hauge wrote:
> Greg already pointed out that this doesn't work.
>
> You can pick one of these instead:
>
> mapfile < "$file"; IFS= foo="${MAPFILE[*]}"; unset -v IFS
>
> or
>
> mapfile < "$file"; printf -v foo %s "${MAPFILE[@]}"
>
Yes, thanks.
The second one looked interesting, but didn't work. I now have:
mapfile <"$file"; IFS= foo="${MAPFILE[*]}"
(without the unset, as it's in a function where it's harmless)
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 03:34:39AM +0700, PePa wrote:
> On 09/06/2560 02:14, Greg Wooledge wrote:
> > Well, it leaves IFS changed, because you didn't revert or unset it,
> > or run the entire thing in a function with local IFS. Also, I'd use
> > "${MAPFILE[*]}" to reinforce that you are joining an array into a string,
> > rather than expanding the array as a list.
>
> Thanks for pointing all this out. I settled on:
> mapfile <"$file"; IFS= foo=${MAPFILE[*]} true
Greg already pointed out that this doesn't work.
You can pick one of these instead:
mapfile < "$file"; IFS= foo="${MAPFILE[*]}"; unset -v IFS
or
mapfile < "$file"; printf -v foo %s "${MAPFILE[@]}"
--
Geir Hauge
Re: Trailing newlines disappear
On 09/06/2560 02:14, Greg Wooledge wrote:
> Well, it leaves IFS changed, because you didn't revert or unset it,
> or run the entire thing in a function with local IFS. Also, I'd use
> "${MAPFILE[*]}" to reinforce that you are joining an array into a string,
> rather than expanding the array as a list.
Thanks for pointing all this out. I settled on:
mapfile <"$file"; IFS= foo=${MAPFILE[*]} true
Again, much faster than a subshell. Are there order guarantees in the
case of "a=x b=y command"??
Peter
Re: Trailing newlines disappear
ARGH! You dropped the list address! This was on bug-bash, right??
On Fri, Jun 09, 2017 at 03:14:10AM +0700, PePa wrote:
> On 09/06/2560 02:14, Greg Wooledge wrote:
> > Well, it leaves IFS changed, because you didn't revert or unset it,
> > or run the entire thing in a function with local IFS. Also, I'd use
> > "${MAPFILE[*]}" to reinforce that you are joining an array into a string,
> > rather than expanding the array as a list.
>
> Thanks for pointing all this out. I settled on:
> mapfile <"$file"; IFS= foo=${MAPFILE[*]} true
>
> Again, much faster than a subshell. Are there order guarantees in the
> case of "a=x b=y command"??
>
> Peter
You have a massive error here. Your variables exist only for the
duration of that "true" command.
imadev:~$ unset x y
imadev:~$ x=1 y=3 true
imadev:~$ declare -p x y
bash: declare: x: not found
bash: declare: y: not found
In the case of "x=1 y=2", yes, there is a guarantee of processing order.
You may safely write things like:
tmp=${x##*[} tmp=${tmp%%]*} tmp=${tmp//x/y}
And they will be done from left to right. But in your example, the
question is moot, because your variables don't survive at all.
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 01:52:44AM +0700, Peter & Kelly Passchier wrote:
> On 09/06/2560 00:42, Greg Wooledge wrote:
> > foo=$(cat "$file"; printf x) foo=${foo%x}
>
> The workaround I came up with is:
> mapfile <"$file"; IFS= foo=${MAPFILE[@]}
>
> This seems to be faster, but it probably has other disadvantages...
Well, it leaves IFS changed, because you didn't revert or unset it,
or run the entire thing in a function with local IFS. Also, I'd use
"${MAPFILE[*]}" to reinforce that you are joining an array into a string,
rather than expanding the array as a list. Semantically, x=$@ is pretty
"huh??" to me.
It also leaves the array in memory, and requires bash 4. Those may not
be significant.
Re: Trailing newlines disappear
On 09/06/2560 00:42, Greg Wooledge wrote:
> It's not a bug. This is how command substitution has worked since
> the original Bourne shell.
>
> The workaround is to put something inside the command substitution,
> so that the newlines aren't trailing any more, and then strip it away
> afterward:
>
> foo=$(cat "$file"; printf x) foo=${foo%x}
Thanks for the reply.
Seeing the hackish workaround, I would call it an ossified (or codified)
flaw...
The workaround I came up with is:
mapfile <"$file"; IFS= foo=${MAPFILE[@]}
This seems to be faster, but it probably has other disadvantages...
Peter
Re: Trailing newlines disappear
On Fri, Jun 09, 2017 at 12:38:19AM +0700, PePa wrote:
> I would think this is a bug:
> 4.3.48(1)-release> printf "q\n\n\n" >w
> 4.3.48(1)-release> printf "$(cat w)"
> q
> 4.3.48(1)-release>
>
> Is there some workaround to somehow preserve the trailing newlines?
It's not a bug. This is how command substitution has worked since
the original Bourne shell.
The workaround is to put something inside the command substitution,
so that the newlines aren't trailing any more, and then strip it away
afterward:
foo=$(cat "$file"; printf x) foo=${foo%x}
This also means that you can't use the $(< "$file") shortcut any more,
if you need to preserve the full contents of a file.
Trailing newlines disappear
I would think this is a bug: 4.3.48(1)-release> printf "q\n\n\n" >w 4.3.48(1)-release> cat w q 4.3.48(1)-release> printf "$(cat w)" q 4.3.48(1)-release> Is there some workaround to somehow preserve the trailing newlines? Peter
