Re: Editing a piped in stream?

[john doe]

On 7/10/2018 2:16 PM, mick crane wrote:

On 2018-07-08 17:41, John Darrah wrote:

On 7/8/2018 7:00 AM, Richard Owlett wrote:


Until this thread I didn't have concept of "stream editor", much less 
the existence of "sed".

I had heard of "AWK", but had an image of it being a regex parser.


Awk is easy to learn because it is a minimal language. It can do
anything sed can do and much more. I am always surprised that so many
people opt to build long commands with cut/sed instead of using awk.
FWIW, I use mawk because it is orders of magnitude faster than gawk.

-- john



I'd like to learn some awk having found it handy once.
apt says mawk is installed

in /usr/bin I have
lrwxrwxrwx  1 root root 21 Apr  2 22:29 awk -> 
/etc/alternatives/awk

-rwxr-xr-x  1 root root 658072 Jan 25 12:55 gawk
-rwxr-xr-x  1 root root   3189 Jan 25 12:55 igawk
-rwxr-xr-x  1 root root 121976 Mar 23  2012 mawk
lrwxrwxrwx  1 root root 22 Apr  2 22:29 nawk -> 
/etc/alternatives/nawk


"awk" returns options for gawk and "mawk does nothing ( back to prompt)



At the end of those 'gawk' options:

"gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output."

$ printf "%s" "123 456 789" | awk '{print $2}'
456

--
John Doe

Re: Editing a piped in stream?

[mick crane]

On 2018-07-10 13:31, Greg Wooledge wrote:

On Tue, Jul 10, 2018 at 01:16:09PM +0100, mick crane wrote:

I'd like to learn some awk having found it handy once.
apt says mawk is installed

in /usr/bin I have
lrwxrwxrwx  1 root root 21 Apr  2 22:29 awk -> 
/etc/alternatives/awk

-rwxr-xr-x  1 root root 658072 Jan 25 12:55 gawk
-rwxr-xr-x  1 root root   3189 Jan 25 12:55 igawk
-rwxr-xr-x  1 root root 121976 Mar 23  2012 mawk
lrwxrwxrwx  1 root root 22 Apr  2 22:29 nawk ->
/etc/alternatives/nawk

"awk" returns options for gawk and "mawk does nothing ( back to 
prompt)


So, what's the problem?


how to run mawk ?


mawk

You have already run it once, with no arguments, and it did nothing.
Now try giving it some arguments.

wooledg:~$ mawk -v foo=bar 'BEGIN {print foo}'
bar

See, just like awk.


OK
mick
--
Key ID4BFEBB31

Re: Editing a piped in stream?

[mick crane]

On 2018-07-08 12:18, Andy Smith wrote:

Hello,

On Sun, Jul 08, 2018 at 07:39:07AM +0200, john doe wrote:

The issue here is that we don't know what the OP wants


A situation sadly familiar when dealing with this particular
poster's threads.

Also in the general case, if you ever find yourself parsing the
output of "ls", you will probably find that there is a better way to
be doing what you're trying to do. There are a lot of issues with
parsing "ls" output:




Cheers,
Andy


about 10 years ago on another mailing list there was a guy who claimed 
to be dyslexic who said was uncertain of things but once you'd 
deciphered the jumble of words was asking quite complicated networking 
questions and would come back with a knowledgeable other question ( once 
you'd deciphered it ) I did think it might be a wind up.


mick

--
Key ID4BFEBB31

Re: Editing a piped in stream?

[Greg Wooledge]

On Tue, Jul 10, 2018 at 01:16:09PM +0100, mick crane wrote:
> I'd like to learn some awk having found it handy once.
> apt says mawk is installed
> 
> in /usr/bin I have
> lrwxrwxrwx  1 root root 21 Apr  2 22:29 awk -> /etc/alternatives/awk
> -rwxr-xr-x  1 root root 658072 Jan 25 12:55 gawk
> -rwxr-xr-x  1 root root   3189 Jan 25 12:55 igawk
> -rwxr-xr-x  1 root root 121976 Mar 23  2012 mawk
> lrwxrwxrwx  1 root root 22 Apr  2 22:29 nawk ->
> /etc/alternatives/nawk
> 
> "awk" returns options for gawk and "mawk does nothing ( back to prompt)

So, what's the problem?

> how to run mawk ?

mawk

You have already run it once, with no arguments, and it did nothing.
Now try giving it some arguments.

wooledg:~$ mawk -v foo=bar 'BEGIN {print foo}'
bar

See, just like awk.

Re: Editing a piped in stream?

[mick crane]

On 2018-07-08 17:41, John Darrah wrote:

On 7/8/2018 7:00 AM, Richard Owlett wrote:


Until this thread I didn't have concept of "stream editor", much less 
the existence of "sed".

I had heard of "AWK", but had an image of it being a regex parser.


Awk is easy to learn because it is a minimal language. It can do
anything sed can do and much more. I am always surprised that so many
people opt to build long commands with cut/sed instead of using awk.
FWIW, I use mawk because it is orders of magnitude faster than gawk.

-- john



I'd like to learn some awk having found it handy once.
apt says mawk is installed

in /usr/bin I have
lrwxrwxrwx  1 root root 21 Apr  2 22:29 awk -> 
/etc/alternatives/awk

-rwxr-xr-x  1 root root 658072 Jan 25 12:55 gawk
-rwxr-xr-x  1 root root   3189 Jan 25 12:55 igawk
-rwxr-xr-x  1 root root 121976 Mar 23  2012 mawk
lrwxrwxrwx  1 root root 22 Apr  2 22:29 nawk -> 
/etc/alternatives/nawk


"awk" returns options for gawk and "mawk does nothing ( back to prompt)

In /etc/alternatives there is
awk
awk.1.gz
nawk
nawk.1.gz

how to run mawk ?


mick



--
Key ID4BFEBB31

Re: Editing a piped in stream?

[Greg Wooledge]

On Sun, Jul 08, 2018 at 07:39:07AM +0200, john doe wrote:
> The issue here is that we don't know what the OP wants (original and desired
> output)!!! :)

Indeed.  That's nearly always the problem (not just with this individual).

https://mywiki.wooledge.org/XyProblem
https://mywiki.wooledge.org/ParsingLs

Re: Editing a piped in stream?

[John Darrah]

On 7/8/2018 7:00 AM, Richard Owlett wrote:


Until this thread I didn't have concept of "stream editor", much less 
the existence of "sed".

I had heard of "AWK", but had an image of it being a regex parser.

Awk is easy to learn because it is a minimal language. It can do 
anything sed can do and much more. I am always surprised that so many 
people opt to build long commands with cut/sed instead of using awk. 
FWIW, I use mawk because it is orders of magnitude faster than gawk.


-- john

Re: Editing a piped in stream?

[Richard Owlett]

On 07/08/2018 08:12 AM, Richard Owlett wrote:

[snip]
In some threads I'm chided for lack of details.
In this and other threads I lose for giving too many.

Y'all will have admit that I did say "Subject line is poorly phrased."
Having read the responses, my subject line my not be all that bad.

A better subject/body might have been [How does one edit a piped in 
stream?] .

I'll lay odds that I would be asked for more details.



A philosophically good answer to that phrasing might be:

https://duckduckgo.com/html?q=%2B%22sed%22%20%2B%22awk%22%20%2B%22stream%22%20%2B%22tutorial%22

Until this thread I didn't have concept of "stream editor", much less 
the existence of "sed".

I had heard of "AWK", but had an image of it being a regex parser.

Re: Editing a piped in stream?

[Richard Owlett]

On 07/08/2018 06:18 AM, Andy Smith wrote:

Hello,

On Sun, Jul 08, 2018 at 07:39:07AM +0200, john doe wrote:

The issue here is that we don't know what the OP wants


A situation sadly familiar when dealing with this particular
poster's threads.

Also in the general case, if you ever find yourself parsing the
output of "ls", you will probably find that there is a better way to
be doing what you're trying to do. There are a lot of issues with
parsing "ls" output:

 
 


Sometimes I just can't win {//??? ;}
In some threads I'm chided for lack of details.
In this and other threads I lose for giving too many.

Y'all will have admit that I did say "Subject line is poorly phrased."
Having read the responses, my subject line my not be all that bad.

A better subject/body might have been [How does one edit a piped in 
stream?] .

I'll lay odds that I would be asked for more details.
If I can't win, can I break even? <*chuckle*>
Thanks

Re: Editing a piped in stream?

[Andy Smith]

Hello,

On Sun, Jul 08, 2018 at 07:39:07AM +0200, john doe wrote:
> The issue here is that we don't know what the OP wants

A situation sadly familiar when dealing with this particular
poster's threads.

Also in the general case, if you ever find yourself parsing the
output of "ls", you will probably find that there is a better way to
be doing what you're trying to do. There are a lot of issues with
parsing "ls" output:




Cheers,
Andy

Eureka;/ Re: Editing a piped in stream?

[Richard Owlett]

On 07/07/2018 08:33 PM, David Wright wrote:



On 07/06/18 09:17, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
     ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' > data.txt
I would then manually edit data.txt by replacing the space
character between the two fields with a tab.
I suspect I should be able to do:
     ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' |
*something* > prettydata.txt

What you want is a tool that can handle fields delimited by one
or more whitespace characters.  Regular expressions come to
mind, but RTFM cut(1) doesn't look promising:

 [snip]


One might assume the OP wants something like

$ lsblk -l -o NAME,LABEL,MOUNTPOINT
NAME LABEL   MOUNTPOINT
sda
sda2 SYSTEM_DRV
sda3 LRS_ESP
sda4
sda6 swan06  /
sda7 swan07  /wrenbk
sda10swan10  [SWAP]
swanhome /home



One might assume correctly :!
It even pretty prints the way I prefer.
That it can be sorted on a column is a bonus.
*THANK YOU*

Re: Editing a piped in stream?

[john doe]

On 7/8/2018 3:38 AM, John Crawley wrote:

On 2018-07-08 04:35, David Christensen wrote:

On 07/07/18 01:51, John Crawley wrote:

On 2018-07-07 11:02, David Christensen wrote:

On 07/06/18 09:17, Richard Owlett wrote:

While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space 
character between the two fields with a tab.

I suspect I should be able to do:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt


Ideal would be a more focussed way of reading out the disk info, 
perhaps something with blkid or lsblk?


I don't follow.  blkid(8) and lsblk(8) would be ideal for what 
purpose? Using what metric?


I meant that there might be a utility that would output the disk info 
directly, avoiding the need to do any post-processing with cut, sed, awk 
and friends, and suggested blkid and lsblk as possible candidates.


Just a suggestion I threw out for those with the time and motivation to 
plough though the man pages.




The issue here is that we don't know what the OP wants (original and 
desired output)!!! :)


There is probably a better solution that  the one of using multiples pipes.

--
John Doe

Re: Editing a piped in stream?

[John Crawley]

On 2018-07-08 04:35, David Christensen wrote:

On 07/07/18 01:51, John Crawley wrote:

On 2018-07-07 11:02, David Christensen wrote:

On 07/06/18 09:17, Richard Owlett wrote:

While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.

I suspect I should be able to do:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt


Ideal would be a more focussed way of reading out the disk info, 
perhaps something with blkid or lsblk?


I don't follow.  blkid(8) and lsblk(8) would be ideal for what purpose? 
Using what metric?


I meant that there might be a utility that would output the disk info 
directly, avoiding the need to do any post-processing with cut, sed, awk 
and friends, and suggested blkid and lsblk as possible candidates.


Just a suggestion I threw out for those with the time and motivation to 
plough though the man pages.


--
John

Re: Editing a piped in stream?

[David Wright]

On Sat 07 Jul 2018 at 12:35:02 (-0700), David Christensen wrote:
> On 07/07/18 01:51, John Crawley wrote:
> >On 2018-07-07 11:02, David Christensen wrote:
> >>On 07/06/18 09:17, Richard Owlett wrote:
> >>>Subject line is poorly phrased.
> >>>While working on a problem {solved by a different approach} I had:
> >>>    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
> >>>I would then manually edit data.txt by replacing the space
> >>>character between the two fields with a tab.
> >>>I suspect I should be able to do:
> >>>    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' |
> >>>*something* > prettydata.txt
> >>What you want is a tool that can handle fields delimited by one
> >>or more whitespace characters.  Regular expressions come to
> >>mind, but RTFM cut(1) doesn't look promising:
> >
> >>Perhaps awk(1) or sed(1) (?).
> >
> >awk is indeed one alternative, it accepts multi-spaces between fields:
> >ls -l /dev/disk/by-label/ | awk '/total/{next};{print $9"\t"$11}'
> >> prettydata.txt
> 
> Thank you for posting an Awk solution.  Continuing my example:
> 
> 2018-07-07 12:20:30 dpchrist@po ~
> $ ls -l /bin/e* | awk '/total/{next};{print $6"\t"$7"\t"$8}'
> Feb   22  2017
> Jan   23  2017
> 
> 
> >Ideal would be a more focussed way of reading out the disk info,
> >perhaps something with blkid or lsblk?
> 
> I don't follow.  blkid(8) and lsblk(8) would be ideal for what
> purpose? Using what metric?

One might assume the OP wants something like

$ lsblk -l -o NAME,LABEL,MOUNTPOINT
NAME LABEL   MOUNTPOINT
sda  
sda2 SYSTEM_DRV  
sda3 LRS_ESP 
sda4 
sda6 swan06  /
sda7 swan07  /wrenbk
sda10swan10  [SWAP]
swanhome /home
$ 

or whichever two columns are of interest.
(Just a selection of the output lines here.)

Cheers,
David.

Re: Editing a piped in stream?

[David Christensen]

On 07/07/18 01:51, John Crawley wrote:

On 2018-07-07 11:02, David Christensen wrote:

On 07/06/18 09:17, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.

I suspect I should be able to do:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt
What you want is a tool that can handle fields delimited by one or 
more whitespace characters.  Regular expressions come to mind, but 
RTFM cut(1) doesn't look promising:



Perhaps awk(1) or sed(1) (?).


awk is indeed one alternative, it accepts multi-spaces between fields:
ls -l /dev/disk/by-label/ | awk '/total/{next};{print $9"\t"$11}' > 
prettydata.txt


Thank you for posting an Awk solution.  Continuing my example:

2018-07-07 12:20:30 dpchrist@po ~
$ ls -l /bin/e* | awk '/total/{next};{print $6"\t"$7"\t"$8}'
Feb 22  2017
Jan 23  2017


Ideal would be a more focussed way of reading out the disk info, perhaps 
something with blkid or lsblk?


I don't follow.  blkid(8) and lsblk(8) would be ideal for what purpose? 
Using what metric?



David

Re: Editing a piped in stream?

[John Crawley]

On 2018-07-07 11:02, David Christensen wrote:

On 07/06/18 09:17, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.

I suspect I should be able to do:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt
What you want is a tool that can handle fields delimited by one or more 
whitespace characters.  Regular expressions come to mind, but RTFM 
cut(1) doesn't look promising:



Perhaps awk(1) or sed(1) (?).


awk is indeed one alternative, it accepts multi-spaces between fields:
ls -l /dev/disk/by-label/ | awk '/total/{next};{print $9"\t"$11}' > 
prettydata.txt


Ideal would be a more focussed way of reading out the disk info, perhaps 
something with blkid or lsblk?


--
John

Re: Editing a piped in stream?

[David Christensen]

On 07/06/18 09:17, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.


I assume you meant:

ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' > data.txt


(It is best to cut and paste from a console session, rather than typing 
untested commands and/or output into a post.)




I suspect I should be able to do:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt


I searched for examples/tutorials found that it should be conceptually 
possible. However the examples I found were processing streams I didn't 
understand.


What should I be looking for?


Beware that ls(1) with the -l option can produce output with a variable 
number of spaces between fields:


2018-07-06 18:51:53 dpchrist@po ~
$ ls -l /bin/e*
-rwxr-xr-x 1 root root 31464 Feb 22  2017 /bin/echo
-rwxr-xr-x 1 root root28 Jan 23  2017 /bin/egrep


When you feed that to cut(1) and change the delimiter to a space, the 
result is broken (and impossible to fix?) because cut(1) expects exactly 
one delimiter between each field:


2018-07-06 18:52:27 dpchrist@po ~
$ ls -l /bin/e* | cut -f 6,7,9 -d ' '
Feb 22 2017
  Jan


What you want is a tool that can handle fields delimited by one or more 
whitespace characters.  Regular expressions come to mind, but RTFM 
cut(1) doesn't look promising:


2018-07-06 18:53:45 dpchrist@po ~
$ ls -l /bin/e* | cut -f 6,7,9 -d '\s+'
cut: the delimiter must be a single character
Try 'cut --help' for more information.


Perhaps awk(1) or sed(1) (?).


Perl with the -a (autosplit) and -e (evaluate) options is ideal in this 
situation, replacing both cut(1) and "something":


2018-07-06 18:55:29 dpchrist@po ~
$ ls -l /bin/e* | perl -ae 'print join("\t", @F[5,6,7]), "\n"'
Feb 22  2017
Jan 23  2017


Perhaps other people will post solutions in their favorite scripting/ 
programming language.  These give you more power and control than 
traditional shell pipelines and command-line tools.  You should consider 
learning one.



David

Re: Editing a piped in stream?

[Richard Owlett]

On 07/06/2018 11:46 AM, Michael Wagner wrote:

On Jul 06, 2018 um 11:17:56, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character between
the two fields with a tab.

I suspect I should be able to do:
ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* >
prettydata.txt

I searched for examples/tutorials found that it should be conceptually
possible. However the examples I found were processing streams I didn't
understand.


Hello Richard,

one way is to do it with sed:

ls -l /dev/disk/by-label/|cut -f 10,12 -d ' '|sed 's/ /\t/g'>prettydata.txt

Hth Michael



Thank you.
https://www.gnu.org/software/sed/manual/sed.html makes interesting reading.

Re: Editing a piped in stream?

[john doe]

On 7/6/2018 6:17 PM, Richard Owlett wrote:

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
    ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.




What is your original output and what is the desired output?

--
John Doe

Re: Editing a piped in stream?

[Michael Wagner]

On Jul 06, 2018 um 11:17:56, Richard Owlett wrote:
> Subject line is poorly phrased.
> While working on a problem {solved by a different approach} I had:
>ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
> I would then manually edit data.txt by replacing the space character between
> the two fields with a tab.
> 
> I suspect I should be able to do:
>ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* >
> prettydata.txt
> 
> I searched for examples/tutorials found that it should be conceptually
> possible. However the examples I found were processing streams I didn't
> understand.

Hello Richard,

one way is to do it with sed:

ls -l /dev/disk/by-label/|cut -f 10,12 -d ' '|sed 's/ /\t/g'>prettydata.txt

Hth Michael



signature.asc
Description: PGP signature

Editing a piped in stream?

[Richard Owlett]

Subject line is poorly phrased.
While working on a problem {solved by a different approach} I had:
   ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' data.txt
I would then manually edit data.txt by replacing the space character 
between the two fields with a tab.


I suspect I should be able to do:
   ls -l /dev/disk/by-label/ | cut -f 10,12 -d ' ' | *something* > 
prettydata.txt


I searched for examples/tutorials found that it should be conceptually 
possible. However the examples I found were processing streams I didn't 
understand.


What should I be looking for?
TIA