Re: Help with sed

[Tom Munro Glass]

Thanks for the feedback guys. In the end I discovered the following REALLY 
helpful web page "Sed - An Introduction" at 
http://www.grymoire.com/Unix/Sed.html. As it says in th introduction, the 
documentation for sed is poor, but it's a really powerful tool if you can 
find out how to use it.

This helped me to produce the following script which does exactly what I want, 
and does it very quickly. Although my requirement is very specific, I hope 
the technique for dealing with multiline search and replace might be useful 
for others.

#!/bin/bash
sed '
# Look for 'Type=0x21'
/Type=0x21/ {
# append the next line to the pattern space
N
# Look for 'Label=...00'
/Label=...00/ {
# Replace 'Type=0x21' with 'Type=0x22'
s/Type=0x21/Type=0x22/
# print the modified data
P
# Append the extra line
a\
EndLevel=1
# then delete the first line
D
}
}' $1 > /tmp/tempfile

mv /tmp/tempfile $1



Tom

On Tue, 25 Oct 2005 18:02, Volker Kuhlmann wrote:
> > Type=0x21   Type=0x22   # Same as previous except
> > Label= 2200 Label= 2200 # different value for Label
> > blah, blah  EndLevel=1
> > blah, blah
> >
> > The bit that's got me stumped is how to handle the Label line. Can
> > someone please give me hint how to do this?
>
> The problem I have with sed is that it's line oriented, not record
> oriented, and your job is record oriented. (Personally I'd use gawk.) It
> can be a pain to match across lines in sed, but it's possible in GNU
> sed.
>
> The match condition I'd try for the label line is whether the number
> ends in 2 zeros. You'd have to match all from the type= to the label= in
> one expression, and if matched, perform one or more search/replaces, and
> write out the result with any additional stuff you want appended after
> label= . You can ignore any text following label= in the same record.
>
> Does that help?
>
> Volker

Re: Help with sed

[Carl Cerecke]

Here. Reads from stdin and writes to stdout. Should do what you want.
Don't forget to make it executable. Save to a file, e.g. "type_munger.py"
then:
chmod +x type_munger.py
then:
./type_munger.py < input_file.txt > output_file.txt

Your line terminators, whatever they are (\n or \r\n or \r) are
stripped off and whatever line terminators are native to the system
you are running the script on will be used.

#!/usr/bin/env python

from sys import stdin

lines = stdin.readlines() # suck up all the lines
state = 'normal'
for line in lines:
line = line.rstrip() # Strip whitespace off the end
if line == "Type=0x21":
state = "seen 0x21"
continue
if state == "seen 0x21":
if line[9:11] == "00":
print "Type=0x22"
else:
print "Type=0x21"
state = 'normal'
print line



On 25/10/05, Tom Munro Glass <[EMAIL PROTECTED]> wrote:
> I've got several large files that require some repetitive multiline search and
> replace operations, and I feel sure that sed can do this, but I can't figure
> out how to do it.
>
> The pattern I'm searching for in the input file is "Type=0x21" followed by
> "Label=abcde" on the next line, where abcde is a 5 character number with
> leading spaces. If abcde is a multiple of 100, (d=0, e=0) I want to replace
> the first line with "Type=0x22", and add a third line "EndLevel=1"
>
> For example:
>
> BEFORE: AFTER:
> Type=0x21   Type=0x21   # No change required because
> Label=  120 Label=  120 # Label not multiple of 100
> blah, blah  blah, blah
>
> Type=0x21   Type=0x22   # Change Type value
> Label=  100 Label=  100
> blah, blah  EndLevel=1  # Insert extra line
> blah, blah
>
> Type=0x21   Type=0x22   # Same as previous except
> Label= 2200 Label= 2200 # different value for Label
> blah, blah  EndLevel=1
> blah, blah
>
> The bit that's got me stumped is how to handle the Label line. Can someone
> please give me hint how to do this?
>
> Tom
>

Re: Help with sed

[Volker Kuhlmann]

> Type=0x21 Type=0x22   # Same as previous except
> Label= 2200   Label= 2200 # different value for Label
> blah, blahEndLevel=1
>   blah, blah
> 
> The bit that's got me stumped is how to handle the Label line. Can someone 
> please give me hint how to do this?

The problem I have with sed is that it's line oriented, not record
oriented, and your job is record oriented. (Personally I'd use gawk.) It
can be a pain to match across lines in sed, but it's possible in GNU
sed.

The match condition I'd try for the label line is whether the number
ends in 2 zeros. You'd have to match all from the type= to the label= in
one expression, and if matched, perform one or more search/replaces, and
write out the result with any additional stuff you want appended after
label= . You can ignore any text following label= in the same record.

Does that help?

Volker

-- 
Volker Kuhlmann is possibly list0570 with the domain in header
http://volker.dnsalias.net/ Please do not CC list postings to me.

Help with sed

[Tom Munro Glass]

I've got several large files that require some repetitive multiline search and 
replace operations, and I feel sure that sed can do this, but I can't figure 
out how to do it.

The pattern I'm searching for in the input file is "Type=0x21" followed by 
"Label=abcde" on the next line, where abcde is a 5 character number with 
leading spaces. If abcde is a multiple of 100, (d=0, e=0) I want to replace 
the first line with "Type=0x22", and add a third line "EndLevel=1"

For example:

BEFORE: AFTER:
Type=0x21   Type=0x21   # No change required because
Label=  120 Label=  120 # Label not multiple of 100
blah, blah  blah, blah

Type=0x21   Type=0x22   # Change Type value
Label=  100 Label=  100
blah, blah  EndLevel=1  # Insert extra line
blah, blah

Type=0x21   Type=0x22   # Same as previous except
Label= 2200 Label= 2200 # different value for Label
blah, blah  EndLevel=1
blah, blah

The bit that's got me stumped is how to handle the Label line. Can someone 
please give me hint how to do this?

Tom

Re: [Fwd: Re: Help with sed]

[Carey Evans]

Phill Coxon wrote:
Steve sent me this which works great. 

Thanks Nick & Rex for your suggestions to. 
I can't resist posting another couple of options.  In sed, it's a one-liner:
$ sed -e 's/.*/&\n"&"\n[&]/' infile > outfile
It's also possible to read line-by-line in a shell script without 
messing with IFS:

  while read -r; do
echo "$REPLY"
echo "\"$REPLY\""
echo "[$REPLY]"
  done < infile > outfile
This seems to strip leading and trailing spaces, depending on the shell.
--
"A story, I decided, is anything that keeps http://carey.geek.nz/
the people reading turning the pages, and
doesn't leave them feeling cheated at the
end.  Everything else was up for grabs." -- Neil Gaiman


signature.asc
Description: OpenPGP digital signature

Re: Help with sed

[Matthew Gregan]

At 2004-06-30T123531+1200, Jim Cheetham wrote:
> Still, it sounds like another homework question to me. Is someone
> running through an "introduction to unix" course somewhere?

As part of the previously discussed renaming of the CLUG, perhaps we
need a slogan to go with the name... 

"We're edGNUcational, but not to the point that we'll do your homework."

Two birds, one stone.

Groans > /dev/null

Cheers,
-mjg
-- 
Matthew Gregan |/
  /|[EMAIL PROTECTED]

Re: Help with sed

[Phill Coxon]

No homework. :)  

I'm doing some work with Google Adwords at the moment. 

Google Adwords provides different search responses based on whether the
search terms have quotes or square brackets. 

search term - will match any search terms containing either of these
words
"search term" - will match only this exact search term
[search term] - will match any search term containing this string

On Wed, 2004-06-30 at 12:35, Jim Cheetham wrote:

> Still, it sounds like another homework question to me. Is someone 
> running through an "introduction to unix" course somewhere?

Re: Help with sed

[Matthew Gregan]

At 2004-06-30T122050+1200, Nick Rout wrote:

> for line in $(cat $1) ; do

Assuming the input is your 'testin' file below, this will expand to:

for line in foo bar foo bar ; do

> unfortunately, and for reasons unknown to me it doesn't work properly
> on lines with a space in.

Once you realise what the 'for...' line is expanding into, it should be
obvious why you script behaves the way it does.

> $ cat testin
> foo
> bar
> foo bar

Cheers,
-mjg
-- 
Matthew Gregan |/
  /|[EMAIL PROTECTED]

Re: Help with sed

[Jim Cheetham]

Nick Rout wrote:
for line in $(cat $1) ; do
echo $line
echo "["$line"]"
echo "*"${line}"*"
done
unfortunately, and for reasons unknown to me it doesn't work properly on
lines with a space in.
The reason is the IFS variable setting.
The  $(cat $1)  statement expands to a whole series of things, separated 
by spaces, newlines and tabs. IFS by default treats all these as being 
equivalent, hence the "spaces are the same as newlines" problem.

As was posted later, setting IFS to contain just a newline
IFS='
'
means that spaces (and tabs) are no longer treated as field separators.
perl doesn't do this :-
[EMAIL PROTECTED]:~$ perl -e 'while (<>) {chomp;print "$_\n[$_]\n*$_*\n"}'
fred
fred
[fred]
*fred*
foo bar
foo bar
[foo bar]
*foo bar*
Still, it sounds like another homework question to me. Is someone 
running through an "introduction to unix" course somewhere?

-jim

Re: [Fwd: Re: Help with sed]

[Nick Rout]

of course the standard unix way is to make the output go to stdout and
redirect it with > if you want it in a file.

I think I was missing the IFS line, other than that, and outputting to a
file, its the same as mine.


On Wed, 30 Jun 2004 12:24:48 +1200
Phill Coxon <[EMAIL PROTECTED]> wrote:

> Steve sent me this which works great. 
> 
> Thanks Nick & Rex for your suggestions to. 
> 
> -Forwarded Message-
> From: [EMAIL PROTECTED]
> To: Phill Coxon <[EMAIL PROTECTED]>
> Subject: Re: Help with sed
> Date: Wed, 30 Jun 2004 12:09:57 +1200
> 
> Hi Phill,
> 
> Try this...
> 
> 8>< cut here 8><
> #!/bin/bash
> 
> IFS='
> '
> for Line in `cat `
> do
> echo $Line
> echo '"'$Line'"'
> echo '['$Line']'
> done > 
> 8>< cut here 8><
> 
> Replacing  and  as necessary
> 

-- 
Nick Rout <[EMAIL PROTECTED]>

[Fwd: Re: Help with sed]

[Phill Coxon]

Steve sent me this which works great. 

Thanks Nick & Rex for your suggestions to. 

-Forwarded Message-
From: [EMAIL PROTECTED]
To: Phill Coxon <[EMAIL PROTECTED]>
Subject: Re: Help with sed
Date: Wed, 30 Jun 2004 12:09:57 +1200

Hi Phill,

Try this...

8>< cut here 8><
#!/bin/bash

IFS='
'
for Line in `cat `
do
echo $Line
echo '"'$Line'"'
echo '['$Line']'
done > 
8>< cut here 8><

Replacing  and  as necessary

Re: Help with sed

[Nick Rout]

On Wed, 30 Jun 2004 12:15:15 +1200
Phill Coxon <[EMAIL PROTECTED]> wrote:

> Not at all. :)
> 
> On Wed, 2004-06-30 at 12:09, Nick Rout wrote:
> > does it have to be sed?
> 


#!/bin/bash

for line in $(cat $1) ; do
echo $line
echo "["$line"]"
echo "*"${line}"*"
done

unfortunately, and for reasons unknown to me it doesn't work properly on
lines with a space in.

eg (the script is called forphil)

$ cat testin
foo
bar
foo bar


$ forphil testin
foo
[foo]
*foo*
bar
[bar]
*bar*
foo
[foo]
*foo*
bar
[bar]
*bar*


-- 
Nick Rout <[EMAIL PROTECTED]>

Re: Help with sed

[Phill Coxon]

Not at all. :)

On Wed, 2004-06-30 at 12:09, Nick Rout wrote:
> does it have to be sed?

Re: Help with sed

[Rex Johnston]

Phill Coxon wrote:
Hey guys. 

Can someone help me with a sed / bash script that takes an input file
and adds quotes and square brackets to the contents of each line,
outputting each variation to a new file? 

i.e.:
Input file:
foo
bar
foo bar wizz bang
Output file:
foo
"foo"
[foo]
bar
"bar"
[bar]
foo bar wizz bang
"foo bar wizz bang"
[foo bar wizz bang"
Something like
sed -e 's/^.*$/\1\n"\1"\n[\1]\n/'
not tested
Cheers, Rex

Re: Help with sed

[Nick Rout]

does it have to be sed?


On Wed, 30 Jun 2004 12:05:09 +1200
Phill Coxon <[EMAIL PROTECTED]> wrote:

> Hey guys. 
> 
> Can someone help me with a sed / bash script that takes an input file
> and adds quotes and square brackets to the contents of each line,
> outputting each variation to a new file? 
> 
> i.e.:
> 
> Input file:
> 
> foo
> bar
> foo bar wizz bang
> 
> Output file:
> 
> foo
> "foo"
> [foo]
> bar
> "bar"
> [bar]
> foo bar wizz bang
> "foo bar wizz bang"
> [foo bar wizz bang"
> 
> 
> Thank you. 
>  

-- 
Nick Rout <[EMAIL PROTECTED]>

Help with sed

[Phill Coxon]

Hey guys. 

Can someone help me with a sed / bash script that takes an input file
and adds quotes and square brackets to the contents of each line,
outputting each variation to a new file? 

i.e.:

Input file:

foo
bar
foo bar wizz bang

Output file:

foo
"foo"
[foo]
bar
"bar"
[bar]
foo bar wizz bang
"foo bar wizz bang"
[foo bar wizz bang"


Thank you.