Re: Whitespace problem with bash script for Icedove

[Mark Neyhart]

Jimmy Wu wrote:

> 1. Just use the shell's globbing - it was meant for this purpose -
> instead of assigning a glob to a variable.  Glob expansions can be
> iterated over in loops correctly and safely.  Expanded strings are
> much more tricky to get right (if it is even possible) and less safe
> when you don't.
> 
> FILES="/home/bill/.../Local Folders/2-Personal.sbd"
> 
> for i in "$FILES"/*
> 
> note the glob is outside the quotes
> 
I like this solution better than the one I had suggested.

Mark Neyhart


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4e2f15f4.5050...@legis.state.ak.us

Re: Whitespace problem with bash script for Icedove

[Stanisław Findeisen]

On 2011-07-25 20:07, Bill M wrote:
> Hi folks,
> 
> I'm piecing together a small script to remove duplicate messages from
> various directories in Icedove. By default the files are stored in a
> directory called 'Local Folders' - with the space - and this seems to be
> creating a problem for bash. I can both cd to "Local Folders" and cd to
> Local\ Folders from the command line but within the script it's a
> different story. Bash refuses to recognize "Local Folders" as a
> directory and breaks at the whitespace. Here's the script and the error
> output. Any help appreciated.
> 
> bill
> 
> #! /bin/bash
> 
> # A script to remove duplicate messages
> 
> FILES=/home/bill/.icedove/qjimvr85.default/Mail/Local\
> Folders/2-Personal.sbd/*
> # or "/home/bill/.icedove/qjimvr85.default/Mail/Local
> Folders/2-Personal.sbd/*"
> 
> for i in $FILES
> do
> mv "$i" ""$i".saved"
> formail -D 65536 .msgid.cache -s < ""$i".saved" > "$i"
> done
> 
> mv: cannot stat `/home/bill/.icedove/qjimvr85.default/Mail/Local': No
> such file or directory
> ./mailscript: line 10:
> /home/bill/.icedove/qjimvr85.default/Mail/Local.saved: No such file or
> directory
> mv: cannot stat `Folders/2-Personal.sbd/*': No such file or directory
> ./mailscript: line 10: Folders/2-Personal.sbd/*.saved: No such file or
> directory

Hi Bill

Not sure about your Bash script, but 2 months ago I had the same problem
of removing duplicate messages from Thunderbird/Icedove directory tree
and crafted this: http://people.eisenbits.com/~stf/fsd/mboxgrep-perl/ .

Maybe this will be of some help to you, or maybe you will find a bug or
fix the documentation. :-)

mboxgrep-perl comes with no warranty, of course. Backup all your files
before using it.

mboxgrep-perl.pl 0.1 (2011-05-28)

Copyright (C) 2011 Stanislaw Findeisen 
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

Usage:
  mboxgrep-perl-0.1.pl [OPTION]...

DESCRIPTION

mboxgrep-perl is a very simple mbox file parser. Features include:
* ability to grep on e-mail headers using regular expressions
* ability to filter out duplicate messages.

Each e-mail message encountered on input is processed once and classified
either as a match or not-match. Matching messages are printed out on
standard output. They can also be deleted from input files.

Using --dup you can filter out duplicate messages. There are 4 different
ways
to define when 2 messages are equal.

OPTIONS

Input control

--source
mbox file to parse. If this parameter is not specified, standard input
is read. If this parameter is a directory, it is being read recursively.

E-mail matching

--equiv
Message identity (equivalence relation) definition. Use 'message-id' for
Message-ID header fields, 'identity' for binary identity (i.e. hashes
over whole message) or 'content' for hashes over several fields (i.e.:
Message-ID, Date, From, To, Cc, Subject, body prefix, body suffix).

By default, all e-mail messages are considered different to each other.

Note: not every e-mail message has Message-ID field. If this parameter
is set to 'message-id' then all such messages will be considered
identical
(and a warning will be printed on standard error).

Note: message equivalence has precedence over regular expression
matching, i.e. if 2 messages are equivalent to each other and one of
them matches the regex while the other does not, it is unspecified
which one of them will be considered a match.

--expr
Perl-compatible regular expression to match the e-mail header against.
If this parameter is not specified, the expression is considered empty
and is satisfied by every e-mail message.

THIS IS A SECURITY HOLE (the expression is passed as-is to Perl's qr//).

-i
The regular expression specified with --expr is case-insensitive.

--dup
In order to match, the message must be a duplicate (in processing
order),
i.e. it must have been preceeded by some other, equivalent message
(see --equiv).

-v
Invert match, i.e. match messages that *do not* satisfy --expr *or are
not* a duplicate (if --dup is specified).

Output control

--del
Delete matching e-mail messages from input file(s).
You'd better make a backup first!

--tmpDir
Temporary directory to use.

-q
Quiet mode: do not output any e-mails (just parse the file and report
the count of matching e-mail messages).

--qq
Be even more quiet: suppress debug messages. This implies -q.

Miscellaneous

--help
Print this help message.

-w
By default, mboxgrep-perl uses SHA-256 hashes. Use this option for MD5
instead. This is weaker but may be faster.

BUGS

T

Re: Whitespace problem with bash script for Icedove

[Jimmy Wu]

On Tue, Jul 26, 2011 at 02:40, Mark Neyhart
 wrote:
> Bill M wrote:
>
>> FILES=/home/bill/.icedove/qjimvr85.default/Mail/Local\
>> Folders/2-Personal.sbd/*
>> # or "/home/bill/.icedove/qjimvr85.default/Mail/Local
>> Folders/2-Personal.sbd/*"
>>
>> for i in $FILES
[...]
>
> This appears to be related to the default value of the BASH internal
> variable $IFS (internal field separator).  It defaults to whitespace,
> which includes the space character.  The for loop interprets the space
> inside $FILES as a field separator.  Try setting $IFS to ONLY new line
> above the for loop.
>
> IFS=$'\n'
> for i in $FILES

The problem as Mark has correctly pointed out is this line.  Since
$FILES is not quoted, bash will expand it and then split on IFS.

If you are on an ext? file system the only safe separator for
filenames is the null byte '\0', unless you can guarantee somehow that
your script will never see a file with a newline in the name.  Thus,
I'd recommend not using IFS for this purpose.

Two possible more robust solutions exist:

1. Just use the shell's globbing - it was meant for this purpose -
instead of assigning a glob to a variable.  Glob expansions can be
iterated over in loops correctly and safely.  Expanded strings are
much more tricky to get right (if it is even possible) and less safe
when you don't.

FILES="/home/bill/.../Local Folders/2-Personal.sbd"

for i in "$FILES"/*

note the glob is outside the quotes

2. Use find with it's own exec options or xargs.

I suggested this second b/c it is more complicated, depends on an
external command, and the first does what you need, so why bother with
this one? :-)

3. Ok I lied, there's a third: you could use BASH(-only) arrays, but
since you are working with files anyways it basically comes down to
doing #1 less portably and with more syntax.

>> do
>>          mv "$i" ""$i".saved"
>>          formail -D 65536 .msgid.cache -s < ""$i".saved" > "$i"

what's with the double double quotes "" ? typo?


--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/can0j2y6opzabavrnvryy6zaqljii7s9_4xys-qaxpzb8gsa...@mail.gmail.com

Re: Whitespace problem with bash script for Icedove

[Mark Neyhart]

Bill M wrote:

> FILES=/home/bill/.icedove/qjimvr85.default/Mail/Local\ 
> Folders/2-Personal.sbd/*
> # or "/home/bill/.icedove/qjimvr85.default/Mail/Local 
> Folders/2-Personal.sbd/*"
> 
> for i in $FILES
> do
>  mv "$i" ""$i".saved"
>  formail -D 65536 .msgid.cache -s < ""$i".saved" > "$i"
> done
> 
> mv: cannot stat `/home/bill/.icedove/qjimvr85.default/Mail/Local': No 
> such file or directory
> ./mailscript: line 10: 
> /home/bill/.icedove/qjimvr85.default/Mail/Local.saved: No such file or 
> directory
> mv: cannot stat `Folders/2-Personal.sbd/*': No such file or directory
> ./mailscript: line 10: Folders/2-Personal.sbd/*.saved: No such file or 
> directory
>

This appears to be related to the default value of the BASH internal
variable $IFS (internal field separator).  It defaults to whitespace,
which includes the space character.  The for loop interprets the space
inside $FILES as a field separator.  Try setting $IFS to ONLY new line
above the for loop.

IFS=$'\n'
for i in $FILES
..
..
..

Mark Neyhart


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4e2db8af.8000...@legis.state.ak.us

Re: Whitespace problem with bash script for Icedove

[Michael Checca]

On Mon, 25 Jul 2011 14:07:11 -0400, Bill M  wrote:


Hi folks,

I'm piecing together a small script to remove duplicate messages from  
various directories in Icedove. By default the files are stored in a  
directory called 'Local Folders' - with the space - and this seems to be

creating a problem for bash. I can both cd to "Local Folders" and cd to
Local\ Folders from the command line but within the script it's a  
different story. Bash refuses to recognize "Local Folders" as a  
directory and breaks at the whitespace. Here's the script and the error  
output. Any help appreciated.


bill

#! /bin/bash

# A script to remove duplicate messages

FILES=/home/bill/.icedove/qjimvr85.default/Mail/Local\  
Folders/2-Personal.sbd/*
# or "/home/bill/.icedove/qjimvr85.default/Mail/Local  
Folders/2-Personal.sbd/*"


for i in $FILES
do
 mv "$i" ""$i".saved"
 formail -D 65536 .msgid.cache -s < ""$i".saved" > "$i"
done

mv: cannot stat `/home/bill/.icedove/qjimvr85.default/Mail/Local': No  
such file or directory
./mailscript: line 10:  
/home/bill/.icedove/qjimvr85.default/Mail/Local.saved: No such file or  
directory

mv: cannot stat `Folders/2-Personal.sbd/*': No such file or directory
./mailscript: line 10: Folders/2-Personal.sbd/*.saved: No such file or  
directory





Bash is treating FILES as an array of
/home/bill/.icedove/qjimvr85.default/Mail/Local and
Folders/2-Personal.sbd/*
Try this instead:
FILES = `ls '/home/bill/.icedove/qjimvr85.default/Mail/Local
Folders/2-Personal.sbd/'` OR
FILES = `ls /home/bill/.icedove/qjimvr85.default/Mail/Local\
Folders/2-Personal.sbd/`
Notice the backticks around the command and the single quotes (could also
be double) around the path.
The backticks means to substitute the output of the command into $FILES.

HTH,
Mike

--
Michael Checca
echo "complaints" > /dev/null


--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Archive: http://lists.debian.org/op.vy6tkixz4b8ft1@michael-laptop