amandas group membership in FC6?
Greetings; Despite the fact that the user 'amanda' is a member of the group 'disk', all compilations and new files generated by the user amanda seem to be owned by amanda:amanda instead of the expected amanda:disk. The end result is that many of my backup operations are failing because the amanda utility doesn't have perms to delete or write to files actually owned by amanda:disk. I just went thru all the directory trees amanda needs to access and chowned everything back the way its supposed to be, but then I built the 20061124 tarball just now, and everything is still owned by amanda:amanda. From my /etc/group file: disk:x:6:root,amanda So I blew it away, called up KUsr and verified that amanda was indeed a member of the group disk. Even deleted the user and re-added it and made sure this new copy of amanda was a member of the group disk. Then as amanda, I unpacked it again and rebuilt it, but I still have the same problem. Because none of the files are owned by amanda:disk, the end result is several megs of dead, can't do a thing code that I'd just as well not bother with the 'make install'. Anything that amanda has touched over the last 4 days since I started running it again has been converted to being owned by amanda:amanda, and if the file existed, and was to be deleted as part of the housekeeping, was not because the old file was owned by amanda:disk. So my backups are being slowly trashed because the indice files are not updatable. Whats the deal with FC6 and its owner:group handling? Am I setting up the user wrong or what? -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
lev 1 FAILED [no backup size line]
Hi,
Since a few days one of my DLEs consistently fails with:
| /-- anakin /home/src lev 1 FAILED [no backup size line]
| sendbackup: start [anakin:/home/src level 1]
| sendbackup: info BACKUP=/bin/tar
| sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/tar -f - ...
| sendbackup: info COMPRESS_SUFFIX=.gz
| sendbackup: info end
| ? gtar: /var/lib/amanda/gnutar-lists/anakin_home_src_1.new: Missing record
terminator
| ? gtar: Error is not recoverable: exiting now
| sendbackup: error [no backup size line]
sendbackup.20061125005608.debug has:
| sendbackup: start: anakin:/home/src lev 1
| sendbackup: time 0.000: spawning /bin/gzip in pipeline
| sendbackup: argument list: /bin/gzip --fast
| sendbackup-gnutar: time 0.001: pid 8618: /bin/gzip --fast
| sendbackup-gnutar: time 7.362: doing level 1 dump as listed-incremental from
'/var/lib/amanda/gnutar-lists/anakin_home_src_0' to
'/var/lib/amanda/gnutar-lists/anakin_home_src_1.new'
| sendbackup-gnutar: time 7.407: doing level 1 dump from date: 2006-11-20
12:45:43 GMT
| sendbackup: time 7.419: started index creator: /bin/tar -tf - 2/dev/null |
sed -e 's/^\.//'
| sendbackup: time 7.420: spawning /usr/lib/amanda/runtar in pipeline
| sendbackup: argument list: runtar DailySet1 gtar --create --file -
--directory /home --one-file-system --listed-incremental
/var/lib/amanda/gnutar-lists/anakin_home_src_1.new --sparse
--ignore-failed-read --totals --exclude-from
/tmp/amanda/sendbackup._home_src.20061125005616.exclude --files-from
/tmp/amanda/sendbackup._home_src.20061125005616.include
| sendbackup-gnutar: time 7.423: /usr/lib/amanda/runtar: pid 8623
| sendbackup: time 7.423: started backup
| sendbackup: time 13.134: 118: strange(?): gtar:
/var/lib/amanda/gnutar-lists/anakin_home_src_1.new: Missing record terminator
| sendbackup: time 13.148: 118: strange(?): gtar: Error is not recoverable:
exiting now
| sendbackup: time 13.162: index created successfully
| sendbackup: time 13.163: error [no backup size line]
| sendbackup: time 13.163: pid 8616 finish time Sat Nov 25 00:56:21 2006
Anyone with a clue? Could this be /var running out of diskspace?
I'm using Debian, with Amanda 2.5.1p1-2 and tar 1.16-1.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say programmer or something like that.
-- Linus Torvalds
Re: 2.5.1p2: planner returns error=EOF on read
* Jean-Louis Martineau [EMAIL PROTECTED] [20061123 17:20]: Do amandad is running? To be sure that I would hit the problem again I didn't load new tapes so after the holddding filled up I have a bunch of gtar running and just wasting a lot of cycles. A typical example (sorry for the long lines): amanda35559363557855 0- - 0:00 defunct amanda3557855 1 0 05:37:39 ? 1:30 /opt/amanda/amanda1/libexec/sendbackup amandad bsdtcp root35680873557855 0 05:40:02 ? 277:35 gtar --create --file - --directory /data/mafalda/mafalda1/susanita/sandra/DTI - Tracing the sendbackup process yields no output while the gtar one gives just an uninterupted flow of lines: 68mS tar(3568087): write(1, fd 63 e3 6c ff 42 3c df 87 41 82 bf e9 dd 95 2c..., 10240) errno = 32 (Broken pipe) Don't know if this is going to be of any help... jf Could you try to find what each process (amanda,sendbackup,tar) are doing? on which system call they are hung? On linux, I would use strace for that. Jean-Louis Jean-Francois Malouin wrote: Jean-Louis, The amdump just finished and because amanda ran out of tapes (runtapes=10) it completed badly leaving stuff in the holdding disk. I'm flushing it at the moment but I noticed that when amanda gave up it left a lot of gnutar lying around, with amandad as parent. I had to manually kill them. Looks like signals are not doing ok. I've attached one example of a DLE's sendbackup and runtar debug file. It's not the first time I notice that when a DLE fails to make it to tape successfully processes are left running... * Jean-Louis Martineau [EMAIL PROTECTED] [20061122 09:32]: Could you post amandad.timestamp.debug file from yorick? Jean-Francois Malouin wrote: On a server running irix-6.5 and amanda 2.5.1p2 planner debug shows: security_getdriver(name=bsdtcp) returns 4075aa8 security_handleinit(handle=1001d4a8, driver=4075aa8 (BSDTCP)) security_streaminit(stream=1005e538, driver=4075aa8 (BSDTCP)) security_close(handle=1001cfe0, driver=4075aa8 (BSDTCP)) security_stream_close(10053a80) security_stream_seterr(1005e538, SOCKET_EOF) security_seterror(handle=1001d4a8, driver=4075aa8 (BSDTCP) error=EOF on read from yorick) security_close(handle=1001d4a8, driver=4075aa8 (BSDTCP)) security_stream_close(1005e538) and the amanda report shows things like: yorick DATA_sub101 lev 0 FAILED [hmm, disk was stranded on waitq] planner: ERROR Request to yorick failed: EOF on read from yorick and the backup fails. The funny thing is that I have 2 other configurations running 2.5.1p2 in parallel that doesn't exhibit this behaviour. Any clues? jf runtar: debug 1 pid 2826413 ruid 666 euid 0: start at Thu Nov 23 08:04:13 2006 runtar: version 2.5.1p2 /usr/freeware/bin/tar version: tar (GNU tar) 1.13.25 config: stk_80-conf1 runtar: debug 1 pid 2826413 ruid 0 euid 0: rename at Thu Nov 23 08:04:13 2006 running: /usr/freeware/bin/tar: 'gtar' '--create' '--file' '-' '--directory' '/data/mafalda/mafalda1/susanita/jen/anxiety_version1/sub101' '--one-file-system' '--listed-incremental' '/opt/amanda/amanda1/var/amanda/gnutar-lists/yoricksub101_1.new' '--sparse' '--ignore-failed-read' '--totals' '.' runtar: pid 2826413 finish time Thu Nov 23 08:04:13 2006 sendbackup: debug 1 pid 2836263 ruid 666 euid 666: start at Thu Nov 23 08:03:20 2006 sendbackup: version 2.5.1p2 Could not open conf file /opt/amanda/amanda1/etc/amanda/amanda-client.conf: No such file or directory Reading conf file /opt/amanda/amanda1/etc/amanda/stk_80-conf1/amanda-client.conf. sendbackup: debug 1 pid 2836263 ruid 666 euid 666: rename at Thu Nov 23 08:03:20 2006 sendbackup req: GNUTAR sub101 /data/mafalda/mafalda1/susanita/jen/anxiety_version1/sub101 1 2006:11:13:13:21:32 OPTIONS |;auth=bsdtcp;index; parsed request as: program `GNUTAR' disk `sub101' device `/data/mafalda/mafalda1/susanita/jen/anxiety_version1/sub101' level 1 since 2006:11:13:13:21:32 options `|;auth=bsdtcp;index;' sendbackup: start: yorick:sub101 lev 1 sendbackup-gnutar: time 0.208: doing level 1 dump as listed-incremental from '/opt/amanda/amanda1/var/amanda/gnutar-lists/yoricksub101_0' to '/opt/amanda/amanda1/var/amanda/gnutar-lists/yoricksub101_1.new' sendbackup-gnutar: time 53.057: doing level 1 dump from date: 2006-11-13 13:22:26 GMT sendbackup: time 53.246: started index creator: /usr/freeware/bin/tar -tf - 2/dev/null | sed -e 's/^\.//' sendbackup: time 53.254: spawning /opt/amanda/amanda1/libexec/runtar in pipeline sendbackup: argument list: runtar stk_80-conf1 gtar --create
upgrade issue, 2.5.1 to 2.5.1p2: things are worse than I thought
As a temporary measure, I have re-installed 2.5.1 on my server and it's working again, but I'd prefer to be running the same version everywhere. Well, I lied. amcheck worked OK, but amflush 2.5.1 refuses to flush the dumps made with 2.5.1p2 on the holding disk, saying: [EMAIL PROTECTED]:~$ amflush coolheads Scanning /nobackup/AMANDASPOOL... 20061125001502: found Amanda directory. Could not find any valid dump image, check directory. The directory has many dump files in it, and it looks OK. Is there a trick I should know? -- Steve Steven R. Newcomb, Consultant Coolheads Consulting Co-editor, Topic Maps International Standard (ISO/IEC 13250) Co-editor, draft Topic Maps -- Reference Model (ISO/IEC 13250-5) [EMAIL PROTECTED] http://www.coolheads.com direct: +1 540 951 9773 main: +1 540 951 9774 fax:+1 540 951 9775 208 Highview Drive Blacksburg, Virginia 24060 USA (Confidential to all US government personnel to whom this private letter is not addressed and who are reading it in the absence of a specific search warrant: You, along with the corrupt and pusillanimous 109th Congress, are co-conspiring to subvert the Constitution that you are sworn to defend. You can either refuse to commit this crime, or you can expect to suffer criminal sanctions in the future, when the current administration of the United States of America has been replaced by one that respects the rule of law. I do not envy you for having to make this difficult choice, but I urge you to make it wisely.) chg-multi.conf Description: Binary data amanda.conf Description: Binary data
upgrade issue, 2.5.1 to 2.5.1p2
I'm using chg-multi. It works under 2.5.1. When I upgraded to 2.5.1p2, it stopped working, even with no changes to any configuration file. With 2.5.1p2, the backups are made to the holding disk, but nothing gets written to tape. There's a warning: WARNING: No tapedev specified It's true that the tapedev parameter is not set in amanda.conf, but that's because, according to the documentation, it's not supposed to be set when using chg-multi. As a temporary measure, I have re-installed 2.5.1 on my server and it's working again, but I'd prefer to be running the same version everywhere. I'm attaching my amanda.conf and my chg-multi.conf, in case that helps. -- Steve Steven R. Newcomb, Consultant Coolheads Consulting Co-editor, Topic Maps International Standard (ISO/IEC 13250) Co-editor, draft Topic Maps -- Reference Model (ISO/IEC 13250-5) [EMAIL PROTECTED] http://www.coolheads.com direct: +1 540 951 9773 main: +1 540 951 9774 fax:+1 540 951 9775 208 Highview Drive Blacksburg, Virginia 24060 USA (Confidential to all US government personnel to whom this private letter is not addressed and who are reading it in the absence of a specific search warrant: You, along with the corrupt and pusillanimous 109th Congress, are co-conspiring to subvert the Constitution that you are sworn to defend. You can either refuse to commit this crime, or you can expect to suffer criminal sanctions in the future, when the current administration of the United States of America has been replaced by one that respects the rule of law. I do not envy you for having to make this difficult choice, but I urge you to make it wisely.) chg-multi.conf Description: Binary data amanda.conf Description: Binary data
Re: amandas group membership in FC6?
Gene Heskett wrote: Greetings; Despite the fact that the user 'amanda' is a member of the group 'disk', all compilations and new files generated by the user amanda seem to be owned by amanda:amanda instead of the expected amanda:disk. The end result is that many of my backup operations are failing because the amanda utility doesn't have perms to delete or write to files actually owned by amanda:disk. I just went thru all the directory trees amanda needs to access and chowned everything back the way its supposed to be, but then I built the 20061124 tarball just now, and everything is still owned by amanda:amanda. From my /etc/group file: disk:x:6:root,amanda So I blew it away, called up KUsr and verified that amanda was indeed a member of the group disk. Even deleted the user and re-added it and made sure this new copy of amanda was a member of the group disk. Then as amanda, I unpacked it again and rebuilt it, but I still have the same problem. Because none of the files are owned by amanda:disk, the end result is several megs of dead, can't do a thing code that I'd just as well not bother with the 'make install'. Anything that amanda has touched over the last 4 days since I started running it again has been converted to being owned by amanda:amanda, and if the file existed, and was to be deleted as part of the housekeeping, was not because the old file was owned by amanda:disk. So my backups are being slowly trashed because the indice files are not updatable. Whats the deal with FC6 and its owner:group handling? Am I setting up the user wrong or what? Perhaps something changed the amanda user's primary group in /etc/passwd? When new files are created, the user/group set are the ones in passwd, and /etc/group is only consulted by the system if the user is not the owner of the directory, then it checks if it is in the same group (assuming you have group write perms). Another possibility is that you are forcing the group amanda runs as in xinetd to be 'amanda' and not 'disk'. Frank -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: amandas group membership in FC6?
On Saturday 25 November 2006 20:42, Frank Smith wrote:
Gene Heskett wrote:
Greetings;
Despite the fact that the user 'amanda' is a member of the group
'disk', all compilations and new files generated by the user amanda
seem to be owned by amanda:amanda instead of the expected amanda:disk.
The end result is that many of my backup operations are failing
because the amanda utility doesn't have perms to delete or write to
files actually owned by amanda:disk.
I just went thru all the directory trees amanda needs to access and
chowned everything back the way its supposed to be, but then I built
the 20061124 tarball just now, and everything is still owned by
amanda:amanda.
From my /etc/group file:
disk:x:6:root,amanda
So I blew it away, called up KUsr and verified that amanda was indeed
a member of the group disk. Even deleted the user and re-added it and
made sure this new copy of amanda was a member of the group disk.
Then as amanda, I unpacked it again and rebuilt it, but I still have
the same problem. Because none of the files are owned by amanda:disk,
the end result is several megs of dead, can't do a thing code that I'd
just as well not bother with the 'make install'.
Anything that amanda has touched over the last 4 days since I started
running it again has been converted to being owned by amanda:amanda,
and if the file existed, and was to be deleted as part of the
housekeeping, was not because the old file was owned by amanda:disk.
So my backups are being slowly trashed because the indice files are
not updatable.
Whats the deal with FC6 and its owner:group handling? Am I setting up
the user wrong or what?
Perhaps something changed the amanda user's primary group in
/etc/passwd? When new files are created, the user/group set are
the ones in passwd, and /etc/group is only consulted by the system
if the user is not the owner of the directory, then it checks if it
is in the same group (assuming you have group write perms).
Another possibility is that you are forcing the group amanda runs as
in xinetd to be 'amanda' and not 'disk'.
I hadn't thought of that, but the amanda file in the xinetd.d dir is the
same one I used for FC2:
# default = off
#
# description: Part of the Amanda server package
# This is the list of daemons such it needs
service amanda
{
only_from = coyote.coyote.den # gene.coyote.den
disable = no
socket_type = dgram
protocol= udp
wait= yes
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amandad
server_args = -auth=bsd amdump amindexd amidxtaped
}
service amandaidx
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amindexd
}
service amidxtape
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amidxtaped
}
According to the amanda coders, this is correct usage. Is it not now?
Thanks Frank.
--
Cheers, Gene
There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
Re: amandas group membership in FC6?
Gene Heskett wrote:
On Saturday 25 November 2006 20:42, Frank Smith wrote:
Gene Heskett wrote:
Greetings;
Despite the fact that the user 'amanda' is a member of the group
'disk', all compilations and new files generated by the user amanda
seem to be owned by amanda:amanda instead of the expected amanda:disk.
The end result is that many of my backup operations are failing
because the amanda utility doesn't have perms to delete or write to
files actually owned by amanda:disk.
I just went thru all the directory trees amanda needs to access and
chowned everything back the way its supposed to be, but then I built
the 20061124 tarball just now, and everything is still owned by
amanda:amanda.
From my /etc/group file:
disk:x:6:root,amanda
So I blew it away, called up KUsr and verified that amanda was indeed
a member of the group disk. Even deleted the user and re-added it and
made sure this new copy of amanda was a member of the group disk.
Then as amanda, I unpacked it again and rebuilt it, but I still have
the same problem. Because none of the files are owned by amanda:disk,
the end result is several megs of dead, can't do a thing code that I'd
just as well not bother with the 'make install'.
Anything that amanda has touched over the last 4 days since I started
running it again has been converted to being owned by amanda:amanda,
and if the file existed, and was to be deleted as part of the
housekeeping, was not because the old file was owned by amanda:disk.
So my backups are being slowly trashed because the indice files are
not updatable.
Whats the deal with FC6 and its owner:group handling? Am I setting up
the user wrong or what?
Perhaps something changed the amanda user's primary group in
/etc/passwd? When new files are created, the user/group set are
the ones in passwd, and /etc/group is only consulted by the system
if the user is not the owner of the directory, then it checks if it
is in the same group (assuming you have group write perms).
So what group does amanda have in /etc/passwd (the number in the fourth
field)? See what that number maps to in /etc/group. I'm betting it
goes to an 'amanda' group and not the 'disk' group. It's also possible
that you have two amanda lines in your passwd file, or two groups in
/etc/group that map to the same number (or the same group name in twice
with two different numbers). In those cases the first match is what
the system uses, but it can certainly be confusing to debug if you
don't notice the other one.
Your system is finding an amanda group for the amanda user somewhere,
it's just a mater of finding out where it is getting it from. I would
suggest looking into whether you might have compiled it in, but I know
you always use your same build script, so I'll just mention it as a
possibility for future readers of the archives.
Another possibility is that you are forcing the group amanda runs as
in xinetd to be 'amanda' and not 'disk'.
I hadn't thought of that, but the amanda file in the xinetd.d dir is the
same one I used for FC2:
# default = off
#
# description: Part of the Amanda server package
# This is the list of daemons such it needs
service amanda
{
only_from = coyote.coyote.den # gene.coyote.den
disable = no
socket_type = dgram
protocol= udp
wait= yes
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amandad
server_args = -auth=bsd amdump amindexd amidxtaped
}
service amandaidx
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amindexd
}
service amidxtape
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amidxtaped
}
According to the amanda coders, this is correct usage. Is it not now?
That looks correct to me, so I'd look more into the passwd/group
files mentioned above.
Frank
--
Frank Smith [EMAIL PROTECTED]
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501
Re: amandas group membership in FC6?
On Saturday 25 November 2006 22:29, Frank Smith wrote:
Gene Heskett wrote:
On Saturday 25 November 2006 20:42, Frank Smith wrote:
Gene Heskett wrote:
Greetings;
Despite the fact that the user 'amanda' is a member of the group
'disk', all compilations and new files generated by the user amanda
seem to be owned by amanda:amanda instead of the expected
amanda:disk.
The end result is that many of my backup operations are failing
because the amanda utility doesn't have perms to delete or write to
files actually owned by amanda:disk.
I just went thru all the directory trees amanda needs to access and
chowned everything back the way its supposed to be, but then I built
the 20061124 tarball just now, and everything is still owned by
amanda:amanda.
From my /etc/group file:
disk:x:6:root,amanda
So I blew it away, called up KUsr and verified that amanda was
indeed a member of the group disk. Even deleted the user and
re-added it and made sure this new copy of amanda was a member of
the group disk.
Then as amanda, I unpacked it again and rebuilt it, but I still
have the same problem. Because none of the files are owned by
amanda:disk, the end result is several megs of dead, can't do a
thing code that I'd just as well not bother with the 'make install'.
Anything that amanda has touched over the last 4 days since I
started running it again has been converted to being owned by
amanda:amanda, and if the file existed, and was to be deleted as
part of the housekeeping, was not because the old file was owned by
amanda:disk. So my backups are being slowly trashed because the
indice files are not updatable.
Whats the deal with FC6 and its owner:group handling? Am I setting
up the user wrong or what?
Perhaps something changed the amanda user's primary group in
/etc/passwd? When new files are created, the user/group set are
the ones in passwd, and /etc/group is only consulted by the system
if the user is not the owner of the directory, then it checks if it
is in the same group (assuming you have group write perms).
So what group does amanda have in /etc/passwd (the number in the fourth
field)?
It was originally the same as amanda, 501::501 but I've edited that to
501::6 now. I've also SGID'd the amanda home dir. It ran alright this
morning I believe. But the real answer will be when Jean-Louis releases
the next snapshot.
See what that number maps to in /etc/group. I'm betting it
goes to an 'amanda' group and not the 'disk' group.
There was not, and still is not, a group named amanda, just the amanda
entry in the disk line.
It's also possible
that you have two amanda lines in your passwd file, or two groups in
/etc/group that map to the same number (or the same group name in twice
with two different numbers). In those cases the first match is what
the system uses, but it can certainly be confusing to debug if you
don't notice the other one.
Nope, just one.
Your system is finding an amanda group for the amanda user somewhere,
it's just a mater of finding out where it is getting it from. I would
suggest looking into whether you might have compiled it in, but I know
you always use your same build script, so I'll just mention it as a
possibility for future readers of the archives.
Another possibility is that you are forcing the group amanda runs as
in xinetd to be 'amanda' and not 'disk'.
I hadn't thought of that, but the amanda file in the xinetd.d dir is
the same one I used for FC2:
# default = off
#
# description: Part of the Amanda server package
# This is the list of daemons such it needs
service amanda
{
only_from = coyote.coyote.den # gene.coyote.den
disable = no
socket_type = dgram
protocol= udp
wait= yes
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amandad
server_args = -auth=bsd amdump amindexd amidxtaped
}
service amandaidx
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amindexd
}
service amidxtape
{
disable = no
socket_type = stream
protocol= tcp
wait= no
user= amanda
group = disk
groups = yes
server = /usr/local/libexec/amidxtaped
}
According to the amanda coders, this is correct usage. Is it not now?
That looks correct to me, so I'd look more into the passwd/group
files mentioned above.
Frank
Thanks Frank. We'll see if its fixed in due time I guess.
--
Cheers, Gene
There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
Yahoo.com and
