Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
David Cantrell wrote: One thing to bear in mind is that while rsync -a does preserve permission bits, it does *not* preserve owner and group information. [snip] I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. So use zip. That's what I use for backing up two small sites I have to penderel -- zip up the files with the -u (update) option so that only changed and new files get written, then rsync the zip file over. Since zip format, as opposed to (say) compressed tar, keeps the bits of a zip file corresponding to unchanged files the same, rsync works well. I can't access the place from here, but from memory it goes something like this: #!/bin/bash cd /home/newton find . -print0 | perl -ln0e 'print unless ' | \ zip -9uo@ newton rsync --rsh=ssh newton.zip [EMAIL PROTECTED]:. This compresses with best compression (-9), with only new and updated files (-u), gives the resulting zip file the date of the newest member of the archive (-o) and takes filenames from stdin (-@). Although I must admit I haven't tried to recover data from such a backup, I believe that Unix zip preserves uid and gid information. BICBW. Anyway, you could also use tar, which does definitely preserve such information, if you -c(reate) a new file each time and don't compress it afterwards (I think -u(pdate) doesn't remove the first copy for changed files). The result should still rsync well, and you'll only need twice as much space as normal. Which is bad if you're filling up your hard disk but may not be a problem in practice (depends on you). Cheers, Philip -- Philip Newton [EMAIL PROTECTED] All opinions are my own, not my employer's. If you're not part of the solution, you're part of the precipitate.
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, Jan 09, 2002 at 10:27:00PM +, David Cantrell wrote: so that I have a record. I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. I don't follow this.. That is whole point of rsync, it'll send over what it deduces are the changed parts, not the entire file. Of course, you need to rsync the .tar, not .tar.gz Paul
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
Newton, == Newton, Philip [EMAIL PROTECTED] writes: Newton, David Cantrell wrote: One thing to bear in mind is that while rsync -a does preserve permission bits, it does *not* preserve owner and group information. Huh? as root, it seems to preserve everything. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Thu, Jan 10, 2002 at 09:52:09AM +0100, Newton, Philip wrote: David Cantrell wrote: I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. So use zip. zip does not store ownership information. Although I must admit I haven't tried to recover data from such a backup Do let me know when you have to rely on them. I could do with a laugh :-) I believe that Unix zip preserves uid and gid information. BICBW. You is :-) It can't preserve it because the zip format, which was designed for Mess DOS, does not havae sufficient fields. Anyway, you could also use tar, which does definitely preserve such information, if you -c(reate) a new file each time and don't compress it afterwards (I think -u(pdate) doesn't remove the first copy for changed files)... I'll look into it. -- David Cantrell | [EMAIL PROTECTED] | http://www.cantrell.org.uk/david Every normal man must be tempted at times to spit on his hands, hoist the black flag, and begin slitting throats. -- H. L. Mencken
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Thu, Jan 10, 2002 at 11:38:30AM +, David Cantrell wrote: On Thu, Jan 10, 2002 at 09:52:09AM +0100, Newton, Philip wrote: David Cantrell wrote: I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. So use zip. zip does not store ownership information. Although I must admit I haven't tried to recover data from such a backup Do let me know when you have to rely on them. I could do with a laugh :-) Which is what I was thinking - what use is a backup when you haven't tested it in peacetime to see if you can recover data from it? I believe that Unix zip preserves uid and gid information. BICBW. You is :-) It can't preserve it because the zip format, which was designed for Mess DOS, does not havae sufficient fields. As standard. But there's an extra data field (I forget its proper name) that can be used to store arbitrary small tagged blocks of binary data with each file, and I can remember that there is a tag allocated for storing unix timestamps, which I believe infozip's zip unzip use. I cannot offhand remember what if any other unix related blocks were defined. But your previous comment holds - test to see if your (well, Philip's in this case) backups can be restored. If they can't, report yourself to trading standards for calling them backups. (Although Philip is in Germany, so other rules may apply) Nicholas Clark
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
David Cantrell wrote: On Thu, Jan 10, 2002 at 09:52:09AM +0100, Newton, Philip wrote: David Cantrell wrote: I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. So use zip. zip does not store ownership information. Hm. I'm not completely convinced, though I'll agree that v2.2 which I tested just now does not -- however, it appears that it is capable of doing so. Note also this bit from `man zip`: -X Do not save extra file attributes (Extended Attributes on OS/2, uid/gid and file times on Unix). Which sounds as if the plan is to include uid/gid by default. I believe that Unix zip preserves uid and gid information. BICBW. You is :-) It can't preserve it because the zip format, which was designed for Mess DOS, does not havae sufficient fields. Not quite. There's an extra field, where you can put all sorts of information. The field apparently starts off with two key bytes which say what sort of information is stored in each subfield of the extra field. Here's some output from `zipinfo -v zipfilename archive member` on a random zipfile I just created on a HP-UX system: file system or operating system of origin:Unix version of encoding software: 2.2 minimum file system compatibility required: MS-DOS, OS/2 or NT FAT minimum software version required to extract: 2.0 [...] file last modified on (DOS date/time):2000 Mar 30 15:02:10 file last modified on (UT extra field modtime): 2000 Mar 30 15:02:09 local file last modified on (UT extra field modtime): 2000 Mar 30 13:02:09 UTC [...] length of extra field:13 bytes [...] Unix file attributes (100664 octal): -rw-rw-r-- MS-DOS file attributes (00 hex): none The central-directory extra field contains: - A subfield with ID 0x5455 (universal time) and 5 data bytes. The local extra field has UTC/GMT modification/access times. - A subfield with ID 0x7855 (Unix UID/GID) and 0 data bytes. So the extra field stores some Unix time info (which are also required by the MS-DOS origin, which has only 5 bits for the seconds subfield of the timestamp and therefore a 2-second granularity, just like FAT), and apparently there's provision for UID/GID. Not sure why there are 0 data bytes, however. Cheers, Philip -- Philip Newton [EMAIL PROTECTED] All opinions are my own, not my employer's. If you're not part of the solution, you're part of the precipitate.
Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, 9 Jan 2002, Chris Ball wrote:
Mark mentions looking for co-lo at a price/scheme similar to Mailbox's.
I'd recommend http://blackcatnetworks.co.uk/colo.html - AIUI, it's in
RedBus and at the same price as Mailbox's deal. The Black Cat people
are nice, too.
I assume you mean http://www.blackcatnetworks.co.uk/colo.html
50ukp/m x 12 = 600 ukp/a
mailbox cost = 775 ukp/a (375 + 250 + 150)
775 - 600 = 175 ukp to spend on extra bandwidth
175 / 5ukp/Gb = 35Gb extra bandwidth
12*10GB = 120GB free transfer = 155Gb transfer before black cats are
more expensive
155*1024*1024/256/60/60/24 = 7.348 days at maximum transfer rate to become
more expensive
Hmm. Not bad...but...
Assuming 1GB of actual data backed up once a week (both to and from
box)
1GB = 8Gb
155Gb/8Gb = 20 weeks till bandwidth limit exceeded.
Arse.
Mark.
(Yes, yes, I know I can use rsync to somewhat reduce the bandwidth use,
but I haven't even counted mail and web and other use yet)
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, 9 Jan 2002, Mark Fowler wrote: Assuming 1GB of actual data backed up once a week (both to and from box) You're really not wanting to do that, I suspect. If you're actually getting 64k/sec between your colo and your backup server, that'll take 24 hours (64k/s for a whole month works out as about 20GB) If you happen to have fast networked machines elsewhere, and can get say 256k/s then obviously that reduces to a mere 6 hours. I'm guessing that if you had somewhere with enough bandwidth to spare for anything bigger than that, then you wouldn't be looking for a colo elsewhere. Tape is your friend, leave them a box of tapes and a box of prepaid envelopes, get them to post you one when you feel the need for an off-site backup. Or host somewhere you can drop in on and pick up the backups yourself. the hatter
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, 9 Jan 2002, the hatter wrote:
I'm guessing that if you had
somewhere with enough bandwidth to spare for anything bigger than that,
then you wouldn't be looking for a colo elsewhere.
Nope. Bandwidth is easy, serving from it less so. And note that I said
backups *to* and from the box.
a) Direct backup to and from other co-lo boxes (to be negotiated)
b) Backup using idle bandwidth at work (where we have very fat pipes)
Later.
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, Jan 09, 2002 at 10:51:20AM +, the hatter wrote: On Wed, 9 Jan 2002, Mark Fowler wrote: Assuming 1GB of actual data backed up once a week (both to and from box) You're really not wanting to do that, I suspect. I backup my server by rsync. That accounts for well under 100Mb of traffic a day. WELL under. It's only the biggest most insane days that go above that. Most days it's more like 20Mb. -- David Cantrell | [EMAIL PROTECTED] | http://www.cantrell.org.uk/david If a job's worth doing, it's worth dieing for
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
Mark Fowler [EMAIL PROTECTED] writes: 155*1024*1024/256/60/60/24 = 7.348 days at maximum transfer rate to become pedant s/1024/1000/g /pedant -- Steve Mynott [EMAIL PROTECTED]
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, 9 Jan 2002, David Cantrell wrote:
I backup my server by rsync. That accounts for well under 100Mb of
traffic a day. WELL under. It's only the biggest most insane days that
go above that. Most days it's more like 20Mb.
Cool. I shall definatly be bothering you for help with that then... ;-)
On the most insane day I create well over 10MB of data. On an average day
I probably get at least half a megabyte of mail alone...
I guess what I'm worried about is potential billing for unplanned useage.
I like the fact with my current deal that I can use as much bandwidth as I
am able to and it won't effect my billing. If I choose to do a full
backup of my box via dump and ssh, it may take a day or so, but it'll do
it for free. If I want to run X off my box, then it's a little slow but
works for emacs. If I want to fetch 100MB file to my box and then scp it
down (as blue younder's transproxy is being shite) then I can...
Maybe I'm being stuck in my ways. Maybe I just need to learn to be a bit
more conservative over my bandwidth useage.
For the record, I've done lots and lots of looking around on the net, and
blackcatsnetworks are pretty competative. dipsy knows about my results
(keyed under $isp or $isp cost) if you want to look.
*sigh*
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, Jan 09, 2002 at 02:57:55PM +, robin szemeti wrote: On Wednesday 09 January 2002 14:26, Mark Fowler wrote: If I choose to do a full backup of my box via dump and ssh, it may take a day or so, but it'll do it for free. once you get sorted with rsync you'll never need to do this ... because rsync only copies down the changed bits, you can do it pretty much as often as you like with no huge bandwidth penalty ... That's fine if you're working with a typical filesystem, but what if you're working with database tables? I'd normally dump the tables to a file and backup the dump. However, I guess you'd end up rsyncing the whole database if you did this. I guess you could archive transaction logs, but for a frequently changing database the logs will get quite large. Any alternatives? Tom
RE: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
Tom Hukins [mailto:[EMAIL PROTECTED]] wrote: That's fine if you're working with a typical filesystem, but what if you're working with database tables? I'd normally dump the tables to a file and backup the dump. However, I guess you'd end up rsyncing the whole database if you did this. I guess you could archive transaction logs, but for a frequently changing database the logs will get quite large. Any alternatives? The database vendor will provide tools to sync and backup databases. Depending on the level of sophistication required, this can be done by offlining a mirror copy of the database - needed for 24x7 availability if this is a requirement. The database usually also provides open transaction recovery journalling, and after image journalling. The former permits the database to be rolled back to a consistent state in the event of a system failure, and the latter allows replay of transactions from a sync point (a backup snapshot). Don't know if this helps. I suggest looking on the DB vendor's website for more technical info. Ivor. --- The information contained in this e-mail is confidential and solely for the intended addressee(s). Unauthorised reproduction, disclosure, modification, and/or distribution of this email may be unlawful. If you have received this email in error, please notify the sender immediately and delete it from your system. The views expressed in this message do not necessarily reflect those of LIFFE (Holdings) Plc or any of its subsidiary companies. ---
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
Tom Hukins [EMAIL PROTECTED] writes: That's fine if you're working with a typical filesystem, but what if you're working with database tables? I'd normally dump the tables to a file and backup the dump. However, I guess you'd end up rsyncing the whole database if you did this. I guess you could archive transaction logs, but for a frequently changing database the logs will get quite large. Like you say, dump the tables to a file. Then use rsync on the dump file. Of course, one must also remember to /not/ rsync the actual database. :) -Dom -- | Semantico: creators of major online resources | | URL: http://www.semantico.com/ | | Tel: +44 (1273) 72 | | Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |
Re: Co-lo, was Re: London.pm List Weekly Summary 2001-12-31
On Wed, Jan 09, 2002 at 02:57:55PM +, robin szemeti wrote: On Wednesday 09 January 2002 14:26, Mark Fowler wrote: If I choose to do a full backup of my box via dump and ssh, it may take a day or so, but it'll do it for free. rsync -avuv -e ssh \ my.host.somwhere.com:/ \ /backups/somewhere_com/ One thing to bear in mind is that while rsync -a does preserve permission bits, it does *not* preserve owner and group information. When I do my backups, I do ... ls -AlR /bin /boot /etc [tum to tum[ /ls -Alr rsync ... so that I have a record. I could use tar to pack all the data up, complete with ownership information, but of course, then I would be backing up a load of unchanged data which is wasteful of bandwidth. -- David Cantrell | [EMAIL PROTECTED] | http://www.cantrell.org.uk/david This is a signature. There are many like it but this one is mine.
