Re: [GENERAL] need for in-place upgrades (was Re: State of

[Christopher Browne]

In the last exciting episode, [EMAIL PROTECTED] (Ron Johnson) wrote:
> On Sun, 2003-09-14 at 14:17, Christopher Browne wrote:
>>  discusses how to use their hardware and
>> software products to do terabytes of backups in an hour.  They sell a
>> software product called "Alexandria" that knows how to (at least
>> somewhat) intelligently backup SAP R/3, Oracle, Informix, and Sybase
>> systems.  (When I was at American Airlines, that was the software in
>> use._
>
> HP, Hitachi, and a number of other vendors make similar hardware.
>
> You mean the database vendors don't build that parallelism into
> their backup procedures?

They don't necessarily build every conceivable bit of possible
functionality into the backup procedures they provide, if that's what
you mean.

Of thee systems mentioned, I'm most familiar with SAP's backup
regimen; if you're using it with Oracle, you'll use tools called
"brbackup" and "brarchive", which provide a _moderately_ sophisticated
scheme for dealing with backing things up.

But if you need to do something wild, involving having two servers
each having 8 tape drives on a nearby servers that are used to manage
backups for a whole cluster of systems, including a combination of OS
backups, DB backups, and application backups, it's _not_ reasonable to
expect one DB vendor's backup tools to be totally adequate to that.

Alexandria (and similar software) certainly needs tool support from DB
makers to allow them to intelligently handle streaming the data out of
the databases.

At present, this unfortunately _isn't_ something PostgreSQL does, from
two perspectives:

 1.  You can't simply keep the WALs and reapply them in order to bring
 a second database up to date;

 2.  A pg_dump doesn't provide a way of streaming parts of the
 database in parallel, at least not if all the data is in
 one database.  (There's some nifty stuff in eRServ that
 might eventually be relevant, but probably not yet...)

There are partial answers:

 - If there are multiple databases, starting multiple pg_dump
   sessions provides some useful parallelism;

 - A suitable logical volume manager may allow splitting off
   a copy atomically, and then you can grab the resulting data
   in "strips" to pull it in parallel.

Life isn't always perfect.

>> Generally, this involves having a bunch of tape drives that are
>> simultaneously streaming different parts of the backup.
>> 
>> When it's Oracle that's in use, a common strategy involves
>> periodically doing a "hot" backup (so you can quickly get back to a
>> known database state), and then having a robot tape drive assigned
>> to regularly push archive logs to tape as they are produced.
>
> Rdb does the same thing.  You mean DB/2 can't/doesn't do that?

I haven't the foggiest idea, although I would be somewhat surprised if
it doesn't have something of the sort.
-- 
(reverse (concatenate 'string "moc.enworbbc" "@" "enworbbc"))
http://www.ntlug.org/~cbbrowne/wp.html
Rules of  the Evil Overlord #139. "If  I'm sitting in my  camp, hear a
twig  snap, start  to  investigate, then  encounter  a small  woodland
creature, I  will send out some scouts  anyway just to be  on the safe
side. (If they disappear into the foliage, I will not send out another
patrol; I will break out napalm and Agent Orange.)"


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [GENERAL] need for in-place upgrades (was Re: State of

[Ron Johnson]

On Sun, 2003-09-14 at 14:17, Christopher Browne wrote:
> After a long battle with technology,[EMAIL PROTECTED] (Martin Marques), an 
> earthling, wrote:
> > El Dom 14 Sep 2003 12:20, Lincoln Yeoh escribió:
> >> >At 07:16 PM 9/13/2003 -0400, Lamar Owen wrote:
[snip]
> Certainly there are backup systems designed to cope with those sorts
> of quantities of data.  With 8 tape drives, and a rack system that
> holds 200 cartridges, you not only can store a HUGE pile of data, but
> you can push it onto tape about as quickly as you can generate it.
> 
>  discusses how to use their hardware and
> software products to do terabytes of backups in an hour.  They sell a
> software product called "Alexandria" that knows how to (at least
> somewhat) intelligently backup SAP R/3, Oracle, Informix, and Sybase
> systems.  (When I was at American Airlines, that was the software in
> use._

HP, Hitachi, and a number of other vendors make similar hardware.

You mean the database vendors don't build that parallelism into
their backup procedures?

> Generally, this involves having a bunch of tape drives that are
> simultaneously streaming different parts of the backup.
> 
> When it's Oracle that's in use, a common strategy involves
> periodically doing a "hot" backup (so you can quickly get back to a
> known database state), and then having a robot tape drive assigned to
> regularly push archive logs to tape as they are produced.

Rdb does the same thing.  You mean DB/2 can't/doesn't do that?

[snip]
> None of this is particularly cheap or easy; need I remind gentle
> readers that if you can't afford that, then you essentially can't
> afford to claim "High Availability?"

-- 
-
Ron Johnson, Jr. [EMAIL PROTECTED]
Jefferson, LA USA

"(Women are) like compilers. They take simple statements and 
make them into big productions."
Pitr Dubovitch


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [GENERAL] need for in-place upgrades (was Re: State of

[Christopher Browne]

After a long battle with technology,[EMAIL PROTECTED] (Martin Marques), an earthling, 
wrote:
> El Dom 14 Sep 2003 12:20, Lincoln Yeoh escribió:
>> >At 07:16 PM 9/13/2003 -0400, Lamar Owen wrote:
>> >'migration' server.  And I really don't want to think about dump/restore
>> >of 100TB (if PostgreSQL actually stores the image files, which it might).
>>
>> Hmm. Just curious, do people generally backup 100TB of data, or once most
>> reach this point they have to hope that it's just hardware failures they'll
>> deal with and not software/other issues?
>
> Normally you would have a RAID with mirroring and CRC, so that if one of the 
> disks in the array of disks falls, the system keeps working. You can even 
> have hot-pluggable disks, so you can change the disk that is broken without 
> rebooting.
>
> You can also have a hot backup using eRServ (Replicate your DB server on a 
> backup server, just in case).

In a High Availability situation, there is little choice but to create
some form of "hot backup."  And if you can't afford that, then reality
is that you can't afford to pretend to have "High Availability."

>> 100TB sounds like a lot of backup media and time... Not to mention
>> ensuring that the backups will work with available and functioning
>> backup hardware.
>
> I don't know, but there may be backup systems for that amount of
> space. We have just got some 200Gb tape devices, and they are about
> 2 years old. With a 5 tape robot, you have 1TB of backup.

Certainly there are backup systems designed to cope with those sorts
of quantities of data.  With 8 tape drives, and a rack system that
holds 200 cartridges, you not only can store a HUGE pile of data, but
you can push it onto tape about as quickly as you can generate it.

 discusses how to use their hardware and
software products to do terabytes of backups in an hour.  They sell a
software product called "Alexandria" that knows how to (at least
somewhat) intelligently backup SAP R/3, Oracle, Informix, and Sybase
systems.  (When I was at American Airlines, that was the software in
use._

Generally, this involves having a bunch of tape drives that are
simultaneously streaming different parts of the backup.

When it's Oracle that's in use, a common strategy involves
periodically doing a "hot" backup (so you can quickly get back to a
known database state), and then having a robot tape drive assigned to
regularly push archive logs to tape as they are produced.

That would more or less resemble taking a "consistent filesystem
backup" of a PG database, and then saving the sequence of WAL files.
(The disanalogies are considerable; that should improve at least a
_little_ once PITR comes along for PostgreSQL...)

None of this is particularly cheap or easy; need I remind gentle
readers that if you can't afford that, then you essentially can't
afford to claim "High Availability?"
-- 
select 'cbbrowne' || '@' || 'cbbrowne.com';
http://www.ntlug.org/~cbbrowne/nonrdbms.html
Who's afraid of ARPA?

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings