Re: [Bacula-users] [Bacula-devel] Misc
Hello, On 3/2/2007 12:08 AM, Steen wrote: > Torsdag 01 marts 2007 21:07 skrev Arno Lehmann: ... >>>So if anyone's interested, I have a python script which will actually >>>do non-blocking IO (gasp, shock) over the FIFOs, waiting until any >>>one of them can accept data and only *then* launching connecting to >>>the database to fill the FIFO. It will also do the converse for >>>restoring data back to the databases, noticing newly created FIFOs >>>(created by a Bacula restore job) in the directory and trying to grab >>>data from them. >> >>Currently, I'm not interested but this sounds like areally great >>enhancement! >> >>You wrote that for PostgreSQL, but I suppose adapting it to other >>databases or whatever would not be too much of a problem, so I suggest >>you post it to the wiki at http://paramount.ind.wpi.edu/wiki/doku.php or >>mail to the list. I'm sure that script will be helpful to many users, >>and I suppose it could be added to the user-contributed examples in the >>source code. >> >> >>>Note that this is definite overkill if you're okay >>>with doing one dump per server of all active databases. >> >>Depends on the available dump space vs. database size... > > I am reading this thread with interest and trying to understand. I thought > that the databases would dump 'through' the FIFO directly to Bacula - thus > eliminating the need for dump space - am I wrong here? That's what can be achieved with a FIFO. The simpler solution is to dump to file and then back up that file. This is what the "standard" catalog backup script does, for example. Arno > Steen > >>Arno >> >> >>>-- >>>--Darien A. Hager >>>[EMAIL PROTECTED] >>> >>> >>> >>>- >>>Take Surveys. Earn Cash. Influence the Future of IT >>>Join SourceForge.net's Techsay panel and you'll get the chance to share >>>your opinions on IT & business topics through brief surveys-and earn cash >>>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >>>___ >>>Bacula-users mailing list >>>Bacula-users@lists.sourceforge.net >>>https://lists.sourceforge.net/lists/listinfo/bacula-users > > -- IT-Service Lehmann[EMAIL PROTECTED] Arno Lehmann http://www.its-lehmann.de - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Misc
Torsdag 01 marts 2007 21:07 skrev Arno Lehmann: > Hi, > > On 3/1/2007 9:00 PM, Darien Hager wrote: > > On Mar 1, 2007, at 6:50 AM, Ryan Novosielski wrote: > >>>On Thursday 01 March 2007 10:32:49 Kern Sibbald wrote: > >>> > >>>this item does indeed work very stable. I've been using this in > >>>production for > >>>exporting large Oracle databases (100+ GBytes each) once per week > >>>for about > >>>two years. I've never encountered stability issue with the FIFOS. :-) > >> > >>I would love, if you have the time, to write a brief mention about how > >>this is done. Perhaps it could be posted somewhere (if it isn't > >>already)? > > > > My experience with FIFO database dumps: > > > > > > Basics: > > ... > > > So if anyone's interested, I have a python script which will actually > > do non-blocking IO (gasp, shock) over the FIFOs, waiting until any > > one of them can accept data and only *then* launching connecting to > > the database to fill the FIFO. It will also do the converse for > > restoring data back to the databases, noticing newly created FIFOs > > (created by a Bacula restore job) in the directory and trying to grab > > data from them. > > Currently, I'm not interested but this sounds like areally great > enhancement! > > You wrote that for PostgreSQL, but I suppose adapting it to other > databases or whatever would not be too much of a problem, so I suggest > you post it to the wiki at http://paramount.ind.wpi.edu/wiki/doku.php or > mail to the list. I'm sure that script will be helpful to many users, > and I suppose it could be added to the user-contributed examples in the > source code. > > > Note that this is definite overkill if you're okay > > with doing one dump per server of all active databases. > > Depends on the available dump space vs. database size... I am reading this thread with interest and trying to understand. I thought that the databases would dump 'through' the FIFO directly to Bacula - thus eliminating the need for dump space - am I wrong here? Steen > > Arno > > > -- > > --Darien A. Hager > > [EMAIL PROTECTED] > > > > > > > > - > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > your opinions on IT & business topics through brief surveys-and earn cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > ___ > > Bacula-users mailing list > > Bacula-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/bacula-users -- Regards Steen - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Misc
Hi, On 3/1/2007 9:00 PM, Darien Hager wrote: > On Mar 1, 2007, at 6:50 AM, Ryan Novosielski wrote: > > >>>On Thursday 01 March 2007 10:32:49 Kern Sibbald wrote: >>> >>>this item does indeed work very stable. I've been using this in >>>production for >>>exporting large Oracle databases (100+ GBytes each) once per week >>>for about >>>two years. I've never encountered stability issue with the FIFOS. :-) >> >>I would love, if you have the time, to write a brief mention about how >>this is done. Perhaps it could be posted somewhere (if it isn't >>already)? > > > My experience with FIFO database dumps: > > > Basics: ... > So if anyone's interested, I have a python script which will actually > do non-blocking IO (gasp, shock) over the FIFOs, waiting until any > one of them can accept data and only *then* launching connecting to > the database to fill the FIFO. It will also do the converse for > restoring data back to the databases, noticing newly created FIFOs > (created by a Bacula restore job) in the directory and trying to grab > data from them. Currently, I'm not interested but this sounds like areally great enhancement! You wrote that for PostgreSQL, but I suppose adapting it to other databases or whatever would not be too much of a problem, so I suggest you post it to the wiki at http://paramount.ind.wpi.edu/wiki/doku.php or mail to the list. I'm sure that script will be helpful to many users, and I suppose it could be added to the user-contributed examples in the source code. > Note that this is definite overkill if you're okay > with doing one dump per server of all active databases. Depends on the available dump space vs. database size... Arno > > -- > --Darien A. Hager > [EMAIL PROTECTED] > > > > - > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users -- IT-Service Lehmann[EMAIL PROTECTED] Arno Lehmann http://www.its-lehmann.de - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Misc
On Mar 1, 2007, at 6:50 AM, Ryan Novosielski wrote:
>>
>> On Thursday 01 March 2007 10:32:49 Kern Sibbald wrote:
>>
>> this item does indeed work very stable. I've been using this in
>> production for
>> exporting large Oracle databases (100+ GBytes each) once per week
>> for about
>> two years. I've never encountered stability issue with the FIFOS. :-)
>
> I would love, if you have the time, to write a brief mention about how
> this is done. Perhaps it could be posted somewhere (if it isn't
> already)?
My experience with FIFO database dumps:
Basics: On backup, when Bacula encounters an explicit FIFO (with
readfifo=yes), It tries to read data from that FIFO until end-of-
file. On restore, if the FIFO doesn't exist, bacula creates it. It
then waits (1 min timeout) for something else to read from the FIFO.
If the mystery stranger connects, it just pours all the data it saved
back into the FIFO and moves on.
1. A 'gotcha': Your fileset must explicitly list the FIFOs as
individual files, they cannot be something it comes across via
recursion or regular expressions. One simple way of doing this for
all FIFOS in a directory is to run a command on the remote machine
that lists all fifos. Example fileset (replace directory name as
needed):
FileSet {
Name = "DB FIFOs"
Include {
Options {
readfifo=yes
}
# Documentation says fifos must be "explicitly"
# Mentioned. You can't just specify the directory
# That they'll all be created in, or "readfifo"
won't apply.
# This is a workaround for that limitation: Run a
script to list them.
File = "\\|bash -c \"find /tmp/bacula_fifos/ -type p
-maxdepth 1\""
}
}
2. When Bacula hits the FIFOs, it will wait ~60 seconds on each file
for someone on the other end to start giving data (when Job=Backup)
or consuming it (Job=Restore). Thus you must start something in the
background to supply or take data from them.
RunScript {
# Uncomment appropriate command for desired effect, either backing
up, restoring to DB, or restoring to real files
#Command = "/etc/bacula/launchwrapper.sh restoretodb /tmp/
bacula_fifos/"
#Command = "/etc/bacula/launchwrapper.sh restoretofolder /var/
eas_restore/tmp/bacula_fifos/ /var/eas_restore/bacula_fifodump/"
#Command = "/etc/bacula/launchwrapper.sh backup /tmp/bacula_fifos/"
RunsWhen = Before
AbortJobOnError = yes
RunsOnClient = yes # BUG in early vers of Bacula 2.01, cannot use
uppercase Y in Yes for RunsOnClient.
}
RunScript {
# Clean out our FIFOs
Command = "/etc/bacula/launchwrapper.sh clean /tmp/bacula_fifos/"
RunsWhen = After
RunsOnSuccess = Yes
RunsOnFailure = Yes
RunsOnClient = yes
}
3. I had a bit of a problem with our set up, because we have many
databases per server which I wanted to handle individually. Spawning
lots of pg_dump processes writing to FIFOs (and waiting for Bacula to
come along) doesn't work, because you exhaust the connection pool and
only the first X databases will dump successfully.
So if anyone's interested, I have a python script which will actually
do non-blocking IO (gasp, shock) over the FIFOs, waiting until any
one of them can accept data and only *then* launching connecting to
the database to fill the FIFO. It will also do the converse for
restoring data back to the databases, noticing newly created FIFOs
(created by a Bacula restore job) in the directory and trying to grab
data from them. Note that this is definite overkill if you're okay
with doing one dump per server of all active databases.
--
--Darien A. Hager
[EMAIL PROTECTED]
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Misc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephan Ebelt wrote: > Hello, > > On Thursday 01 March 2007 10:32:49 Kern Sibbald wrote: > > [...] > >> 2. Doing a backup/restore from/to a FIFO (as defined in a FileSet resource) >> is supported and from my tests works perfectly well in Bacula 2.0.x. In >> some previous versions there were some problems, but they have long since >> been corrected. > > this item does indeed work very stable. I've been using this in production > for > exporting large Oracle databases (100+ GBytes each) once per week for about > two years. I've never encountered stability issue with the FIFOS. :-) I would love, if you have the time, to write a brief mention about how this is done. Perhaps it could be posted somewhere (if it isn't already)? - -- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer III |$&| |__| | | |__/ | \| _| |[EMAIL PROTECTED] - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF5ughmb+gadEcsb4RAtGVAKCQY/3mcFsViPZaWtccLZWhKetBcQCffpS9 RBfUY1o7SAhUCjTAl6Bw+GE= =SUvR -END PGP SIGNATURE- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Misc
Hello, On Thursday 01 March 2007 10:32:49 Kern Sibbald wrote: [...] > 2. Doing a backup/restore from/to a FIFO (as defined in a FileSet resource) > is supported and from my tests works perfectly well in Bacula 2.0.x. In > some previous versions there were some problems, but they have long since > been corrected. this item does indeed work very stable. I've been using this in production for exporting large Oracle databases (100+ GBytes each) once per week for about two years. I've never encountered stability issue with the FIFOS. :-) There was only one issue during restores (not a stability one though): the FIFO permissions were -rw--- and owned by root. Therefore the 'oracle' user was not able to read it for importing... though, I do not remember whether the reason was bacula or my own scripts?! Anyhow, bacula does not seem to wait long when there is nothing reading from the FIFO. So one must be quick with fixing the permissions... all my saying goes for 1.38.11 and previous versions. Unfortunately I can not tell for 2.0.x since I no longer have control over this installation... regards, Stephan - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
