Re: [BackupPC-users] "Clusters" of hosts
Jeffrey J. Kosowsky wrote at about 11:38:30 -0400 on Wednesday, June 10, 2009:
> Les Mikesell wrote at about 10:23:10 -0500 on Wednesday, June 10, 2009:
> > John Rouillard wrote:
> > > On Sat, May 30, 2009 at 10:16:33PM +0200, Pieter Wuille wrote:
> > >> I don't know how common this usage is, but in our setup we have a lot
> of
> > >> backuppc "hosts" that are physically located on a few machines only. It
> > >> would be nice if it were possible to allow hosts on different machines
> to
> > >> be backupped simultaneously, but prevent simultaneous backups(dumps) of
> > >> hosts on the same machine.
> > >>
> > >> Any thoughts?
> > >
> > > If you have a way of mapping the host names to a physical machine, you
> > > can use my queing/locking strategy described in:
> > >
> > >
> > >
> http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg13698.html
> > >
> > > Create one queue/semaphore per physical machine and have the
> > > $Conf{DumpPreUserCmd} command exit with an error if it can't get a
> > > slot/lock (also you will have to set $Conf{UserCmdCheckStatus} = 1;).
> >
> > As a feature request, I think it would be nice to have a way to add
> > hosts to groups, then limit how many in each group the scheduler would
> > start at once. There are several scenarios where this is needed to
> > avoid overloading some common reasource - like a low-bandwidth link as
> > well as sharing a physical host or filesystem.
> >
>
> I think the notion of host groups is a good idea. Even more generally,
> it would be nice to be able to define config files at the group level
> rather than the current choice between the default config.pl file and
> host-specific config files.
>
> For example, this would allow one to define a config file for Linux
> vs. Windows machines or for desktops vs. notebooks or for critical
> machines vs. less critical machines (I know you can currently do this
> in a kludgey fashion using links or by adding perl code to the config
> file but it would be nice to have a better way to do it).
>
> This generalization of host groups could easily include the notion of
> maximum simultaneous group backups to run.
>
It would also be nice to use the group notion to allow the ability to
specify a different topdir for different groups. This could be useful
in cases where there is not much overlap (i.e. pooling potential)
between groups and where there might be reasons to split the pool
between drives. This would presumably be better than the kludge recently
discussed on this list of running multiple instances of backuppc on
one server.
It would further be beneficial to carry the group distinction to the
web interface so that you could view results by group which could be
helpful if you have *many* machines or if you want to subtotal various
stats by group.
--
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] "Clusters" of hosts
John Rouillard wrote: > >> For example, this would allow one to define a config file for Linux >> vs. Windows machines or for desktops vs. notebooks or for critical >> machines vs. less critical machines (I know you can currently do this >> in a kludgey fashion using links or by adding perl code to the config >> file but it would be nice to have a better way to do it). > > At one point I looked at including a series of perl files in an > existing per host config file to build up the default settings. > I need to go back and look at that again. > >> This generalization of host groups could easily include the notion of >> maximum simultaneous group backups to run. > > Only if the two host and simultaneous backup groups overlapped > 100%. > > E.G. I have two redundant database servers db10 and db11. Because > of the impact of doing backups on the servers, I never want both > of them at a site to be backed up at the same time. So these > would share the same configurations (or part of a configuration) > and also be a group that would be limited to 1 backup from the > group at a time. > > Now add a second (redundant) site/cluster with database > servers db21 and db22. Now all 4 servers can share a config, but > I have two different sub-groups of servers (db11, db12) and (db21 > and db22) that have different rules about the max number of > backups to be done. Maybe a 'groups' concept could be added such that you could put a host in multiple groups with an order specified and the perl-snippet configurations are just evaluated in cascading order (site level, group1, group2..., host). That should be easy to do and would make sense if you generally don't overlap the group variables but might get confusing if you get carried away. You'd still have to tie the rate-limiting value to the group definition where it was specified, though, so a host in multiple groups wouldn't start a backup if it would exceed any of the limits that had been picked up. -- Les Mikesell lesmikes...@gmail.com -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] "Clusters" of hosts
On Wed, Jun 10, 2009 at 11:38:30AM -0400, Jeffrey J. Kosowsky wrote: > Les Mikesell wrote at about 10:23:10 -0500 on Wednesday, June 10, 2009: > > John Rouillard wrote: > > > On Sat, May 30, 2009 at 10:16:33PM +0200, Pieter Wuille wrote: > > >> I don't know how common this usage is, but > > >> in our setup we have a lot of backuppc > > >> "hosts" that are physically located on a > > >> few machines only. It would be nice if it > > >> were possible to allow hosts on different > > >> machines to be backupped simultaneously, > > >> but prevent simultaneous backups(dumps) of > > >> hosts on the same machine. > > > If you have a way of mapping the host names to a physical machine, you > > > can use my queing/locking strategy described in: > > > > > > > > > > http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg13698.html > > > > > As a feature request, I think it would be nice to have a way to add > > hosts to groups, then limit how many in each group the scheduler would > > start at once. There are several scenarios where this is needed to > > avoid overloading some common reasource - like a low-bandwidth link as > > well as sharing a physical host or filesystem. > > I think the notion of host groups is a good idea. Even more generally, > it would be nice to be able to define config files at the group level > rather than the current choice between the default config.pl file and > host-specific config files. I agree with both group definitions, but the hosts should be able to participate in multiple groups. The groups used to define what gets backed up and the group that defines how the schedule of backups occurs should be able to be different. E.G. if you have two data centers that are being backed up, you may have the same types of machines in both sites, but one of the data centers is remote from the backup-pc server and you only want 4 machines at the remote site to be simultaneously backed up to restrict bandwidth use etc. > For example, this would allow one to define a config file for Linux > vs. Windows machines or for desktops vs. notebooks or for critical > machines vs. less critical machines (I know you can currently do this > in a kludgey fashion using links or by adding perl code to the config > file but it would be nice to have a better way to do it). At one point I looked at including a series of perl files in an existing per host config file to build up the default settings. I need to go back and look at that again. > This generalization of host groups could easily include the notion of > maximum simultaneous group backups to run. Only if the two host and simultaneous backup groups overlapped 100%. E.G. I have two redundant database servers db10 and db11. Because of the impact of doing backups on the servers, I never want both of them at a site to be backed up at the same time. So these would share the same configurations (or part of a configuration) and also be a group that would be limited to 1 backup from the group at a time. Now add a second (redundant) site/cluster with database servers db21 and db22. Now all 4 servers can share a config, but I have two different sub-groups of servers (db11, db12) and (db21 and db22) that have different rules about the max number of backups to be done. -- -- rouilj John Rouillard System Administrator Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111 -- Crystal Reports - New Free Runtime and 30 Day Trial Check out the new simplified licensing option that enables unlimited royalty-free distribution of the report engine for externally facing server and web deployment. http://p.sf.net/sfu/businessobjects ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] "Clusters" of hosts
Les Mikesell wrote at about 10:23:10 -0500 on Wednesday, June 10, 2009:
> John Rouillard wrote:
> > On Sat, May 30, 2009 at 10:16:33PM +0200, Pieter Wuille wrote:
> >> I don't know how common this usage is, but in our setup we have a lot of
> >> backuppc "hosts" that are physically located on a few machines only. It
> >> would be nice if it were possible to allow hosts on different machines to
> >> be backupped simultaneously, but prevent simultaneous backups(dumps) of
> >> hosts on the same machine.
> >>
> >> Any thoughts?
> >
> > If you have a way of mapping the host names to a physical machine, you
> > can use my queing/locking strategy described in:
> >
> >
> >
> > http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg13698.html
> >
> > Create one queue/semaphore per physical machine and have the
> > $Conf{DumpPreUserCmd} command exit with an error if it can't get a
> > slot/lock (also you will have to set $Conf{UserCmdCheckStatus} = 1;).
>
> As a feature request, I think it would be nice to have a way to add
> hosts to groups, then limit how many in each group the scheduler would
> start at once. There are several scenarios where this is needed to
> avoid overloading some common reasource - like a low-bandwidth link as
> well as sharing a physical host or filesystem.
>
I think the notion of host groups is a good idea. Even more generally,
it would be nice to be able to define config files at the group level
rather than the current choice between the default config.pl file and
host-specific config files.
For example, this would allow one to define a config file for Linux
vs. Windows machines or for desktops vs. notebooks or for critical
machines vs. less critical machines (I know you can currently do this
in a kludgey fashion using links or by adding perl code to the config
file but it would be nice to have a better way to do it).
This generalization of host groups could easily include the notion of
maximum simultaneous group backups to run.
--
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] "Clusters" of hosts
John Rouillard wrote:
> On Sat, May 30, 2009 at 10:16:33PM +0200, Pieter Wuille wrote:
>> I don't know how common this usage is, but in our setup we have a lot of
>> backuppc "hosts" that are physically located on a few machines only. It
>> would be nice if it were possible to allow hosts on different machines to
>> be backupped simultaneously, but prevent simultaneous backups(dumps) of
>> hosts on the same machine.
>>
>> Any thoughts?
>
> If you have a way of mapping the host names to a physical machine, you
> can use my queing/locking strategy described in:
>
>
>
> http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg13698.html
>
> Create one queue/semaphore per physical machine and have the
> $Conf{DumpPreUserCmd} command exit with an error if it can't get a
> slot/lock (also you will have to set $Conf{UserCmdCheckStatus} = 1;).
As a feature request, I think it would be nice to have a way to add
hosts to groups, then limit how many in each group the scheduler would
start at once. There are several scenarios where this is needed to
avoid overloading some common reasource - like a low-bandwidth link as
well as sharing a physical host or filesystem.
--
Les Mikesell
lesmikes...@gmail.com
--
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] "Clusters" of hosts
On Sat, May 30, 2009 at 10:16:33PM +0200, Pieter Wuille wrote:
> I don't know how common this usage is, but in our setup we have a lot of
> backuppc "hosts" that are physically located on a few machines only. It
> would be nice if it were possible to allow hosts on different machines to
> be backupped simultaneously, but prevent simultaneous backups(dumps) of
> hosts on the same machine.
>
> Any thoughts?
If you have a way of mapping the host names to a physical machine, you
can use my queing/locking strategy described in:
http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg13698.html
Create one queue/semaphore per physical machine and have the
$Conf{DumpPreUserCmd} command exit with an error if it can't get a
slot/lock (also you will have to set $Conf{UserCmdCheckStatus} = 1;).
--
-- rouilj
John Rouillard System Administrator
Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111
--
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
