Re: Selectively duplicating client data across servers
I agree. Multiple storage pools really aren't difficult to manage, and it will make your process very simple, and incremental. With backupsets, you have to copy ALL the data onto a new backupset, which will be time-consuming. Another disadvantage of using backupsets, think about what happens if A B go bang - you are left sitting at C, trying to figure out how to use your backupsets. You will have to process entire backupsets to get anything back, and you have no inventory DB that shows you what is on them at the file level. If you send the data to C: as part of the BACKUP STGPOOL process, you can always open the client GUI to see what should be there, and do restores selectively. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Steven Pemberton Sent: Wednesday, August 25, 2004 10:40 PM To: [EMAIL PROTECTED] Subject: Re: Selectively duplicating client data across servers On Thursday 26 August 2004 09:19, Stuart Lamble wrote: Hey ho. Here's the skinny. We will, eventually, have a number of clients backing up to a TSM server on a regular basis (we're still setting up the SAN and other ancillary things that are needed to support the TSM server). Some of them will be filesystem backups; others will be database backups (which, if I understand correctly, are most likely to be seen as archives rather than backups as such). The setup involves two sites, A and B, which have multiple gigabit fibre connections between them (so they're effectively on the same LAN; the only real difference is a very small amount of additional latency.) Systems at site A will backup to a server at site B, and vice versa. So, you have the following? 1/ Clients at A backup to TSM server at B. 2/ Clients at B backup to TSM server at A. Where are you producing the copypool versions? Eg: 1/ Client A - TSM B (primary) - TSM A (copy) ? 2/ Client A - TSM B (primary) - TSM B (copy) ? 3/ What copy pools? :( The project I'm working on involves a third site; call it site C. The connectivity from sites A and B to site C is significantly poorer than that between A and B, largely because site C is significantly remote to A and B (whilst A and B are within a few km of each other), precluding running of fibre to it. (Not sure what the current connections are; that's not my area.) The idea is that if things go boom in a major way, and we lose everything at _both_ site A and site B, we want to have copies of the university's critical data (student records, SAP, that sort of thing) available for restore. Maybe the data will be a little old, but better than nothing. Something like this? 1/ Client A - TSM B (primary) - TSM A (copy) (all) - TSM C (copy/export) (critical only) A healthy paranoia. :) So the idea is this. The servers holding the critical data backup to their backup server as normal. Once every so often (once a week, once a fortnight, once a month...), we want to take the data off the _backup server_, and copy it to another TSM server at site C. We want to do this only for those critical servers, not for all servers. The data may be in the form of either archives or filesystem backups, or some combination of the two. We _don't_ want to be running extra backups for the servers in question to provide this redundancy. The two solutions I've come up with involve data export, and copy pools (via virtual volumes). The problem is, both of those operate at the storage pool level; there's no way to specify copy/export only the data for _this_ client, and no others that I can see. Actually, you can export node for individual hosts, but I'm not sure if it's the best way to do what you're planning. However, export node can specify a client, time range, backup and/or archive data, active files only, and export/import directly via a server to server connection. It's preferable that we not have to create separate storage pools (all the way down to the tape level) for these systems just so we can do this -- we'd prefer to have one disk pool and one tape pool for the whole shebang if possible. I'd normally recommed that you DO create multiple storage pools, so that you can better control the physical location of the backup data. This can improve recovery performance by separating critical clients to their own storage pools and tapes. With only one huge disk/tape storage pool hierarchy each client's data will tend to fragment across a large number of tapes (unless you use collocation, which may greatly reduce tape efficiency instead). If you do create seperate storage pools, then it's simply a matter of running an additional backup stgpool command to produce the extra off-site copy for site C. This has another advantage in that it's a completely incremental process, and you can probably afford to run it every day (for the critical nodes/storage pools only). Backup sets may be doable, but I'm a little uncertain
Re: Selectively duplicating client data across servers
Hi Everyone, I have deleted the thread of this email since I am sure everyone has read a couple of times already. I figure I would throw my $0.02 into the mix as well. I have done a few configs like this over the years so I have some experience in this area. I think you have been getting some good feedback here and when it all said and done you should be able to boil this down to something very workable. So here is what I would propose. Client A non-crit - TSM B non-crit Pri-pool - TSM B non-crit Copypool - Ship tapes to TSM C (Optional) |- TSM B non-crit onsite Copypool Client A crit - TSM B crit Pri-pool - TSM B crit Copypool (Virtual Volumes on TSM C) (Optional) |- TSM B crit onsite Copypool Mirror the config for client B I am assuming that C will be your DR site and therefore that is where your Copy pool data should wind up. It is not a good idea to keep both the primary and the copy (if you are having only one copypool copy) on the same site. Keep in mind that if you loose site A, site B could still be missing some critical data, i.e. Archive data that was deleted from the owner's system and inactive versions would all be missing! So it is important that at least one copypool's data be separated from the primary pool. I am recommending that you use Virtual Volumes for the most critical machine's copypool. This can be done by setting up a separate storage pool hierarchy for these critical machines of course this is predicated on having the necessary bandwidth to transfer all the critical data to site C. Keep in mind to make this functional you should also be backing up TSM A's and TSM B's DB through Virtual Volumes to TSM C as well. Now one thing to remember, backup environments are like buying insurance, so the more protection you want the more expensive it will get. However, you must make sure that you spend enough to protect your self! Having said that there is a few other things you can consider. You might consider keeping an onsite and an offsite copypool. This would allow for much faster recovery of data in the event of a primary tape failure. You will have a time delay exposure if the tape that fails is non-crit primary you will have to retrieve the copypool tape from offsite. If it is a critical primary tape then you will have to move the data back through the pipe and this could have some impact based on what else is trying to go through the pipe at that time. You might consider having clones of TSM A and TSM B at site C. This would allow for much quicker DR recovery times if that is a concern. And to take this one step further you could even to DB restores to those machines on a regular basis if needed. By the way those clone instances could run on the same system as TSM C. That would save licensing costs as well as have some performance advantages during DR restores. Make sure that you plan on testing this environment. Backing up is fine but restores is what it is all about! That means you must build into the plan up front how you are going to do your testing without impacting production and still feel confident that you can restore to production if necessary. This is the most common failure point for projects like this. They get designed and implemented with no fore thought on how to test it and when it comes time to do the DR test they got themselves in a jam. There is a lot more to consider for the testing phase them what information we have been given. As I said before I have done a few setups that were similar to this so if you have any further questions please feel free to ask. Good Luck. -- Regards, Mark D. Rodriguez President MDR Consulting, Inc. === MDR Consulting The very best in Technical Training and Consulting. IBM Advanced Business Partner SAIR Linux and GNU Authorized Center for Education IBM Certified Advanced Technical Expert, CATE AIX Support and Performance Tuning, RS6000 SP, TSM/ADSM and Linux Red Hat Certified Engineer, RHCE ===
Selectively duplicating client data across servers
Hey ho. Here's the skinny. We will, eventually, have a number of clients backing up to a TSM server on a regular basis (we're still setting up the SAN and other ancillary things that are needed to support the TSM server). Some of them will be filesystem backups; others will be database backups (which, if I understand correctly, are most likely to be seen as archives rather than backups as such). The setup involves two sites, A and B, which have multiple gigabit fibre connections between them (so they're effectively on the same LAN; the only real difference is a very small amount of additional latency.) Systems at site A will backup to a server at site B, and vice versa. The project I'm working on involves a third site; call it site C. The connectivity from sites A and B to site C is significantly poorer than that between A and B, largely because site C is significantly remote to A and B (whilst A and B are within a few km of each other), precluding running of fibre to it. (Not sure what the current connections are; that's not my area.) The idea is that if things go boom in a major way, and we lose everything at _both_ site A and site B, we want to have copies of the university's critical data (student records, SAP, that sort of thing) available for restore. Maybe the data will be a little old, but better than nothing. So the idea is this. The servers holding the critical data backup to their backup server as normal. Once every so often (once a week, once a fortnight, once a month...), we want to take the data off the _backup server_, and copy it to another TSM server at site C. We want to do this only for those critical servers, not for all servers. The data may be in the form of either archives or filesystem backups, or some combination of the two. We _don't_ want to be running extra backups for the servers in question to provide this redundancy. The two solutions I've come up with involve data export, and copy pools (via virtual volumes). The problem is, both of those operate at the storage pool level; there's no way to specify copy/export only the data for _this_ client, and no others that I can see. It's preferable that we not have to create separate storage pools (all the way down to the tape level) for these systems just so we can do this -- we'd prefer to have one disk pool and one tape pool for the whole shebang if possible. Backup sets may be doable, but I'm a little uncertain about how they'd go with virtual volumes, and also whether they'd cover archive data sets. So the question is: is there any way we can say to TSM Copy this client's data (backup and archive) to that server over there, ignoring all other client data in that storage pool? Or am I smoking crack (again)? Any pointers or suggestions on where to look would be very gratefully received. For me, this isn't so much a learning curve as a learning cliff. :) If it's relevant, the clients in question are mostly Solaris, and we'll be running Tivoli Storage Manager 5.2 on Solaris servers. Many, many thanks for any tips. I'm coming from a background with Legato Networker, and I'm still trying to get my head around some of the more interesting aspects of TSM, learning as I go. :) Cheers, Stuart.
Re: Selectively duplicating client data across servers
Hi Stuart. Sorry. If backupset to virtual volumes won't fit, then you'll have to split your backups into two storagepool hierarchies. but, since this is a belt-and-braces-both-fail scenario, and is thus unlikely to be used, is it possible to make the backup by shipping tapes? Create a third copy using copy stg and ship it, with a database snapshot offsite. Either restore the DB to a recovery TSM server as a matter of course, or only do it when necessary. When its time for a refresh, either delete every volume in the stgpool and recreate it, or just run another backup stg and send the new tapes and DB snapshot offsite. DRM will handle the tape movements. Regards Steve Harris AIX and TSM Admin Queensland Health, Brisbane Australia [EMAIL PROTECTED] 26/08/2004 9:19:16 Hey ho. Here's the skinny. We will, eventually, have a number of clients backing up to a TSM server on a regular basis (we're still setting up the SAN and other ancillary things that are needed to support the TSM server). Some of them will be filesystem backups; others will be database backups (which, if I understand correctly, are most likely to be seen as archives rather than backups as such). The setup involves two sites, A and B, which have multiple gigabit fibre connections between them (so they're effectively on the same LAN; the only real difference is a very small amount of additional latency.) Systems at site A will backup to a server at site B, and vice versa. The project I'm working on involves a third site; call it site C. The connectivity from sites A and B to site C is significantly poorer than that between A and B, largely because site C is significantly remote to A and B (whilst A and B are within a few km of each other), precluding running of fibre to it. (Not sure what the current connections are; that's not my area.) The idea is that if things go boom in a major way, and we lose everything at _both_ site A and site B, we want to have copies of the university's critical data (student records, SAP, that sort of thing) available for restore. Maybe the data will be a little old, but better than nothing. So the idea is this. The servers holding the critical data backup to their backup server as normal. Once every so often (once a week, once a fortnight, once a month...), we want to take the data off the _backup server_, and copy it to another TSM server at site C. We want to do this only for those critical servers, not for all servers. The data may be in the form of either archives or filesystem backups, or some combination of the two. We _don't_ want to be running extra backups for the servers in question to provide this redundancy. The two solutions I've come up with involve data export, and copy pools (via virtual volumes). The problem is, both of those operate at the storage pool level; there's no way to specify copy/export only the data for _this_ client, and no others that I can see. It's preferable that we not have to create separate storage pools (all the way down to the tape level) for these systems just so we can do this -- we'd prefer to have one disk pool and one tape pool for the whole shebang if possible. Backup sets may be doable, but I'm a little uncertain about how they'd go with virtual volumes, and also whether they'd cover archive data sets. So the question is: is there any way we can say to TSM Copy this client's data (backup and archive) to that server over there, ignoring all other client data in that storage pool? Or am I smoking crack (again)? Any pointers or suggestions on where to look would be very gratefully received. For me, this isn't so much a learning curve as a learning cliff. :) If it's relevant, the clients in question are mostly Solaris, and we'll be running Tivoli Storage Manager 5.2 on Solaris servers. Many, many thanks for any tips. I'm coming from a background with Legato Networker, and I'm still trying to get my head around some of the more interesting aspects of TSM, learning as I go. :) Cheers, Stuart. *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s), or if it is transmitted/received in error. Any unauthorised use, alteration, disclosure, distribution or review of this email is prohibited. It may be subject to a statutory duty of confidentiality if it relates to health service matters. If you are not the intended recipient(s), or if you have received this email in error, you are asked to immediately notify the sender by telephone or by return email. You should also delete this email and destroy any hard copies produced. ***
Re: Selectively duplicating client data across servers
On Thursday 26 August 2004 09:19, Stuart Lamble wrote: Hey ho. Here's the skinny. We will, eventually, have a number of clients backing up to a TSM server on a regular basis (we're still setting up the SAN and other ancillary things that are needed to support the TSM server). Some of them will be filesystem backups; others will be database backups (which, if I understand correctly, are most likely to be seen as archives rather than backups as such). The setup involves two sites, A and B, which have multiple gigabit fibre connections between them (so they're effectively on the same LAN; the only real difference is a very small amount of additional latency.) Systems at site A will backup to a server at site B, and vice versa. So, you have the following? 1/ Clients at A backup to TSM server at B. 2/ Clients at B backup to TSM server at A. Where are you producing the copypool versions? Eg: 1/ Client A - TSM B (primary) - TSM A (copy) ? 2/ Client A - TSM B (primary) - TSM B (copy) ? 3/ What copy pools? :( The project I'm working on involves a third site; call it site C. The connectivity from sites A and B to site C is significantly poorer than that between A and B, largely because site C is significantly remote to A and B (whilst A and B are within a few km of each other), precluding running of fibre to it. (Not sure what the current connections are; that's not my area.) The idea is that if things go boom in a major way, and we lose everything at _both_ site A and site B, we want to have copies of the university's critical data (student records, SAP, that sort of thing) available for restore. Maybe the data will be a little old, but better than nothing. Something like this? 1/ Client A - TSM B (primary) - TSM A (copy) (all) - TSM C (copy/export) (critical only) A healthy paranoia. :) So the idea is this. The servers holding the critical data backup to their backup server as normal. Once every so often (once a week, once a fortnight, once a month...), we want to take the data off the _backup server_, and copy it to another TSM server at site C. We want to do this only for those critical servers, not for all servers. The data may be in the form of either archives or filesystem backups, or some combination of the two. We _don't_ want to be running extra backups for the servers in question to provide this redundancy. The two solutions I've come up with involve data export, and copy pools (via virtual volumes). The problem is, both of those operate at the storage pool level; there's no way to specify copy/export only the data for _this_ client, and no others that I can see. Actually, you can export node for individual hosts, but I'm not sure if it's the best way to do what you're planning. However, export node can specify a client, time range, backup and/or archive data, active files only, and export/import directly via a server to server connection. It's preferable that we not have to create separate storage pools (all the way down to the tape level) for these systems just so we can do this -- we'd prefer to have one disk pool and one tape pool for the whole shebang if possible. I'd normally recommed that you DO create multiple storage pools, so that you can better control the physical location of the backup data. This can improve recovery performance by separating critical clients to their own storage pools and tapes. With only one huge disk/tape storage pool hierarchy each client's data will tend to fragment across a large number of tapes (unless you use collocation, which may greatly reduce tape efficiency instead). If you do create seperate storage pools, then it's simply a matter of running an additional backup stgpool command to produce the extra off-site copy for site C. This has another advantage in that it's a completely incremental process, and you can probably afford to run it every day (for the critical nodes/storage pools only). Backup sets may be doable, but I'm a little uncertain about how they'd go with virtual volumes, and also whether they'd cover archive data sets. Backup sets only encompass filesystem backup data. They cannot be used for filesystem archives, nor for application TDP backups (even if the TDP backups are really backup objects). So the question is: is there any way we can say to TSM Copy this client's data (backup and archive) to that server over there, ignoring all other client data in that storage pool? Or am I smoking crack (again)? Probably the easiest way is to use a separate storage pool(s) for the critical clients. As I mentioned above, this may also help in controlling tape fragmentation and improve recovery performance. Give me a call if you want to discuss it further. Regards, Steven P. -- Steven Pemberton Senior Enterprise Management Consultant IBK, Senetas Group Mobile: +61/0 418 335 136 | Phone: +61/3 9820 5811 | Fax: +61/3 9820 9907 Level 1, 11 Queens Road,
Re: Selectively duplicating client data across servers
(Once more, this time with the _right_ From address. Sigh.) On 26/08/2004, at 12:40 PM, Steven Pemberton wrote: On Thursday 26 August 2004 09:19, Stuart Lamble wrote: Hey ho. Here's the skinny. We will, eventually, have a number of clients backing up to a TSM server on a regular basis (we're still setting up the SAN and other ancillary things that are needed to support the TSM server). Some of them will be filesystem backups; others will be database backups (which, if I understand correctly, are most likely to be seen as archives rather than backups as such). The setup involves two sites, A and B, which have multiple gigabit fibre connections between them (so they're effectively on the same LAN; the only real difference is a very small amount of additional latency.) Systems at site A will backup to a server at site B, and vice versa. So, you have the following? 1/ Clients at A backup to TSM server at B. 2/ Clients at B backup to TSM server at A. Yup, pretty much. Where are you producing the copypool versions? Eg: 1/ Client A - TSM B (primary) - TSM A (copy) ? 2/ Client A - TSM B (primary) - TSM B (copy) ? 3/ What copy pools? :( Oh, option three of course -- we want to save money on media! *ducks* Seriously, though, it'll be option 2, is my understanding -- if site A goes down, and some of the media at site B is dead, we'd like to still be able to recover the data. If we lost two sets of media at the same time, well, we're obviously not meant to have that data any more. (Cue the story I heard whilst doing Legato Networker training: a site with several copies of key data. Key data goes poof. First copy on tape is bad. Second copy on tape is bad. They send for the offsite copy. Courier manages to have an accident, and the tapes are ruined...) (The scary thing is, there were a couple of people advocating no copy pools for some of the clients. Thank God _that_ got shot down in short order.) [verbose description of the basic plan snipped in favour of Steven's summary] Something like this? 1/ Client A - TSM B (primary) - TSM A (copy) (all) - TSM C (copy/export) (critical only) A healthy paranoia. :) That's pretty much it. A more accurate picture would be: Client A - TSM B (primary/copy) (all) - TSM C (copy/export) (critical) Client B - TSM A (primary/copy) (all) - TSM C (copy/export) (critical) And remember: just because I'm paranoid, it doesn't mean they're _not_ out to get me... ;) [copying data from the original backup server to the remote site] The two solutions I've come up with involve data export, and copy pools (via virtual volumes). The problem is, both of those operate at the storage pool level; there's no way to specify copy/export only the data for _this_ client, and no others that I can see. Actually, you can export node for individual hosts, but I'm not sure if it's the best way to do what you're planning. However, export node can specify a client, time range, backup and/or archive data, active files only, and export/import directly via a server to server connection. Hm. Going to have to re-read the manual on that; I must have missed that point. *flick flick flick* ... ok, I missed that point. Excuse me whilst I carefully extract my foot from my mouth. :) It's preferable that we not have to create separate storage pools (all the way down to the tape level) for these systems just so we can do this -- we'd prefer to have one disk pool and one tape pool for the whole shebang if possible. I'd normally recommed that you DO create multiple storage pools, so that you can better control the physical location of the backup data. This can improve recovery performance by separating critical clients to their own storage pools and tapes. With only one huge disk/tape storage pool hierarchy each client's data will tend to fragment across a large number of tapes (unless you use collocation, which may greatly reduce tape efficiency instead). Interesting point. Everybody here is an utter newbie when it comes to TSM; we've done the initial training course (you should remember; IIRC, you were the one taking the course I was on :) which is all fine and dandy, but it doesn't really expose you to the little tricks of the trade which come up when you're actually _using_ the product. :) (And besides -- after too many months of not using the product because of wrangling that's out of the hands of the techies, you tend to forget the finer points that were covered on the course.) Still, I have a fair amount of faith that TSM will do the job we need; it's more or less a matter of what problems we run into along the way (and don't tell me we won't run into problems -- we will; it's just a question of how severe they are and how difficult to fix. With luck, they'll be less than what we have with our current backup system.) We've already ruled out collocation for the most part; I seem to recall an upcoming version of TSM has a weaker form of collocation (along the lines of group
