Re: Thoughts on Monthly Archives
Thanks, Andy! As always, you open up the window of thought and different perspectives. Shannon Bach Operations Analyst IMS Data Center Services Madison Gas Electric Co. Andrew Raibeck [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 07/19/2004 12:45 PM Please respond to ADSM: Dist Stor Manager To:[EMAIL PROTECTED] cc: Subject:Re: Thoughts on Monthly Archives For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technology comes in! My personal (not necessarily that of IBM's) opinion: this is a flippant response to a valid concern, unless your responsibilities cover this area as well. >From a TSM administrative perspective, it is the TSM administrator's responsibility to ensure that data backed up by TSM can be restored to the same state it was in at the time it was backed up, plus other duties related to the backup and management of the data, as assigned. Being able to convert from one external data format to another is not a function of TSM, and thus is not naturally a part of administering TSM. In general I would say that resolving the issues related to long-term archive of data belong to the owners of the data and the people who administer that data. After all, they are the experts on that data and are therefore the best resources for addressing these issues. Of course, in the process of planning for the archives, the TSM administrator can raise these issues (as you have apparently done) and contribute to the solution; but I wouldn't put the sole responsibility on the TSM administrator. Nor was it my intent to suggest that these were TSM issues per se when I raised them. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence. ADSM: Dist Stor Manager [EMAIL PROTECTED] wrote on 07/19/2004 10:09:16: For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technoloogy comes in! They don't care if we have the software capable of reading this data again. They just want to be in compliance with Sarbanes-Oxley. And it is starting to look to me that Sarbanes-Oxley believes in keeping everything, forever. Andrew Raibeck [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 07/19/2004 11:25 AM Please respond to ADSM: Dist Stor Manager To:[EMAIL PROTECTED] cc: Subject: Re: Thoughts on Monthly Archives Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
== In article [EMAIL PROTECTED], Steve Harris [EMAIL PROTECTED] writes: There was another management requirement that all production servers be backed up in full once per year and that snapshot be kept forever - there is no reasoning with this, its one of those stupid mandates that applies to the whole of the state government, and if data is not able to be categorized, then it must be kept. Mmm, Arbitrary. Tastes like BUDGET! I think you'll recognise that having a third TSM Server for the yearly backup isn't really an option, so an archive mechanism is the only one that will work for that. Actually, I'd disagree that an extra server is a barrier. The reason is that I am finding it to be very simple to maintain several servers on the same hardware. Right now I've got ~10 on one box, and it works quite well. Though, really, you can use a seprate policy domain instead of a separate server perfectly easily. But I like the separate server scheme... So, permit me to spin a yarn for you, which I will assert will solve your monthly and your yearly-forever problems all in one swell foop, and at a lower cost in management and consumables than your archive scheme. Erect on your TSM server a second TSM instance. I'll call it the ARCHIVE server. On the ARCHIVE server, define nodes for those machines getting the cast-in-stone treatment; On their management classes, define: verexists 1200 verdeleted 1200 retextra nolimit retonlynolimit On each of the client machines, add an ARCHIVE stanza to your dsm.sys. Then, run incrementals for each box once a month. In unix land, I'd say 1 0 1 * * dsmc incr -se=ARCHIVE in your crontab. Now, I recognize that this is only retaining data for 100 years, but what the heck, the pig may learn to sing. ;) You get all the benefits of the incremental scheme, you separate the archive/retention database from the recovery database, and you don't waste any hardware. There, I assert. Please pick holes. - Allen S. Rout
Re: Thoughts on Monthly Archives
Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technoloogy comes in! They don't care if we have the software capable of reading this data again. They just want to be in compliance with Sarbanes-Oxley. And it is starting to look to me that Sarbanes-Oxley believes in keeping everything, forever. Andrew Raibeck [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 07/19/2004 11:25 AM Please respond to ADSM: Dist Stor Manager To:[EMAIL PROTECTED] cc: Subject:Re: Thoughts on Monthly Archives Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
I havn't read the SBO requirements, but from our internal auditors, it looks like we need to have a good business best efforts to keep readable for whatever retention period we publicize. In working with older tape technology in storing archive tapes, we found that 20% of the tapes were not readable within 5 years. It seems that the best idea for long term archives is they need to be re-read on a regular basis (annually?) just to make sure they can be read. The only time I have heard of a real application that must have 'everything forever' was a TSM application where they were storing a local and remote copy of documentation (and each version) for a nuclear power plant. In that case, I think they used a WORM library both locally and remotely. I didn't design it and don't know the details, just that it 'had to work'. It got expensive, but that is what the federal regulators requried. -Original Message- From: Shannon Bach [mailto:[EMAIL PROTECTED] Sent: Monday, July 19, 2004 12:09 PM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technoloogy comes in! They don't care if we have the software capable of reading this data again. They just want to be in compliance with Sarbanes-Oxley. And it is starting to look to me that Sarbanes-Oxley believes in keeping everything, forever. Andrew Raibeck [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 07/19/2004 11:25 AM Please respond to ADSM: Dist Stor Manager To:[EMAIL PROTECTED] cc: Subject:Re: Thoughts on Monthly Archives Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technology comes in! My personal (not necessarily that of IBM's) opinion: this is a flippant response to a valid concern, unless your responsibilities cover this area as well. From a TSM administrative perspective, it is the TSM administrator's responsibility to ensure that data backed up by TSM can be restored to the same state it was in at the time it was backed up, plus other duties related to the backup and management of the data, as assigned. Being able to convert from one external data format to another is not a function of TSM, and thus is not naturally a part of administering TSM. In general I would say that resolving the issues related to long-term archive of data belong to the owners of the data and the people who administer that data. After all, they are the experts on that data and are therefore the best resources for addressing these issues. Of course, in the process of planning for the archives, the TSM administrator can raise these issues (as you have apparently done) and contribute to the solution; but I wouldn't put the sole responsibility on the TSM administrator. Nor was it my intent to suggest that these were TSM issues per se when I raised them. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence. ADSM: Dist Stor Manager [EMAIL PROTECTED] wrote on 07/19/2004 10:09:16: For us, it is the beginning of the Sarbanes-Oxley overhaul. I ask those same questions to people all over my company and their response? Well you (me) had better make sure that the data moves with whatever new Technoloogy comes in! They don't care if we have the software capable of reading this data again. They just want to be in compliance with Sarbanes-Oxley. And it is starting to look to me that Sarbanes-Oxley believes in keeping everything, forever. Andrew Raibeck [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager [EMAIL PROTECTED] 07/19/2004 11:25 AM Please respond to ADSM: Dist Stor Manager To:[EMAIL PROTECTED] cc: Subject:Re: Thoughts on Monthly Archives Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
I haven't read the SBO requirements, but from our internal auditors, it looks like we need to have a good business best efforts to keep readable for whatever retention period we publicize. In working with older tape technology in storing archive tapes, we found that 20% of the tapes were not readable within 5 years. It seems that the best idea for long term archives is they need to be re-read on a regular basis (annually?) just to make sure they can be read. Or refreshed. Actually when I raised the issue of the data being readable, I wasn't referring to hardware or media problems (though these are also good points), but the availability of software that can understand the data. For example, if you open a .zip file with a plain text editor, the content will appear as so much garbage. Without a zip program that can make sense of the data, being able to restore a .zip file doesn't give you anything useful (at least not without an awful lot of hacking). .zip is a common format and likely to be available in the future, but what about more obscure formats, where maybe you archive the files while you use the product? Suppose you switch to a new product that uses a different format, then decide 10 years from now you need to retrieve the data that was used by the old product. Will you still have a copy of that old product around that will run on current operating systems and hardware? Even if you archived the software and could in theory restore and run it, do you know where, amongst all that archive data, the software is? Will your successors be able to find the data and the software? Will they even know which software they need to run? Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
In our case, this is going to be used to back up and archive file servers - NOT operating systems. As for how to read/use the data, here's the directory structure: \\toaster\is\documentation - contains docs, for instance \\toaster\installs - contains setup files for all apps used in our environment I don't know about you, but we're still swimming in 95 and NT CD's, and some of our hardware would have no problems stepping down to run those operating systems. -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Raibeck Sent: Monday, July 19, 2004 11:58 AM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives I haven't read the SBO requirements, but from our internal auditors, it looks like we need to have a good business best efforts to keep readable for whatever retention period we publicize. In working with older tape technology in storing archive tapes, we found that 20% of the tapes were not readable within 5 years. It seems that the best idea for long term archives is they need to be re-read on a regular basis (annually?) just to make sure they can be read. Or refreshed. Actually when I raised the issue of the data being readable, I wasn't referring to hardware or media problems (though these are also good points), but the availability of software that can understand the data. For example, if you open a .zip file with a plain text editor, the content will appear as so much garbage. Without a zip program that can make sense of the data, being able to restore a .zip file doesn't give you anything useful (at least not without an awful lot of hacking). .zip is a common format and likely to be available in the future, but what about more obscure formats, where maybe you archive the files while you use the product? Suppose you switch to a new product that uses a different format, then decide 10 years from now you need to retrieve the data that was used by the old product. Will you still have a copy of that old product around that will run on current operating systems and hardware? Even if you archived the software and could in theory restore and run it, do you know where, amongst all that archive data, the software is? Will your successors be able to find the data and the software? Will they even know which software they need to run? Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: Thoughts on Monthly Archives
As someone who has migrated from Burroughs Medium Systems (B3900) to Honeywell (DPS-8) to IBM (4381, 3090, 9672 - MVS, MVS-XA, MVS-ESA) to IBM RS/6000, I have to agree with Andrew. Any long-term archiving mandated by the business MUST include archiving of the data in an UNLOADED format - either csv or fixed field, ascii format (no binary or packed fields). In addition, the human-readable database schema and subschemas and the source code to the unload and reload programs need to be archived. The cost for this is not trivial. The unload/reload programs should be tested out. Just about any non-trivial database will take long enough to flat-file that you'll need to run the process from a copy of the original. And once it's been unloaded, you need to manage the results. I would be inclined to agree with Dwight -- this isn't really a TSM process. I'd think about using tar or pax (primarily because the source to both is available) to write the archive media (tape, dvd, whatever) and figure on copying the result about every six months to a year. Short of that -- and I have yet to get MY corporation to buy into this -- I have seven archive management classes: one_year, two_year, and on to seven_year, with retentions amounting to one month longer than the class name suggests. And an internal memo I dust off from time to time about why this is a dumb idea. Current archiving activities haven't really caused database issues in TSM - yet. But the requirement for more disk space is one of the 'dumb idea' entries. Good Luck - Tom Kauffman NIBCO, Inc -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Raibeck Sent: Monday, July 19, 2004 12:58 PM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives I haven't read the SBO requirements, but from our internal auditors, it looks like we need to have a good business best efforts to keep readable for whatever retention period we publicize. In working with older tape technology in storing archive tapes, we found that 20% of the tapes were not readable within 5 years. It seems that the best idea for long term archives is they need to be re-read on a regular basis (annually?) just to make sure they can be read. Or refreshed. Actually when I raised the issue of the data being readable, I wasn't referring to hardware or media problems (though these are also good points), but the availability of software that can understand the data. For example, if you open a .zip file with a plain text editor, the content will appear as so much garbage. Without a zip program that can make sense of the data, being able to restore a .zip file doesn't give you anything useful (at least not without an awful lot of hacking). .zip is a common format and likely to be available in the future, but what about more obscure formats, where maybe you archive the files while you use the product? Suppose you switch to a new product that uses a different format, then decide 10 years from now you need to retrieve the data that was used by the old product. Will you still have a copy of that old product around that will run on current operating systems and hardware? Even if you archived the software and could in theory restore and run it, do you know where, amongst all that archive data, the software is? Will your successors be able to find the data and the software? Will they even know which software they need to run? Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence. CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: Thoughts on Monthly Archives
Thanks for your input Andy, yes these are all valid points. The requirement that we have is to be able to go back to the time of a decision and have available all pertinent information so that the same decision could be made again. I don't know who the optimist was who wrote the requirement, but somehow he got it approved by goverment, and I daresay it will take 10 years before the practical difficulties cause the requirement to change. My approach would be to specify archiving functionality in EVERY new application that is installed with a plain-text unload method such as XML. Then, there needs to be an archive process attached to every old application as it is retired. The issue here is not so much to hold every record, but to hold every record *once* rather than multiple times. I understand that this function is mandated to be in all new systems by 2006. I'm having trouble finding new employment in my current specialities (AIX and TSM - its a very small pond where I am) so I'm thinking of getting ahead of the game and becoming an archiving consultant. It will certainly be a world wide growth sector for the next few years. Every Queensland state government department will need to be doing a project in this area in the next year or two. Regards Steve. [EMAIL PROTECTED] 20/07/2004 2:25:00 Some considerations for long-term archive: - Much of today's data, as it is used from day to day, exists in some product-specific format. If you were to retrieve that data, say, 10 years from now, would you have software capable of reading that data? - Even if you archive the software, will operating systems 10 years from now be able to run that software? - Even if you archive the operating system installation files, will the hardware 10 years from now be able to install and run that operating system? - There is a good case to consider carefully what gets archived and how you archive it.For instance, maybe for database data, it would make sense to export that data to some common format, such as tab- or comma-delimited records, which is very likely to be importable by most software. Likewise, for image data, consider a format that is common today and likely to be common tomorrow. - 10 years from now, the people that need to retrieve the archived data will probably not be the same people who originally archived the data. Will your successors know what that data is? Will they know how to get to it? (Gee, we need to get at the accounts payable database from 10 years ago... under which node is it archived?) Will they know how to reconstruct it, and how to use it? I am by no means an expert in this area, but these are some things to consider carefully for long-term archives. Note that most of these issues are not directly related to TSM, but apply regardless of which data storage tool you use. Regards, Andy Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence. *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s), or if it is transmitted/received in error. Any unauthorised use, alteration, disclosure, distribution or review of this email is prohibited. It may be subject to a statutory duty of confidentiality if it relates to health service matters. If you are not the intended recipient(s), or if you have received this email in error, you are asked to immediately notify the sender by telephone or by return email. You should also delete this email and destroy any hard copies produced. ***
Re: Thoughts on Monthly Archives
Allen, Your post got me thinking as to just why I decided what I didand now I remember :) There was another management requirement that all production servers be backed up in full once per year and that snapshot be kept forever - there is no reasoning with this, its one of those stupid mandates that applies to the whole of the state government, and if data is not able to be categorized, then it must be kept. I think you'll recognise that having a third TSM Server for the yearly backup isn't really an option, so an archive mechanism is the only one that will work for that. Backupsets weren't an option when this was set up as they were very new and not mature (IMHO they're still not mature because I can't stack them, track them in a reasonable way, or use them for TDP data). Having an archive mechanism for monthlies means just changing the archive mgmtclass once a year and voila, your monthly becomes a yearly. Given that management loves the idea of permanent retention, database size is obviously not a problem. Hence the argument comes down to ease of maintenance and the monthly archive is better at that - I have a perl script that generates the backup schedules with correct domain statements based on the current backup filespaces every month, and each February we generate with the eternal mgmtclass. Its mostly automated. Steve. [EMAIL PROTECTED] 17/07/2004 2:07:39 == In article [EMAIL PROTECTED], Steve Harris [EMAIL PROTECTED] writes: at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) I think your math is good; I'd add though that there's a -GREAT- deal of locality of reference: Or in other words, if 3% of your files change a day, then for user filespace probably 95% of the files that change tomorrow will be the files that chaged yesterday, and so on. I think that 25 - 30% for userdir space is probably about right. Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. But I would disagree with your logic; In place of the incremental possibilities, which would have led you at worst above to backing up 60% every month, you're choosing to back up 100% every month, without fail. I think that this represents a substantially more costly strategy. In fact, given the monthly-for-five-years figure and your numbers above, it's about twice as expensive in facilities to archive monthly than to run incrementals with similar retention characteristics; If my guess is closer to correct, it's three- to four- times the cost. For a small amount of data, this may be cheaper than the human organizational time to build two sets of nodes; but by the time you get even medium size, I think it would be pretty expensive. - Allen S. Rout *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s), or if it is transmitted/received in error. Any unauthorised use, alteration, disclosure, distribution or review of this email is prohibited. It may be subject to a statutory duty of confidentiality if it relates to health service matters. If you are not the intended recipient(s), or if you have received this email in error, you are asked to immediately notify the sender by telephone or by return email. You should also delete this email and destroy any hard copies produced. ***
Re: Thoughts on Monthly Archives
Sorry for the late reply, weekend and all that but being from Australia as well you're in the same boat. :-) For our monthlys, on our main fileserver which has about a 1tb of data, we see an average of about ~12% change in data a month. I'm not sure the percentage of change for our other servers but most of the other files are just database dumps and web content, whose change rate is pretty constant. We only do selective backups for monthlys so only important (non-OS) data gets backed up. When we were doing full archives of our main fileserver each month, the database was climbing close to 10% extra and at one stage got to 80gb, getting to unwieldy. On average now we only use a bit over 2 LTO tapes a month for the monthlys. We already run three different nodes on our servers for different backups (daily inc, weekly inc and monthly inc) so adding a extra server to manage with it's own set of nodes won't be a huge problem. It's not a elegant TSM setup but it works. I was looking at modifying the management class for our daily backups so we could retire the weekly nodes, but I think we would have a huge blow-out in the amount of data being retained, forcing us to upgrade to a much bigger library for little gain. Our library is already at near capacity, don't think management would want to spend $10 on a bigger library. Cheers, Gordon Woodward Wintel Server Support [EMAIL PROTECTED] .QLD.GOV.AUTo: [EMAIL PROTECTED] Sent by: cc: [EMAIL PROTECTED]Subject: Re: Thoughts on Monthly Archives U 16/07/2004 11:53 AM Please respond to ADSM-L Gordon, I've though of doing this in the past, but, if we postulate a random daily change rate, then the chances of a particular file being changed in any one month are at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. Can I ask what rate of change you are seeing on your monthly backups? Thanks Steve Steve Harris AIX and TSM Admin Queensland Health, Brisbane Australia [EMAIL PROTECTED] 16/07/2004 11:22:20 We use to do full Monthly archives of all our important data on our servers but our TSM database was growing too rapidly, especially when we have to retain our backups for 7 years. What we ended up doing is registering a whole new alternate set of nodes specifically for Monthly backups and now just run incrementals. Our database doesn't grow as rapidly anymore and we don't chew through as many tapes. I'm thinking of now creating a second TSM instance and have that running exclusively for our monthly backups to take the load off our day to day operations. Gordon Woodward Wintel Server Support [EMAIL PROTECTED] Sent by: To: [EMAIL PROTECTED] [EMAIL PROTECTED]cc: EDU Subject: Thoughts on Monthly Archives 16/07/2004 07:00 AM Please respond to ADSM-L We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator -- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s
Re: Thoughts on Monthly Archives
How would I find that out? We're not even doing monthly backups yet - just trying to see the impending impact. From what I've read here so far: Much much more database usage (Obviously) much more tape usage I guess I'm just trying to deal with a TSM deployment that needs to be rebuilt in order to handle this operation. Our database is on a 15gig partition with no chance of extending that. We'll have to purchase two more drives and move it to those drives, which'll entail MOVING the database (this scares me...) - mike -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Steve Harris Sent: Thursday, July 15, 2004 7:53 PM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives Gordon, I've though of doing this in the past, but, if we postulate a random daily change rate, then the chances of a particular file being changed in any one month are at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. Can I ask what rate of change you are seeing on your monthly backups? Thanks Steve Steve Harris AIX and TSM Admin Queensland Health, Brisbane Australia [EMAIL PROTECTED] 16/07/2004 11:22:20 We use to do full Monthly archives of all our important data on our servers but our TSM database was growing too rapidly, especially when we have to retain our backups for 7 years. What we ended up doing is registering a whole new alternate set of nodes specifically for Monthly backups and now just run incrementals. Our database doesn't grow as rapidly anymore and we don't chew through as many tapes. I'm thinking of now creating a second TSM instance and have that running exclusively for our monthly backups to take the load off our day to day operations. Gordon Woodward Wintel Server Support [EMAIL PROTECTED] Sent by: To: [EMAIL PROTECTED] [EMAIL PROTECTED]cc: EDU Subject: Thoughts on Monthly Archives 16/07/2004 07:00 AM Please respond to ADSM-L We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator -- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s), or if it is transmitted/received in error. Any unauthorised use, alteration, disclosure, distribution or review of this email is prohibited. It may be subject to a statutory duty of confidentiality if it relates to health service matters. If you are not the intended recipient(s), or if you have received this email in error, you are asked to immediately notify the sender by telephone or by return email. You should also delete this email and destroy any hard copies produced. ***
Re: Thoughts on Monthly Archives
Mike you don't have to move your database to enlarge it. if you add disk space you can create new volumes and then you just extend the database over these new volumes to gain the extra space that you need. Ie: in our case we had to increase our database from 10 GB to 20 GB so this is what we did. Database Volumes Volume Name (Copy 1) Volume Name (Copy 2) Volume Name (Copy 3) E:\DB1.DSM F:\DB1.DSM E:\DB2.DSM F:\DB2.DSM Hope this helps your comfort zone a bit. Also with our system because we are a banking system we have to do monthly archives and retain them for 7 years by law, so we have a separate policy for archive pools and we move these out when they are full to a shelf in our server room. This is working well for us as it does not take up all the slots in the library with archived tapes. The only downside to this is when you do a retrieve you have to watch for TSM asking for specific tapes to be put in from the shelf and then make sure to move them back out again after the retrieve process. A bit more work but it's not been bad. Tammy Schellenberg Systems Administrator, MCP Prospera Credit Union email: [EMAIL PROTECTED] DID: 604-864-6578 -Original Message- From: Mike Bantz [mailto:[EMAIL PROTECTED] Sent: Friday, July 16, 2004 7:40 AM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives How would I find that out? We're not even doing monthly backups yet - just trying to see the impending impact. From what I've read here so far: Much much more database usage (Obviously) much more tape usage I guess I'm just trying to deal with a TSM deployment that needs to be rebuilt in order to handle this operation. Our database is on a 15gig partition with no chance of extending that. We'll have to purchase two more drives and move it to those drives, which'll entail MOVING the database (this scares me...) - mike -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Steve Harris Sent: Thursday, July 15, 2004 7:53 PM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives Gordon, I've though of doing this in the past, but, if we postulate a random daily change rate, then the chances of a particular file being changed in any one month are at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. Can I ask what rate of change you are seeing on your monthly backups? Thanks Steve Steve Harris AIX and TSM Admin Queensland Health, Brisbane Australia [EMAIL PROTECTED] 16/07/2004 11:22:20 We use to do full Monthly archives of all our important data on our servers but our TSM database was growing too rapidly, especially when we have to retain our backups for 7 years. What we ended up doing is registering a whole new alternate set of nodes specifically for Monthly backups and now just run incrementals. Our database doesn't grow as rapidly anymore and we don't chew through as many tapes. I'm thinking of now creating a second TSM instance and have that running exclusively for our monthly backups to take the load off our day to day operations. Gordon Woodward Wintel Server Support [EMAIL PROTECTED] Sent by: To: [EMAIL PROTECTED] [EMAIL PROTECTED]cc: EDU Subject: Thoughts on Monthly Archives 16/07/2004 07:00 AM Please respond to ADSM-L We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator -- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden
Re: Thoughts on Monthly Archives
== In article [EMAIL PROTECTED], Steve Harris [EMAIL PROTECTED] writes: at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) I think your math is good; I'd add though that there's a -GREAT- deal of locality of reference: Or in other words, if 3% of your files change a day, then for user filespace probably 95% of the files that change tomorrow will be the files that chaged yesterday, and so on. I think that 25 - 30% for userdir space is probably about right. Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. But I would disagree with your logic; In place of the incremental possibilities, which would have led you at worst above to backing up 60% every month, you're choosing to back up 100% every month, without fail. I think that this represents a substantially more costly strategy. In fact, given the monthly-for-five-years figure and your numbers above, it's about twice as expensive in facilities to archive monthly than to run incrementals with similar retention characteristics; If my guess is closer to correct, it's three- to four- times the cost. For a small amount of data, this may be cheaper than the human organizational time to build two sets of nodes; but by the time you get even medium size, I think it would be pretty expensive. - Allen S. Rout
Re: Thoughts on Monthly Archives
== In article [EMAIL PROTECTED], Mike Bantz [EMAIL PROTECTED] writes: How would I find that out? We're not even doing monthly backups yet - just trying to see the impending impact. Count your incremental size over a series of days, and you'll have a measure of the instantaneous change rate. Count the number of different copies of various files in your backups table (BIG LONG EXPENSIVE QUERY) and you'll have a sense of the longer-term rate of change. You can also poke around in your database for how many things have changed since 'a month ago'. May I suggest keeping a 'select * from columns order by tabname' hanging around to inform you about your query options.. :) From what I've read here so far: Much much more database usage In round terms, multiply by number of copies. So if you're retaining three copies now, and you are contemplating monthly-for-5-years, then multiply DB space by something like 20 (!) ... (!) I guess I'm just trying to deal with a TSM deployment that needs to be rebuilt in order to handle this operation. Our database is on a 15gig partition with no chance of extending that. We'll have to purchase two more drives and move it to those drives, which'll entail MOVING the database (this scares me...) Don't be scared of moving the DB; you can shift stuff to new places pretty easily, especially if you're willing to maintain the same-size DB volumes. You can just add a third mirror on the new place, then blow away a mirror on the old; repeat until done. Where you run into nervous-making changes is when you are trying to change the container -size-. That still makes me shudder. - Allen S. Rout - ... (!)
Re: Thoughts on Monthly Archives
In talking with a whole ton of people about this, this is the tentative idea: This is for document retention, so the recovery of a whole server isn't a requirement. If I can get back a MSSQL database file, I can restore it anywhere. Same goes for a .doc or .xls file - I don't need the whole server. So my plan is to create a storage pool, call it ARCHIVE_STGPOOL or something. Load a bunch of scratch tapes in the 3583, run my monthly archives to that stgpool. When they're done, check them out, mark them offsite, etc. But we'd run incrementals, not monthly fulls, to keep down the db and tape usage. We're a fairly small shop (now) and only use around 3 tapes a day. I *love* that TSM is so flexible and I hate that I get to agonize and debate any changes to it for that very reason. We've been bitten really hard in the past due to poor planning, so I'm trying to plan for my next 4-7 years now. :-) -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Friday, July 16, 2004 10:08 AM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives == In article [EMAIL PROTECTED], Steve Harris [EMAIL PROTECTED] writes: at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) I think your math is good; I'd add though that there's a -GREAT- deal of locality of reference: Or in other words, if 3% of your files change a day, then for user filespace probably 95% of the files that change tomorrow will be the files that chaged yesterday, and so on. I think that 25 - 30% for userdir space is probably about right. Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. But I would disagree with your logic; In place of the incremental possibilities, which would have led you at worst above to backing up 60% every month, you're choosing to back up 100% every month, without fail. I think that this represents a substantially more costly strategy. In fact, given the monthly-for-five-years figure and your numbers above, it's about twice as expensive in facilities to archive monthly than to run incrementals with similar retention characteristics; If my guess is closer to correct, it's three- to four- times the cost. For a small amount of data, this may be cheaper than the human organizational time to build two sets of nodes; but by the time you get even medium size, I think it would be pretty expensive. - Allen S. Rout
Re: Thoughts on Monthly Archives
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Mike Bantz I *love* that TSM is so flexible and I hate that I get to agonize and debate any changes to it for that very reason. We've been bitten really hard in the past due to poor planning, so I'm trying to plan for my next 4-7 years now. :-) Would that more admins put as much work into planning! Long-term file preservation can be difficult to plan. The best approach is to go to the customer and ask what the bottom line is--what is it exactly that he/she wants, rather than passively accepting customer-dictated methodology. I want a full backup of my machine once a week doesn't necessarily mean Run a selective backup of my machine once a week. -- Mark Stapleton
Re: Thoughts on Monthly Archives
If you keep the tapes outside the system, what do you do when there is an opportunity to change tape technology in 5 years? How do you migrate that data to new tapes? [RC] Dwight Cook [EMAIL PROTECTED] To:[EMAIL PROTECTED] Sent by: ADSM: Dist cc: Stor Manager Subject: Re: Thoughts on Monthly Archives [EMAIL PROTECTED] 07/15/2004 05:22 PM Please respond to ADSM: Dist Stor Manager |---| | [ ] Secure E-mail | |---| Sure... started a long time ago when directories were bound to the longest retention management class that existed... with a 10 year archive management class defined, I was keeping a lot of stuff WAY to long :-( made me realize I might not want such management classes existing in the environment Remember, with exports, well really with the imports, you can use relative dates so you don't even need a long term management class to keep it a long time. Also, when the user show up at your office and says... remember the set of archives done right before that big merger? make those 10 year retention rather than 7 This way, all you have to do is adjust the spread sheet you track the exported tapes in and you only impact the items you really want to alter. and this points out a wonderful thing about TSM... lots and lots of different ways to get the same thing done, very flexible to meet the needs of just about any situation Dwight E. Cook Systems Management Integration Professional, Advanced Integrated Storage Management TSM Administration (918) 925-8045 Mike Bantz [EMAIL PROTECTED] To Sent by: ADSM: [EMAIL PROTECTED] Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Re: Thoughts on Monthly Archives 07/15/2004 05:43 PM Please respond to ADSM: Dist Stor Manager personally I don't like to keep any archived data in an active TSM server environment that has a retention value of more than 370 days Can I ask why? This seems like an **awfully** short time... - mike (Embedded image moved to file: pic14137.gif)(Embedded image moved to file: pic26055.gif)(Embedded image moved to file: pic00161.gif) == IMPORTANT NOTICE: This communication, including any attachment, contains information that may be confidential or privileged, and is intended solely for the entity or individual to whom it is addressed. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message is strictly prohibited. Nothing in this email, including any attachment, is intended to be a legally binding signature. == attachment: pic14137.gifattachment: pic26055.gifattachment: pic00161.gif
Re: Thoughts on Monthly Archives
I would assume: restore, back it up again, rotate it back offsite? It's a pain, but the technology moves somewhat slowly... -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Robert Clark Sent: Friday, July 16, 2004 12:47 PM To: [EMAIL PROTECTED] Subject: Re: Thoughts on Monthly Archives If you keep the tapes outside the system, what do you do when there is an opportunity to change tape technology in 5 years? How do you migrate that data to new tapes? [RC] Dwight Cook [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent by: ADSM: Dist cc: Stor Manager Subject: Re: Thoughts on Monthly Archives [EMAIL PROTECTED] 07/15/2004 05:22 PM Please respond to ADSM: Dist Stor Manager |---| | [ ] Secure E-mail | |---| Sure... started a long time ago when directories were bound to the longest retention management class that existed... with a 10 year archive management class defined, I was keeping a lot of stuff WAY to long :-( made me realize I might not want such management classes existing in the environment Remember, with exports, well really with the imports, you can use relative dates so you don't even need a long term management class to keep it a long time. Also, when the user show up at your office and says... remember the set of archives done right before that big merger? make those 10 year retention rather than 7 This way, all you have to do is adjust the spread sheet you track the exported tapes in and you only impact the items you really want to alter. and this points out a wonderful thing about TSM... lots and lots of different ways to get the same thing done, very flexible to meet the needs of just about any situation Dwight E. Cook Systems Management Integration Professional, Advanced Integrated Storage Management TSM Administration (918) 925-8045 Mike Bantz [EMAIL PROTECTED] To Sent by: ADSM: [EMAIL PROTECTED] Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Re: Thoughts on Monthly Archives 07/15/2004 05:43 PM Please respond to ADSM: Dist Stor Manager personally I don't like to keep any archived data in an active TSM server environment that has a retention value of more than 370 days Can I ask why? This seems like an **awfully** short time... - mike (Embedded image moved to file: pic14137.gif)(Embedded image moved to file: pic26055.gif)(Embedded image moved to file: pic00161.gif) == IMPORTANT NOTICE: This communication, including any attachment, contains information that may be confidential or privileged, and is intended solely for the entity or individual to whom it is addressed. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message is strictly prohibited. Nothing in this email, including any attachment, is intended to be a legally binding signature. ==
Re: Thoughts on Monthly Archives
one thought... for clients that require the monthly 4 year archives... register alternate nodes by the name of nodename_exp push any data required for long term retention (in the form of either backups or archives or both) using the client nodename_exp export that node remove from environment... doesn't take up space in the tsm database doesn't take up space in the tsm ATL personally I don't like to keep any archived data in an active TSM server environment that has a retention value of more than 370 days Dwight E. Cook Systems Management Integration Professional, Advanced Integrated Storage Management TSM Administration (918) 925-8045 Mike Bantz [EMAIL PROTECTED] To Sent by: ADSM: [EMAIL PROTECTED] Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Thoughts on Monthly Archives 07/15/2004 04:00 PM Please respond to ADSM: Dist Stor Manager We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator inline: graycol.gifinline: pic32595.gifinline: ecblank.gif
Re: Thoughts on Monthly Archives
personally I don't like to keep any archived data in an active TSM server environment that has a retention value of more than 370 days Can I ask why? This seems like an **awfully** short time... - mike
Re: Thoughts on Monthly Archives
Sure... started a long time ago when directories were bound to the longest retention management class that existed... with a 10 year archive management class defined, I was keeping a lot of stuff WAY to long :-( made me realize I might not want such management classes existing in the environment Remember, with exports, well really with the imports, you can use relative dates so you don't even need a long term management class to keep it a long time. Also, when the user show up at your office and says... remember the set of archives done right before that big merger? make those 10 year retention rather than 7 This way, all you have to do is adjust the spread sheet you track the exported tapes in and you only impact the items you really want to alter. and this points out a wonderful thing about TSM... lots and lots of different ways to get the same thing done, very flexible to meet the needs of just about any situation Dwight E. Cook Systems Management Integration Professional, Advanced Integrated Storage Management TSM Administration (918) 925-8045 Mike Bantz [EMAIL PROTECTED] To Sent by: ADSM: [EMAIL PROTECTED] Dist Stor cc Manager [EMAIL PROTECTED] Subject .EDU Re: Thoughts on Monthly Archives 07/15/2004 05:43 PM Please respond to ADSM: Dist Stor Manager personally I don't like to keep any archived data in an active TSM server environment that has a retention value of more than 370 days Can I ask why? This seems like an **awfully** short time... - mike inline: graycol.gifinline: pic15646.gifinline: ecblank.gif
Re: Thoughts on Monthly Archives
We use to do full Monthly archives of all our important data on our servers but our TSM database was growing too rapidly, especially when we have to retain our backups for 7 years. What we ended up doing is registering a whole new alternate set of nodes specifically for Monthly backups and now just run incrementals. Our database doesn't grow as rapidly anymore and we don't chew through as many tapes. I'm thinking of now creating a second TSM instance and have that running exclusively for our monthly backups to take the load off our day to day operations. Gordon Woodward Wintel Server Support [EMAIL PROTECTED] Sent by: To: [EMAIL PROTECTED] [EMAIL PROTECTED]cc: EDU Subject: Thoughts on Monthly Archives 16/07/2004 07:00 AM Please respond to ADSM-L We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator -- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Thoughts on Monthly Archives
Gordon, I've though of doing this in the past, but, if we postulate a random daily change rate, then the chances of a particular file being changed in any one month are at 1% , 1-(0.99**30), or about .25 at 2%, 1-(0.98**30) , or about .45 at 3%, 1-(0.97**30), or about .60 (Please feel free to correct my maths if I'm wrong - probability was never my strong point) Now, of course its often the same files which change day after day, so real experience should be better than this, but at the time, I decided that the overhead of mainitianing two TSMs (and two clients per node) wasn't worth the benefit, and went with archives. Can I ask what rate of change you are seeing on your monthly backups? Thanks Steve Steve Harris AIX and TSM Admin Queensland Health, Brisbane Australia [EMAIL PROTECTED] 16/07/2004 11:22:20 We use to do full Monthly archives of all our important data on our servers but our TSM database was growing too rapidly, especially when we have to retain our backups for 7 years. What we ended up doing is registering a whole new alternate set of nodes specifically for Monthly backups and now just run incrementals. Our database doesn't grow as rapidly anymore and we don't chew through as many tapes. I'm thinking of now creating a second TSM instance and have that running exclusively for our monthly backups to take the load off our day to day operations. Gordon Woodward Wintel Server Support [EMAIL PROTECTED] Sent by: To: [EMAIL PROTECTED] [EMAIL PROTECTED]cc: EDU Subject: Thoughts on Monthly Archives 16/07/2004 07:00 AM Please respond to ADSM-L We are implementing the following: Monthly FULL backups, saved for 4 years for document retention purposes. Backupsets are not viable, as a restore would be a pain. So it looks like we're going to do this with archives. My idea is to create another storage pool with it's own media, another domain with it's own management class (we really need only one: save 1 version of this file for 4 years), and remove that media once the archive runs. Anyone care to comment on if this is the best, easiest, way to do this? Or comment on what to expect with things like tape usage and database size? We're currently at 57% of a 12gig database. Thanks in advance!!! Mike Bantz Systems Administrator -- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. *** This email, including any attachments sent with it, is confidential and for the sole use of the intended recipient(s). This confidentiality is not waived or lost, if you receive it and you are not the intended recipient(s), or if it is transmitted/received in error. Any unauthorised use, alteration, disclosure, distribution or review of this email is prohibited. It may be subject to a statutory duty of confidentiality if it relates to health service matters. If you are not the intended recipient(s), or if you have received this email in error, you are asked to immediately notify the sender by telephone or by return email. You should also delete this email and destroy any hard copies produced. ***
