Hi David, Thank you (and the group) for your suggestions, recommendations, experience, etc. It is greatly appreciated and has helped a lot.
I understand what you are saying about a "realistic" test system but it is simply not possible/feasible. We barely have enough storage for our production systems and having to use slow NFS/ISILON storage hinders the backup performance but is the only storage (besides server internal disk) available. We were using some old Dell Powervaults but those turned out to be problematic (see below)! I have been fighting for a properly designed/configured "enterprise data protection" system for the 25+ years (i.e. ADSM and earlier) I have been responsible for "backups" - to no avail! As I mentioned in my initial post, the reason I am focused on Archives is two-fold. First, there is a high probability that we will be moving away from IBM Spectrum Protect to another "backup solution" and I need to be prepared to extract any long-term archives from ISP. Second, since we have to move to a new, much smaller datacenter by the end of 2023, the magnetic tape subsystem/robot we currently use will NOT be moving (or be replaced) to our new location. As I understand it, if we do decide to move away from/terminate ISP, we can continue to run an ISP server, unsupported of course. My thoughts were that I could retain the archives we need to keep beyond 2023 on a single ISP server in container pools, until they finally expire. FWIW, I want to move to container pools due to the disaster we had when one of our Dell Powervault storage systems failed and we had to restore 150TB of backups. It took 3-months of constantly running restores against 200+ tapes, audits (stgpool & volumes), movedata, reclaims, more restores, audits, rinse-lather-repeat, due to the FILE devclass "weaknesses" and dedupe. Support kept telling me how much better containers would be and we would have avoided most of the recovery issues we went through! On Tue, Jun 28, 2022 at 1:29 AM David L.A. De Leeuw <da...@bgu.ac.il> wrote: > Hi Zoltan, > > I am not the expert on ISP as other writers here. But still will give you > my opinion: > > Building a test system which is so constricted that it doesn't resemble > real life doesn't make sense to me. > > The container based production system is much much easier to handle than > the old file based, tape copy pool etc. > If you have enough disk space to contain your present primary disk and > tape holdings (not including copypools) just take the jump. > > As the others pointed out you can stop and start the conversion. Don't > differentiate between backups, archives, nodes. That defeats the whole > point of the system. > Once you complete it you don't understand how you managed without it. > > Just read some of the IBM docs such as > > https://www.ibm.com/support/pages/best-practices-ibm-spectrum-protect-storage-pool-conversion > > We put all our ISP system on the containers (which are on JBOD disks) . > Make daily scheduled "protect stgpool" copies to tape. Store one set in a > safe and that is it. > Once the system is running, the daily turnover is low, because of dedup > and compression. > Success ! > David > > > > -----Original Message----- > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Zoltan > Forray > Sent: Monday, June 27, 2022 10:03 PM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] AW: [ADSM-L] AW: [ADSM-L] Moving archives into > containers > > My production ISP servers have 3TB SSD for the DB/logs, 150TB internal > disks and multiple NFS/ISILON and other disks as needed as well as tape. > These are test ISP servers, thus the small amount of disk. > > I am thinking of converting the production systems (if we stay with ISP - > we might move to a different "Enterprise Protection" platform which makes > this moot) to directory-containers and want to get experienced with the > conversion processes, first. > > On Mon, Jun 27, 2022 at 2:29 PM David L.A. De Leeuw <da...@bgu.ac.il> > wrote: > > > Zoltan, > > > > Difficult to imagine working with tsm with such small disks. > > > > You can mount any disk system, anywhere in your environment and > > declare as the filespaces, for containers, or for FILE systems. > > > > We have 100 TB mounted on a system, some 2 km away from the server for > > containers, and 1 TB for database backups. > > > > TSM is extremely flexible concerning containerdirs, or filespaces. > > Just make sure to share the disks, make sure that the user running the > > server has full read/write access on the remote system and go. > > > > Using a mini filesystem for this operation, copying and deleting, is > > asking for trouble IMHO. > > > > David. > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of > > Zoltan Forray > > Sent: Monday, June 27, 2022 9:01 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: [ADSM-L] AW: [ADSM-L] AW: [ADSM-L] Moving archives into > > containers > > > > Since I did the RESET CONVERSION against the stgpool I "converted" to > > container, I assume there won't be any collisions if I use the same > > disk filesystem/mount as both FILE and CONTAINER since I only have > > 1-10GB internal disk storage available? > > > > On Mon, Jun 27, 2022 at 1:50 PM Uwe Schreiber > > <uwe.h.schrei...@t-online.de > > > > > wrote: > > > > > Hi Zoltan, > > > > > > DB Backup to Virtual Volumes ist not working with Container Pools. > > > You have to use sequential Tape or sequential File Volumes. > > > > > > Regards, Uwe > > > > > > -----Ursprüngliche Nachricht----- > > > Von: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> Im Auftrag von > > > Zoltan Forray > > > Gesendet: Montag, 27. Juni 2022 19:43 > > > An: ADSM-L@VM.MARIST.EDU > > > Betreff: Re: [ADSM-L] AW: [ADSM-L] Moving archives into containers > > > > > > Next question - how do you perform a DB Backup via Virtual Volume if > > > you can't use a container and have converted the only stgpool you > > > have to container? > > > > > > We have 2-test ISP servers - 1-physical and 1-VM. The VM test ISP > > > server performs DBBackups to the physical one. After I converted > > > the stgpool on the physical test ISP and updated the managementclass > > > to point to the container, now the VM test DBBackup fails with: > > > > > > 6/27/2022 1:25:02 PM ANR2776W Transaction failed for session 6688 > > > for node TSMTESTVM (Linux/x86_64) - A storage pool for the target > > > destination is associated with a container or cloud storage pool. > > > > > > Am I missing some setting required to do this or is this simply not > > > allowed? > > > > > > On Mon, Jun 27, 2022 at 10:43 AM Uwe Schreiber < > > > uwe.h.schrei...@t-online.de> > > > wrote: > > > > > > > HI Zoltan, > > > > > > > > yes, as soon as the "convert stgpool" is getting started, the ISP > > > > server prevents any further ingest of data in the source. > > > > Normally before a "convert stgpool" the management class is > > > > adjusted accordingly, so that the clients don't have to wait until > > > > the convert is done. > > > > The "reset convertion" will "open" the pool for further ingest of > > > > data (or another convert). > > > > > > > > Regards, Uwe > > > > > > > > > > > > -----Ursprüngliche Nachricht----- > > > > Von: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> Im Auftrag von > > > > Zoltan Forray > > > > Gesendet: Montag, 27. Juni 2022 16:05 > > > > An: ADSM-L@VM.MARIST.EDU > > > > Betreff: Re: [ADSM-L] Moving archives into containers > > > > > > > > Thanks for the details/info. So, performing the "*RESET CONVERSION*" > > > > makes the source stgpool accessible/reusable. At this point, I > > > > assume the source stgpool is empty? I am guessing IBM did this to > > > > prevent client backups from landing in the source stgpool until > > > > you have the chance to change the managementclass to point client > > > > backups to the directory-container stgpool? > > > > > > > > On Mon, Jun 27, 2022 at 9:10 AM MM <michael.mal...@mm-it.at> wrote: > > > > > > > > > Hallo Zoltan, > > > > > > > > > > more infos about the undocumented (and very useful) "*RESET > > > CONVERSION*" > > > > > cmd: > > > > > > > > > > After a successful conversion the *Access:* status > > > > > > > > > > (see: q stg <your_pool_name> f=d) will be shown as "*Converted*" > > > > > and you can NOT backup new data into that pool any more. > > > > > > > > > > *BUT*, if you say/enter *RESET CONVERSION" <your_pool_name>*, > > > > > > > > > > you change that *Access:* status to the original *Access:* status > " > > > > > *Read/Write*" again. > > > > > > > > > > Thats all..... > > > > > > > > > > rgds and good look Michael M. > > > > > > > > > > Zoltan Forray <zfor...@vcu.edu> hat am 27.06.2022 13:47 > geschrieben: > > > > > > > > > > > > > > > Thanks for the suggestions and I would like more info on the > > > > > undocumented "reset conversion" command and what it does. > > > > > > > > > > Unfortunately, since archives are <0.1% of total occupancy, we > > > > > didn't separate them from the backups that roll-off to tape. But > > > > > these suggestions do give me lots to go on. Again, thanks for > > > > > the > > help! > > > > > > > > > > On Mon, Jun 27, 2022 at 2:43 AM Loon, Eric van (ITOP DI) - KLM < > > > > > eric-van.l...@klm.com> wrote: > > > > > > > > > > Hi David, > > > > > > > > > > Of course that can be an option too, but I suggested the > > > > > filepool in between because Zoltan only wanted to move the > archives. > > > > > > > > > > Kind regards, > > > > > Eric van Loon > > > > > Core Infra > > > > > > > > > > -----Original Message----- > > > > > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf > > > > > Of David L.A. De Leeuw > > > > > Sent: zondag 26 juni 2022 09:55 > > > > > To: ADSM-L@VM.MARIST.EDU > > > > > Subject: Re: Moving archives into containers > > > > > > > > > > Hi Eric and Zoltan > > > > > > > > > > Maybe I missed something. Why not directly convert stgpool from > > > > > the tape storage pool. > > > > > We did this a year ago, took with us the archives and backups > > > > > (Spectrum Protect doesn't care) and finished the convert in a > > > > > few weeks (using pretty slow equipment). > > > > > Why the manual work ? > > > > > > > > > > David de Leeuw > > > > > Medical Computing Unit > > > > > Ben-Gurion University of the Negev Beer Sheva Israel > > > > > > > > > > -----Original Message----- > > > > > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf > > > > > Of Loon, Eric van (ITOP DI) - KLM > > > > > Sent: Friday, June 24, 2022 6:22 PM > > > > > To: ADSM-L@VM.MARIST.EDU > > > > > Subject: Re: [ADSM-L] Moving archives into containers > > > > > > > > > > Hi Zoltan, > > > > > > > > > > I think it's an undocumented command. It's not in the manuals, > > > > > nor in the help, but I have used in in the past, at least in > > > > > earlier > > > > > 8.1 > > > > versions. > > > > > Since you're constrained on disk space, moving relatively small > > > > > amount of clients at once to your small filepool would be an > option. > > > > > The convert will move the data to your container pool and the > > > > > reset command will allow you to move the next batch of clients > > > > > to your filepool. > > > > > I know, it's a lot of manual work, but I have moved all out > > > > > long-term archives from our old servers into our new > > > > > (container-based) servers this way. > > > > > > > > > > Kind regards, > > > > > Eric van Loon > > > > > Core Infra > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf > > > > > Of Zoltan Forray > > > > > Sent: vrijdag 24 juni 2022 16:15 > > > > > To: ADSM-L@VM.MARIST.EDU > > > > > Subject: Re: Moving archives into containers > > > > > > > > > > Hi Eric, > > > > > > > > > > Thanks for the recommendations. I always thought the conversion > > > > > process required the target to be empty. Unfortunately, we > > > > > simply do not have sufficient free disk space to perform > > > > > complete conversions to containers since everything existing > > > > > stgpools are either on NFS/ISILON (yes I know IBM says this is > > > > > not-recommended) or server > > > internal disk. > > > > > Right now I created my first directory/container pool since we > > > > > had > > > > > a(nother) hard drive failure in our 5+ year old Powervault so we > > > > > took this opportunity to learn directory-containers and their > > > limitations. > > > > > > > > > > Not sure what you mean by "reset conversion command". I couldn't > > > > > find anything like that in the latest 8.1.14 manual? > > > > > > > > > > On Fri, Jun 24, 2022 at 9:20 AM Loon, Eric van (ITOP DI) - KLM < > > > > > eric-van.l...@klm.com> wrote: > > > > > > > > > > Hi Zoltan, > > > > > > > > > > What you could do is create a file device class and move the > > > > > files from tape to this file device class. As soon as the files > > > > > are there, you can use the convert stgpool command to move the > > > > > data into the directory container stgpool. > > > > > If you do this in batches, you don't need a very large filepool. > > > > > After emptying your filepool, you can use the reset conversion > > > > > command to make the filepool available for new data. > > > > > > > > > > Kind regards, > > > > > Eric van Loon > > > > > Air France/KLM > > > > > > > > > > -----Original Message----- > > > > > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf > > > > > Of Zoltan Forray > > > > > Sent: vrijdag 24 juni 2022 14:39 > > > > > To: ADSM-L@VM.MARIST.EDU > > > > > Subject: Moving archives into containers > > > > > > > > > > We are currently in the midst of a big project to relocate our > > > > > datacenter, by the end of 2023, to a new, smaller building. > > > > > > > > > > At the same time, we are actively pursuing redesigning/replacing > > > > > our "Enterprise Data Protection" solution which may likely move > > > > > us completely away from Spectrum Protect. FWIW, this project > > > > > started before we were told we had to move out of our existing > location. > > > > > > > > > > As an absolute minimum, we need to get everything off old > > > > > magnetic tape > > > > > (3592) that we need to retain beyond 2023 (archives), since the > > > > > new datacenter will not have the physical space to accommodate > tapes. > > > > > > > > > > Now that we have upgraded all ISP servers to 8.1.14.100/eFix > > > > > 102, we have started playing with directory-containers, since > > > > > there is now support for "protecting" containers to magnetic > > > > > tape (for offsite > > > > > > > > > > backups). > > > > > > > > > > > > > > > > With the additional directory-container support/tools, we were > > > > > wondering if there is a way to get backups/archives into a > > > > > directory-container stgpool other than during ingestion from a > > client? > > > > > > > > > > If we do move away from Spectrum Protect, is the only way to > > > > > retain long-term archives is to recover them to the original > > > > > platform/server and then move them to whatever long-term storage > > > platform we choose (i.e. > > > > > object, cloud)? > > > > > > > > > > -- > > > > > *Zoltan Forray* > > > > > Backup & VMware Systems Administrator Enterprise Compute & > > > > > Storage Platforms VCU Infrastructure Services www.ucc.vcu.edu > > > > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU > > > > > and other reputable organizations will never use email to > > > > > request that you reply with your password, social security > > > > > number or confidential personal information. For more details > > > > > visit http://phishing.vcu.edu/ > > > > > <https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?u > > > > > rl > > > > > =h > > > > > tt > > > > > ps%3a%2f%2fadminmicro2.questionpro.com&umid=1CBA4B97-E233-2405-8 > > > > > 50 > > > > > A- > > > > > A3 > > > > > 4E51B31D92&auth=701951cf8dfb7cd99762e0f5be252082d875f1cf-9ffb57b > > > > > c3 > > > > > 70 > > > > > 88 > > > > > ff312b2517c52a2ea9888470669> > > > > > ******************************************************** > > > > > For information, services and offers, please visit our web site: > > > > > http://www.klm.com. This e-mail and any attachment may contain > > > > > confidential and privileged material intended for the addressee > only. > > > > > If you are not the addressee, you are notified that no part of > > > > > the e-mail or any attachment may be disclosed, copied or > > > > > distributed, and that any other action related to this e-mail or > > > > > attachment is strictly prohibited, and may be unlawful. If you > > > > > have received this e-mail by error, please notify the sender > > > > > immediately by return e-mail, and delete > > > > > > > > > > this message. > > > > > > > > > > > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > > > > and/or its employees shall not be liable for the incorrect or > > > > > incomplete transmission of this e-mail or any attachments, nor > > > > > responsible for any > > > > > > > > > > delay in receipt. > > > > > > > > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM > > > > > Royal Dutch > > > > > Airlines) is registered in Amstelveen, The Netherlands, with > > > > > registered number 33014286 > > > > > ******************************************************** > > > > > > > > > > > > > > > > > > > > -- > > > > > *Zoltan Forray* > > > > > Backup & VMware Systems Administrator Enterprise Compute & > > > > > Storage Platforms VCU Infrastructure Services www.ucc.vcu.edu > > > > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU > > > > > and other reputable organizations will never use email to > > > > > request that you reply with your password, social security > > > > > number or confidential personal information. For more details > > > > > visit http://phishing.vcu.edu/ < > > > > > > > > > > https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?ur > > > > > l= > > > > > ht > > > > > tp > > > > > s%3a%2f%2fadminmicro2.questionpro.com&umid=1CBA4B97-E233-2405-85 > > > > > 0A > > > > > -A > > > > > 34 > > > > > E51B31D92&auth=701951cf8dfb7cd99762e0f5be252082d875f1cf-9ffb57bc > > > > > 37 > > > > > 08 > > > > > 8f > > > > > f312b2517c52a2ea9888470669 > > > > > > > > > > > ******************************************************** > > > > > For information, services and offers, please visit our web site: > > > > > http://www.klm.com. This e-mail and any attachment may contain > > > > > confidential and privileged material intended for the addressee > only. > > > > > If you are not the addressee, you are notified that no part of > > > > > the e-mail or any attachment may be disclosed, copied or > > > > > distributed, and that any other action related to this e-mail or > > > > > attachment is strictly prohibited, and may be unlawful. If you > > > > > have received this e-mail by error, please notify the sender > > > > > immediately by return e-mail, and delete this message. > > > > > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > > > > and/or its employees shall not be liable for the incorrect or > > > > > incomplete transmission of this e-mail or any attachments, nor > > > > > responsible for any delay in receipt. > > > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM > > > > > Royal Dutch > > > > > Airlines) is registered in Amstelveen, The Netherlands, with > > > > > registered number 33014286 > > > > > ******************************************************** > > > > > ******************************************************** > > > > > For information, services and offers, please visit our web site: > > > > > http://www.klm.com. This e-mail and any attachment may contain > > > > > confidential and privileged material intended for the addressee > only. > > > > > If you are not the addressee, you are notified that no part of > > > > > the e-mail or any attachment may be disclosed, copied or > > > > > distributed, and that any other action related to this e-mail or > > > > > attachment is strictly prohibited, and may be unlawful. If you > > > > > have received this e-mail by error, please notify the sender > > > > > immediately by return e-mail, and delete this message. > > > > > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > > > > and/or its employees shall not be liable for the incorrect or > > > > > incomplete transmission of this e-mail or any attachments, nor > > > > > responsible for any delay in receipt. > > > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM > > > > > Royal Dutch > > > > > Airlines) is registered in Amstelveen, The Netherlands, with > > > > > registered number 33014286 > > > > > ******************************************************** > > > > > > > > > > > > > > > > > > > > -- > > > > > *Zoltan Forray* > > > > > Backup & VMware Systems Administrator Enterprise Compute & > > > > > Storage Platforms VCU Infrastructure Services www.ucc.vcu.edu > > > > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU > > > > > and other reputable organizations will never use email to > > > > > request that you reply with your password, social security > > > > > number or confidential personal information. For more details > > > > > visit http://phishing.vcu.edu/ > > > > > <https://adminmicro2.questionpro.com> > > > > > > > > > > > > > > > > > > -- > > > > *Zoltan Forray* > > > > Backup & VMware Systems Administrator Enterprise Compute & Storage > > > > Platforms VCU Infrastructure Services www.ucc.vcu.edu > > > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU > > > > and other reputable organizations will never use email to request > > > > that you reply with your password, social security number or > > > > confidential personal information. For more details visit > > > > http://phishing.vcu.edu/ <https://adminmicro2.questionpro.com> > > > > > > > > > > > > > -- > > > *Zoltan Forray* > > > Backup & VMware Systems Administrator Enterprise Compute & Storage > > > Platforms VCU Infrastructure Services www.ucc.vcu.edu > > > zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and > > > other reputable organizations will never use email to request that > > > you reply with your password, social security number or confidential > > > personal information. For more details visit > > > http://phishing.vcu.edu/ <https://adminmicro2.questionpro.com> > > > > > > > > > -- > > *Zoltan Forray* > > Backup & VMware Systems Administrator > > Enterprise Compute & Storage Platforms VCU Infrastructure Services > > www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing > > victim - VCU and other reputable organizations will never use email to > > request that you reply with your password, social security number or > > confidential personal information. For more details visit > > http://phishing.vcu.edu/ <https://adminmicro2.questionpro.com> > > > > > -- > *Zoltan Forray* > Backup & VMware Systems Administrator > Enterprise Compute & Storage Platforms > VCU Infrastructure Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://phishing.vcu.edu/ <https://adminmicro2.questionpro.com> > -- *Zoltan Forray* Backup & VMware Systems Administrator Enterprise Compute & Storage Platforms VCU Infrastructure Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/ <https://adminmicro2.questionpro.com>
