I can't see any file-based backups working well in this case. Depending on the filesystem used & size of volumes, I would lean toward a block-based backup product (e.g. Acronis). Or multiple layers of replication & snapshots if feasible. ISP might be a nice hammer, but this is not a nail.
Steve Schaub Systems Engineer II, Backup/Recovery Blue Cross Blue Shield of Tennessee -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Roger Deschner Sent: Wednesday, March 22, 2017 7:24 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] How to backup Billions of files ?? How about periodic image backups, with daily journal-based backups to catch new/changed files? OS-based filesystems are sometimes abused as databases, and have been for many years. (e.g. Z/VM VMSES) When that happens, such a filesystem needs to be backed up more like a database than a filesystem. The answer there is ISP image backups. Roger Deschner University of Illinois at Chicago rog...@uic.edu ======I have not lost my mind -- it is backed up on tape somewhere.===== On Mon, 20 Mar 2017, Harris, Steven wrote: >Bo > >The problem with small files is that the TSM database entry may well be larger >than the file you are storing. If your files are less than about 3000 bytes >that will be the case. > >What is happening is that the file system is being used as a database. A >complex file path becomes the key and the file content is the data. >I realize this has probably been dumped on you without consultation, but a >database is probably a better fit. It could be something as simple as a >key/value store (maybe one per day) or as complex as a document DB like >Couchbase. > >A previous customer of mine did something similar. It was logs of ecommerce >transactions that averaged about 1500 bytes each and had to be kept for 7 >years. A million transactions a day and growing. They killed a TSM 5.5 >database in 2 years, and when I left were well on the way to killing a TSM 6.3 >database as well. Any requests to alter the application were met with active >hostility. > >Good Luck > >Steve >Steven Harris >TSM Admin/Consultant >Canberra Australia > > > > > >-----Original Message----- >From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >Of Rick Adamson >Sent: Tuesday, 21 March 2017 12:08 AM >To: ADSM-L@VM.MARIST.EDU >Subject: Re: [ADSM-L] How to backup Billions of files ?? > >Bo >I suggest you provide a few more details about the data and you backup >environment. >For example; what is this data, how frequently will it be accessed on average, >what is its total space requirements, what is the source stored on? >Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed >between data and target backup server? > >-Rick Adamson > Jacksonville,Fl. > > > >-----Original Message----- >From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf >Of Bo Nielsen >Sent: Monday, March 20, 2017 7:20 AM >To: ADSM-L@VM.MARIST.EDU >Subject: [ADSM-L] How to backup Billions of files ?? > >Hi TSM's > >I have earlier asked for help with archiving of 80 Billion very small files, >But now they want the files backed up. They expect an average change rate of 3 >percent/Month. > >Anyone with experience of such an exercise, and will share it with me?? > >Regards > >Bo > > >Bo Nielsen > > >IT Service > > > >Technical University of Denmark > >IT Service > >Frederiksborgvej 399 > >Building 109 > >DK - 4000 Roskilde > >Denmark > >Mobil +45 2337 0271 > >boa...@dtu.dk<mailto:boa...@dtu.dk> > >This message and any attachment is confidential and may be privileged or >otherwise protected from disclosure. You should immediately delete the message >if you are not the intended recipient. If you have received this email by >mistake please delete it from your system; you should not copy the message or >disclose its content to anyone. > >This electronic communication may contain general financial product advice but >should not be relied upon or construed as a recommendation of any financial >product. The information has been prepared without taking into account your >objectives, financial situation or needs. You should consider the Product >Disclosure Statement relating to the financial product and consult your >financial adviser before making a decision about whether to acquire, hold or >dispose of a financial product. > >For further details on the financial product please go to >http://www.bt.com.au > >Past performance is not a reliable indicator of future performance. > ------------------------------------------------------------------------------ Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm