Re: How to backup Billions of files ??

2017-03-23 Thread Schaub, Steve
I can't see any file-based backups working well in this case.  Depending on the 
filesystem used & size of volumes, I would lean toward a block-based backup 
product (e.g. Acronis).  Or multiple layers of replication & snapshots if 
feasible.  ISP might be a nice hammer, but this is not a nail.

Steve Schaub
Systems Engineer II, Backup/Recovery
Blue Cross Blue Shield of Tennessee

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Roger 
Deschner
Sent: Wednesday, March 22, 2017 7:24 PM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] How to backup Billions of files ??

How about periodic image backups, with daily journal-based backups to catch 
new/changed files?

OS-based filesystems are sometimes abused as databases, and have been for many 
years. (e.g. Z/VM VMSES) When that happens, such a filesystem needs to be 
backed up more like a database than a filesystem. The answer there is ISP image 
backups.

Roger Deschner  University of Illinois at Chicago rog...@uic.edu
==I have not lost my mind -- it is backed up on tape somewhere.=


On Mon, 20 Mar 2017, Harris, Steven wrote:

>Bo
>
>The problem with small files is that the TSM database entry may well be larger 
>than the file you are storing. If your files are less than about 3000 bytes 
>that will be the case.
>
>What is happening is that the file system is being used as a database.  A 
>complex file path becomes the key and the file content is the data.
>I realize this has probably been dumped on you without consultation, but a 
>database is probably a better fit.  It could be something as simple as a 
>key/value store (maybe one per day) or as complex as a document DB like 
>Couchbase.
>
>A previous customer of mine did something similar.  It was logs of ecommerce 
>transactions that averaged about 1500 bytes each and had to be kept for 7 
>years.  A million transactions a day and growing.  They killed a TSM 5.5 
>database in 2 years, and when I left were well on the way to killing a TSM 6.3 
>database as well.  Any requests to alter the application were met with active 
>hostility.
>
>Good Luck
>
>Steve
>Steven Harris
>TSM Admin/Consultant
>Canberra Australia
>
>
>
>
>
>-Original Message-
>From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
>Of Rick Adamson
>Sent: Tuesday, 21 March 2017 12:08 AM
>To: ADSM-L@VM.MARIST.EDU
>Subject: Re: [ADSM-L] How to backup Billions of files ??
>
>Bo
>I suggest you provide a few more details about the data and you backup 
>environment.
>For example; what is this data, how frequently will it be accessed on average, 
>what is its total space requirements, what is the source stored on?
>Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed 
>between data and target backup server?
>
>-Rick Adamson
> Jacksonville,Fl.
>
>
>
>-Original Message-
>From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
>Of Bo Nielsen
>Sent: Monday, March 20, 2017 7:20 AM
>To: ADSM-L@VM.MARIST.EDU
>Subject: [ADSM-L] How to backup Billions of files ??
>
>Hi TSM's
>
>I have earlier asked for help with archiving of 80 Billion very small files, 
>But now they want the files backed up. They expect an average change rate of 3 
>percent/Month.
>
>Anyone with experience of such an exercise, and will share it with me??
>
>Regards
>
>Bo
>
>
>Bo Nielsen
>
>
>IT Service
>
>
>
>Technical University of Denmark
>
>IT Service
>
>Frederiksborgvej 399
>
>Building 109
>
>DK - 4000 Roskilde
>
>Denmark
>
>Mobil +45 2337 0271
>
>boa...@dtu.dk
>
>This message and any attachment is confidential and may be privileged or 
>otherwise protected from disclosure. You should immediately delete the message 
>if you are not the intended recipient. If you have received this email by 
>mistake please delete it from your system; you should not copy the message or 
>disclose its content to anyone.
>
>This electronic communication may contain general financial product advice but 
>should not be relied upon or construed as a recommendation of any financial 
>product. The information has been prepared without taking into account your 
>objectives, financial situation or needs. You should consider the Product 
>Disclosure Statement relating to the financial product and consult your 
>financial adviser before making a decision about whether to acquire, hold or 
>dispose of a financial product.
>
>For further details on the financial product please go to 
>http://www.bt.com.au
>
>Past performance is not a reliable indicator of future performance.
>

--
Please see the following link for the BlueCross BlueShield of Tennessee E-mail 
disclaimer:  http://www.bcbst.com/email_disclaimer.shtm


Re: How to backup Billions of files ??

2017-03-22 Thread Roger Deschner
How about periodic image backups, with daily journal-based backups to
catch new/changed files?

OS-based filesystems are sometimes abused as databases, and have been
for many years. (e.g. Z/VM VMSES) When that happens, such a filesystem
needs to be backed up more like a database than a filesystem. The answer
there is ISP image backups.

Roger Deschner  University of Illinois at Chicago rog...@uic.edu
==I have not lost my mind -- it is backed up on tape somewhere.=


On Mon, 20 Mar 2017, Harris, Steven wrote:

>Bo
>
>The problem with small files is that the TSM database entry may well be larger 
>than the file you are storing. If your files are less than about 3000 bytes 
>that will be the case.
>
>What is happening is that the file system is being used as a database.  A 
>complex file path becomes the key and the file content is the data.
>I realize this has probably been dumped on you without consultation, but a 
>database is probably a better fit.  It could be something as simple as a 
>key/value store (maybe one per day) or as complex as a document DB like 
>Couchbase.
>
>A previous customer of mine did something similar.  It was logs of ecommerce 
>transactions that averaged about 1500 bytes each and had to be kept for 7 
>years.  A million transactions a day and growing.  They killed a TSM 5.5 
>database in 2 years, and when I left were well on the way to killing a TSM 6.3 
>database as well.  Any requests to alter the application were met with active 
>hostility.
>
>Good Luck
>
>Steve
>Steven Harris
>TSM Admin/Consultant
>Canberra Australia
>
>
>
>
>
>-Original Message-
>From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Rick 
>Adamson
>Sent: Tuesday, 21 March 2017 12:08 AM
>To: ADSM-L@VM.MARIST.EDU
>Subject: Re: [ADSM-L] How to backup Billions of files ??
>
>Bo
>I suggest you provide a few more details about the data and you backup 
>environment.
>For example; what is this data, how frequently will it be accessed on average, 
>what is its total space requirements, what is the source stored on?
>Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed 
>between data and target backup server?
>
>-Rick Adamson
> Jacksonville,Fl.
>
>
>
>-Original Message-
>From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Bo 
>Nielsen
>Sent: Monday, March 20, 2017 7:20 AM
>To: ADSM-L@VM.MARIST.EDU
>Subject: [ADSM-L] How to backup Billions of files ??
>
>Hi TSM's
>
>I have earlier asked for help with archiving of 80 Billion very small files, 
>But now they want the files backed up. They expect an average change rate of 3 
>percent/Month.
>
>Anyone with experience of such an exercise, and will share it with me??
>
>Regards
>
>Bo
>
>
>Bo Nielsen
>
>
>IT Service
>
>
>
>Technical University of Denmark
>
>IT Service
>
>Frederiksborgvej 399
>
>Building 109
>
>DK - 4000 Roskilde
>
>Denmark
>
>Mobil +45 2337 0271
>
>boa...@dtu.dk
>
>This message and any attachment is confidential and may be privileged or 
>otherwise protected from disclosure. You should immediately delete the message 
>if you are not the intended recipient. If you have received this email by 
>mistake please delete it from your system; you should not copy the message or 
>disclose its content to anyone.
>
>This electronic communication may contain general financial product advice but 
>should not be relied upon or construed as a recommendation of any financial 
>product. The information has been prepared without taking into account your 
>objectives, financial situation or needs. You should consider the Product 
>Disclosure Statement relating to the financial product and consult your 
>financial adviser before making a decision about whether to acquire, hold or 
>dispose of a financial product.
>
>For further details on the financial product please go to http://www.bt.com.au
>
>Past performance is not a reliable indicator of future performance.
>


Re: How to backup Billions of files ??

2017-03-20 Thread Harris, Steven
Bo

The problem with small files is that the TSM database entry may well be larger 
than the file you are storing. If your files are less than about 3000 bytes 
that will be the case.  

What is happening is that the file system is being used as a database.  A 
complex file path becomes the key and the file content is the data.
I realize this has probably been dumped on you without consultation, but a 
database is probably a better fit.  It could be something as simple as a 
key/value store (maybe one per day) or as complex as a document DB like 
Couchbase.

A previous customer of mine did something similar.  It was logs of ecommerce 
transactions that averaged about 1500 bytes each and had to be kept for 7 
years.  A million transactions a day and growing.  They killed a TSM 5.5 
database in 2 years, and when I left were well on the way to killing a TSM 6.3 
database as well.  Any requests to alter the application were met with active 
hostility.

Good Luck

Steve
Steven Harris
TSM Admin/Consultant 
Canberra Australia

   



-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Rick 
Adamson
Sent: Tuesday, 21 March 2017 12:08 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] How to backup Billions of files ??

Bo
I suggest you provide a few more details about the data and you backup 
environment.
For example; what is this data, how frequently will it be accessed on average, 
what is its total space requirements, what is the source stored on?
Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed 
between data and target backup server?

-Rick Adamson
 Jacksonville,Fl.
 


-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Bo 
Nielsen
Sent: Monday, March 20, 2017 7:20 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] How to backup Billions of files ??

Hi TSM's

I have earlier asked for help with archiving of 80 Billion very small files, 
But now they want the files backed up. They expect an average change rate of 3 
percent/Month.

Anyone with experience of such an exercise, and will share it with me??

Regards

Bo


Bo Nielsen


IT Service



Technical University of Denmark

IT Service

Frederiksborgvej 399

Building 109

DK - 4000 Roskilde

Denmark

Mobil +45 2337 0271

boa...@dtu.dk

This message and any attachment is confidential and may be privileged or 
otherwise protected from disclosure. You should immediately delete the message 
if you are not the intended recipient. If you have received this email by 
mistake please delete it from your system; you should not copy the message or 
disclose its content to anyone. 

This electronic communication may contain general financial product advice but 
should not be relied upon or construed as a recommendation of any financial 
product. The information has been prepared without taking into account your 
objectives, financial situation or needs. You should consider the Product 
Disclosure Statement relating to the financial product and consult your 
financial adviser before making a decision about whether to acquire, hold or 
dispose of a financial product. 

For further details on the financial product please go to http://www.bt.com.au 

Past performance is not a reliable indicator of future performance.


Re: How to backup Billions of files ??

2017-03-20 Thread Rick Adamson
Bo
I suggest you provide a few more details about the data and you backup 
environment.
For example; what is this data, how frequently will it be accessed on average, 
what is its total space requirements, what is the source stored on?
Type of backup storage; tape, disk, cloud? (specifics)
Bandwidth/network speed between data and target backup server?

-Rick Adamson
 Jacksonville,Fl.
 


-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Bo 
Nielsen
Sent: Monday, March 20, 2017 7:20 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] How to backup Billions of files ??

Hi TSM's

I have earlier asked for help with archiving of 80 Billion very small files, 
But now they want the files backed up. They expect an average change rate of 3 
percent/Month.

Anyone with experience of such an exercise, and will share it with me??

Regards

Bo


Bo Nielsen


IT Service



Technical University of Denmark

IT Service

Frederiksborgvej 399

Building 109

DK - 4000 Roskilde

Denmark

Mobil +45 2337 0271

boa...@dtu.dk