Re: [Veritas-bu] Architectural question (staging)

[Peters, Devon C]

Just gonna throw in my $.02...

We use DSSU's in our primary datacenter, and have it setup on 12 x 9TB 
filesystems using VXFS (on Linux).  Each filesystem is made from a 9+1 RAID 
group configured as a single LUN (1TB SATA drives).  We try to keep at least 7 
days worth of data on the filesystems.

We allow 40 concurrent jobs to each filesystem, and regularly see between 
200-300MB/s write throughput to a single filesystem - the disks rarely reach 
even 90% busy.  Using VXFS, we've never seen an issue with fragmentation - we 
get the same performance today as we did when they were implemented 2+ years 
ago, and we have never run a VXFS defrag.

For destaging, we use a single LTO3 or LTO4 tape drive for each filesystem.  If 
the data is mostly large backup images, then the filesystems keep up with the 
tape drives without issue - filesystem reads will be around 100-200MB/s and the 
disks don't exceed 60% utilization.

When the backup images are mostly small incrementals, the destaging performance 
is quite a bit lower - usually in the 50-80MB/s range, I've assumed it's due to 
some sort of tape drive repositioning/backhitch/etc for each image, but I don't 
really know.  Since the total amount of data to be destaged is usually not very 
large, it doesn't really matter to us, since it meets our windows for getting 
images destaged.

Worth noting, is that we also force disk I/O's writes to be 256k, using 
SIZE_DATA_BUFFERS_DISK.  I don't recall what the default size is, but I do know 
that in benchmarking the system before deployment we found that this size 
provided noticeably better throughput.  Also, when destaging I believe that the 
tape buffer size is forced to the same size that is configured for the disk (or 
maybe it's the other way around, where disk read I/O's get forced to the same 
size as the tape I/O's)...Anyhow, if you're seeing slow destaging performance, 
and the disks aren't 100% busy, then be sure you look at the buffer sizes 
configured, and the size of the actual I/O's being issued.

-devon

----------
Date: Thu, 6 May 2010 06:56:51 -0400
From: Travis Kelley 
Subject: Re: [Veritas-bu] Architectural question (staging)
To: "Martin, Jonathan" , Victor Engle
, veritas-bu@mailman.eng.auburn.edu
Message-ID:

Content-Type: text/plain; charset=ISO-8859-1

I agree with Martin here on them "working" in some cases.  I have and
EMC Clariion with 45 1TB SATA disks and I can tell you it screams.  I
routienly see over 600MB/S out of the array wjile destaging.  Sure,  I
have a larger and potentially "smarter" array than some but to say
they don't ever work is wrong.

One other point in regard to fragmentation.  If you are truely using
the disks as a cache and aren't in need of the additional restore
performance they provide then as soon as destaging is done you can
just expire all of the images on disk.  Once you have them on tape,
you may not "need" them on disk anymore anyway.  If you are able to do
this somewhat regularly (as often as you determine is necessay to keep
performance up), fragmentation becomes a non-issue.  In my case
fragmentation has never been an issue anyway, because of the extremely
wide striping.  But if its an issue,  as long as you can clean down
the disk every once in a while, the problem goes away.

Also images are interleved on the disk in the sense that they are not
contigious on the disk from a block perspective, but the image files
are not "multiplexed" as they would be on tape.  Every backup image
has at least one file all its own.

Hope that help.
Travis

On 5/5/10, Martin, Jonathan 
mailto:jmart...@intersil.com>> wrote:
> I'd hate not to disagree with someone as grumpy and disagreeable as Ed.
> Personally, I wouldn't take advice on this matter from someone who
> "worked with disk staging units for at least a year" and "gave up."
> (Also, I think Ed is a wet blanket.) I had this thing figured out 4
> years ago when we first implemented DSSUs in production. I may not be
> the biggest NBU shop on the planet, but I back up more than 50TB a week
> using this method exclusively, so I can tell you that it does work.
>
>
>
> As far "interleaving", there is most certainly interleaving at the file
> system level when you run multiple streams to a DSSU. How Ed can say
> there is no interleaving and then tell you to watch your disk
> fragmentation is beyond me. Fragmentation = disk interleaving as far as
> I am concerned. The point is that the files are non-contiguous.
>
>
>
> Here's my proof.
>
>
>
>
>
>
>
> This is a snippit of a utility called DiskView from SysInternals /
> Microsoft. The yellow bits are the actual 1K fragments of data on 

Re: [Veritas-bu] Architectural question (staging)

[Shekel Tal]

I think to make an informed decision you would need to look at the
bigger picture. Your original goal was to increase your backup
performance and shrink you backup window

Disk can be great but don't expect your backup times to increase just
because you are using it.

Is your disk being provided from the same array where your backup
clients host their data? In that case you could see a reduction in
performance.
Also you could have a negative effect on your production systems when
de-staging during the day

What kind of Networking do you have in place between your backup clients
and your media servers?
Using GbE with Jumbo frames can produce some fantastic results in
shrinking you backup window.

What kind of tape devices do you have and how many?
In most cases you will have equal tape throughput capability to that of
your disk or very likely even more.
Of course you don't want to multiplex too high but with efficient
networking you may not need to. - you should also make sure you perform
all the NetBackup tuning you can e.g. number/size data buffers and
network buffers on media servers and clients whether using disk or tape

I like to use disk for slow backup clients that would force a high level
multiplexing setting or cause shoe shinning across your tape devices.
Where you have servers (i.e. DB systems or systems with large files)
that have the potential to stream at a decent rate - send them direct to
tape.





-Original Message-
From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Travis
Kelley
Sent: 06 May 2010 11:57
To: Martin, Jonathan; Victor Engle; veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)

I agree with Martin here on them "working" in some cases.  I have and
EMC Clariion with 45 1TB SATA disks and I can tell you it screams.  I
routienly see over 600MB/S out of the array wjile destaging.  Sure,  I
have a larger and potentially "smarter" array than some but to say
they don't ever work is wrong.

One other point in regard to fragmentation.  If you are truely using
the disks as a cache and aren't in need of the additional restore
performance they provide then as soon as destaging is done you can
just expire all of the images on disk.  Once you have them on tape,
you may not "need" them on disk anymore anyway.  If you are able to do
this somewhat regularly (as often as you determine is necessay to keep
performance up), fragmentation becomes a non-issue.  In my case
fragmentation has never been an issue anyway, because of the extremely
wide striping.  But if its an issue,  as long as you can clean down
the disk every once in a while, the problem goes away.

Also images are interleved on the disk in the sense that they are not
contigious on the disk from a block perspective, but the image files
are not "multiplexed" as they would be on tape.  Every backup image
has at least one file all its own.

Hope that help.
Travis

On 5/5/10, Martin, Jonathan  wrote:
> I'd hate not to disagree with someone as grumpy and disagreeable as
Ed.
> Personally, I wouldn't take advice on this matter from someone who
> "worked with disk staging units for at least a year" and "gave up."
> (Also, I think Ed is a wet blanket.) I had this thing figured out 4
> years ago when we first implemented DSSUs in production. I may not be
> the biggest NBU shop on the planet, but I back up more than 50TB a
week
> using this method exclusively, so I can tell you that it does work.
>
>
>
> As far "interleaving", there is most certainly interleaving at the
file
> system level when you run multiple streams to a DSSU. How Ed can say
> there is no interleaving and then tell you to watch your disk
> fragmentation is beyond me. Fragmentation = disk interleaving as far
as
> I am concerned. The point is that the files are non-contiguous.
>
>
>
> Here's my proof.
>
>
>
>
>
>
>
> This is a snippit of a utility called DiskView from SysInternals /
> Microsoft. The yellow bits are the actual 1K fragments of data on disk
> for that image file above. The little red dots indicate the beginning
> and end of file fragments. There are 64 little yellow dots between the
> red dots indicating my 64K clusters.
>
>
>
>
>
>
>
> Here's that same section of disk, different image file. These two
> streams ran simultaneously last night (along with 6 others) and I can
> guarantee you that the top image wrote faster, and will destage to
tape
> faster than the image below.
>
>
>
> Why? Imagine you are bpdupicate.exe requesting the first file back to
> write to tape. Compared to the 2nd image, you are going to get a lot
> more reading done and a lot less seeking as your head(s) cross the
disk
> to picku

Re: [Veritas-bu] Architectural question (staging)

[Travis Kelley]

tart with a single stream to that 6TB DSSU and see what
> you get both writing to the DSSU and destaging to tape. Whatever
> performance you get out of that configuration is your best case
> scenario. Multiple streams or creating multiple partitions will only
> drag your numbers down. The crux of the issue (at least for me) is
> balancing the number of streams I need to run to get my backups to DSSU
> within my windows, versus the destaging speed I need to get that data
> off to tape on time.
>
>
>
> Good luck,
>
>
>
> -Jonathan
>
>
>
>
>
> From: veritas-bu-boun...@mailman.eng.auburn.edu
> [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts
> Sent: Wednesday, May 05, 2010 4:06 PM
> To: Victor Engle
> Cc: veritas-bu@mailman.eng.auburn.edu
> Subject: Re: [Veritas-bu] Architectural question (staging)
>
>
>
> On Wed, May 5, 2010 at 2:57 PM, Victor Engle 
> wrote:
>
> So my question is how best to configure the DSSUs with the goal of
> optimized de-staging. I will have 6TB to configure as desired on the
> backup server. If I understand correctly, the more concurrent streams
> allowed to the DSSUs, the slower the de-staging because of interleaved
> backup streams.
>
>
> The DSSU consists of a set of files with each file being a backup image
> and you define the maximum size of each file within an image.  There is
> no "interleaving".  When you destage, one image at a time goes to tape.
>
> Watch your fragment sizes and watch your disk file system
> fragmentation...
>
>.../Ed
>
>
>
> Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
> ewi...@ewilts.org
>
>  Linkedin <http://www.linkedin.com/in/ewilts>
>
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[przemolicc]

Are you sure about that:
> ...  not be able to stream data to a disk array
If data from NBU policy goes as one stream and you direct them to particular 
arrays
disks (each policy to different disks) it is sequential (not random !) IO 
pattern. In such case even SATA disks
are able to cope with it and deliver high IOPS.

Regards
Przemyslaw Bak (przemol)
--
http://przemol.blogspot.com/


On Wed, May 05, 2010 at 03:43:20PM -0500, Bryan Bahnmiller wrote:
> Agreed.
> 
>   Also, be aware that you will typically not be able to stream data to a 
> disk array as fast as you can to tape drives. (Assuming LTO3 or 4 type 
> performance.) Unless you have a pretty beefy disk array with your RAID 
> configured for streaming. The nice part is that since it is disk, small 
> backups and slow backups won't have "shoeshine" problems like you would 
> with tape.
> 
>I like to set a high water mark on the disk to keep it at 85% or lower. 
> Generally, 85% full is the point where disk performance starts getting hit 
> hard. Fragmentation will also start hitting the performance hard at that 
> point too.
> 
>I've yet to see de-staging perform well no matter what the disk array 
> used for the DSSU.
> 
> Bryan
> 
> 
> 
> 
> Ed Wilts  
> Sent by: veritas-bu-boun...@mailman.eng.auburn.edu
> 05/05/2010 03:05 PM
> 
> To
> Victor Engle 
> cc
> veritas-bu@mailman.eng.auburn.edu
> Subject
> Re: [Veritas-bu] Architectural question (staging)
> 
> 
> 
> 
> 
> 
> On Wed, May 5, 2010 at 2:57 PM, Victor Engle  
> wrote:
> So my question is how best to configure the DSSUs with the goal of
> optimized de-staging. I will have 6TB to configure as desired on the
> backup server. If I understand correctly, the more concurrent streams
> allowed to the DSSUs, the slower the de-staging because of interleaved
> backup streams. 
> 
> The DSSU consists of a set of files with each file being a backup image 
> and you define the maximum size of each file within an image.  There is no 
> "interleaving".  When you destage, one image at a time goes to tape.
> 
> Watch your fragment sizes and watch your disk file system 
> fragmentation...  
> 
>.../Ed
> 
> 
> Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE 
> ewi...@ewilts.org
> Linkedin___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> 
> 
> _
> 
> DTCC DISCLAIMER: This email and any files transmitted with it are
> confidential and intended solely for the use of the individual or
> entity to whom they are addressed. If you have received this email
> in error, please notify us immediately and delete the email and any
> attachments from your system. The recipient should check this email
> and any attachments for the presence of viruses.  The company
> accepts no liability for any damage caused by any virus transmitted
> by this email.
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu































--
Szukasz pracy? Zobacz ciekawe oferty w Twoim miescie
Sprawdz >>> http://linkint.pl/f26b2

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Ed Wilts]

On Wed, May 5, 2010 at 4:18 PM, Martin, Jonathan wrote:

>  I wouldn't take advice on this matter from someone who "worked with disk
> staging units for at least a year" and "gave up."
>
We worked extensively with Symantec on this issue.  We were in regular
contact with the customer focus team.  They were onsite.  We had engineering
onsite.  We went to their engineering offices a few miles down the road from
us.  We met with product management.  Symantec was unable to solve the
performance issues after well over a year of trying.

Obviously your mileage will vary.  I'm glad it works for some people -
Symantec was unable to make it work for us.

   .../Ed
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Martin, Jonathan]

I'd hate not to disagree with someone as grumpy and disagreeable as Ed.
Personally, I wouldn't take advice on this matter from someone who
"worked with disk staging units for at least a year" and "gave up."
(Also, I think Ed is a wet blanket.) I had this thing figured out 4
years ago when we first implemented DSSUs in production. I may not be
the biggest NBU shop on the planet, but I back up more than 50TB a week
using this method exclusively, so I can tell you that it does work.

 

As far "interleaving", there is most certainly interleaving at the file
system level when you run multiple streams to a DSSU. How Ed can say
there is no interleaving and then tell you to watch your disk
fragmentation is beyond me. Fragmentation = disk interleaving as far as
I am concerned. The point is that the files are non-contiguous.

 

Here's my proof.

 

 

 

This is a snippit of a utility called DiskView from SysInternals /
Microsoft. The yellow bits are the actual 1K fragments of data on disk
for that image file above. The little red dots indicate the beginning
and end of file fragments. There are 64 little yellow dots between the
red dots indicating my 64K clusters.

 

 

 

Here's that same section of disk, different image file. These two
streams ran simultaneously last night (along with 6 others) and I can
guarantee you that the top image wrote faster, and will destage to tape
faster than the image below. 

 

Why? Imagine you are bpdupicate.exe requesting the first file back to
write to tape. Compared to the 2nd image, you are going to get a lot
more reading done and a lot less seeking as your head(s) cross the disk
to pickup fragments. Or, so goes my theory.  There is a utility
available from Dell that will show me the amount of time spent reading /
writing versus seeking per disk but I didn't have the time to acquire it
and test.

 

Now, I know there are variables here. As I stated before, one of the big
improvements to my speed was using a 64K cluster size. Last time I
checked this wasn't available in Unix/Linux. Then again, ext2/3 file
systems also like to leave "space" between their writes to account for
file growth, which may help (but I doubt it.) I intended to test this
several years back, but my management put the kibosh on Linux media
servers. The raid controller, simultaneous read/write, spindle count,
and disk type also add a lot of variability.

 

I haven't tested any of this on a SAN volume, only on direct attached. I
don't think there is much to be gained by taking a 6TB lun and
partitioning it at the OS or breaking it into multiple luns at the SAN.
After partitioning, the entire DSSU is still on the same raid group /
set, which ultimately controls your performance. If you could take your
6TB lun and break it into 3 x 2TB raid groups / luns then I think that
would help. I've actually considered breaking my 14 disk RAID5s into 14
single disks for performance testing (single stream each), but that's an
entirely different management nightmare (14 DSSUs per media server
etc...) A single SATA disk can drive LTO3, assuming the data is all
nicely lined up.  The minute that head has to go seeking, you are in a
world of hurt. 

 

Again, I would start with a single stream to that 6TB DSSU and see what
you get both writing to the DSSU and destaging to tape. Whatever
performance you get out of that configuration is your best case
scenario. Multiple streams or creating multiple partitions will only
drag your numbers down. The crux of the issue (at least for me) is
balancing the number of streams I need to run to get my backups to DSSU
within my windows, versus the destaging speed I need to get that data
off to tape on time.

 

Good luck,

 

-Jonathan

 

 

From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts
Sent: Wednesday, May 05, 2010 4:06 PM
To: Victor Engle
Cc: veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)

 

On Wed, May 5, 2010 at 2:57 PM, Victor Engle 
wrote:

So my question is how best to configure the DSSUs with the goal of
optimized de-staging. I will have 6TB to configure as desired on the
backup server. If I understand correctly, the more concurrent streams
allowed to the DSSUs, the slower the de-staging because of interleaved
backup streams. 


The DSSU consists of a set of files with each file being a backup image
and you define the maximum size of each file within an image.  There is
no "interleaving".  When you destage, one image at a time goes to tape.

Watch your fragment sizes and watch your disk file system
fragmentation...  

   .../Ed



Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE 
ewi...@ewilts.org

 Linkedin <http://www.linkedin.com/in/ewilts> 

<><><>___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Bryan Bahnmiller]

Agreed.

  Also, be aware that you will typically not be able to stream data to a 
disk array as fast as you can to tape drives. (Assuming LTO3 or 4 type 
performance.) Unless you have a pretty beefy disk array with your RAID 
configured for streaming. The nice part is that since it is disk, small 
backups and slow backups won't have "shoeshine" problems like you would 
with tape.

   I like to set a high water mark on the disk to keep it at 85% or lower. 
Generally, 85% full is the point where disk performance starts getting hit 
hard. Fragmentation will also start hitting the performance hard at that 
point too.

   I've yet to see de-staging perform well no matter what the disk array 
used for the DSSU.

Bryan




Ed Wilts  
Sent by: veritas-bu-boun...@mailman.eng.auburn.edu
05/05/2010 03:05 PM

To
Victor Engle 
cc
veritas-bu@mailman.eng.auburn.edu
Subject
Re: [Veritas-bu] Architectural question (staging)






On Wed, May 5, 2010 at 2:57 PM, Victor Engle  
wrote:
So my question is how best to configure the DSSUs with the goal of
optimized de-staging. I will have 6TB to configure as desired on the
backup server. If I understand correctly, the more concurrent streams
allowed to the DSSUs, the slower the de-staging because of interleaved
backup streams. 

The DSSU consists of a set of files with each file being a backup image 
and you define the maximum size of each file within an image.  There is no 
"interleaving".  When you destage, one image at a time goes to tape.

Watch your fragment sizes and watch your disk file system 
fragmentation...  

   .../Ed


Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE 
ewi...@ewilts.org
Linkedin___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu



_

DTCC DISCLAIMER: This email and any files transmitted with it are
confidential and intended solely for the use of the individual or
entity to whom they are addressed. If you have received this email
in error, please notify us immediately and delete the email and any
attachments from your system. The recipient should check this email
and any attachments for the presence of viruses.  The company
accepts no liability for any damage caused by any virus transmitted
by this email.___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Victor Engle]

Thanks Ed but I'm confused. If I say that Netbackup can send 15 streams to a
DSSU and then I have 15 clients sending streams all at once to that DSSU,
wouldn't the data be more fragmented than if I sent only 1 stream? Could you
elaborate a bit on how that works?

Thanks,
Vic


On Wed, May 5, 2010 at 4:05 PM, Ed Wilts  wrote:

> On Wed, May 5, 2010 at 2:57 PM, Victor Engle wrote:
>
>> So my question is how best to configure the DSSUs with the goal of
>> optimized de-staging. I will have 6TB to configure as desired on the
>> backup server. If I understand correctly, the more concurrent streams
>> allowed to the DSSUs, the slower the de-staging because of interleaved
>> backup streams.
>
>
> The DSSU consists of a set of files with each file being a backup image and
> you define the maximum size of each file within an image.  There is no
> "interleaving".  When you destage, one image at a time goes to tape.
>
> Watch your fragment sizes and watch your disk file system fragmentation...
>
>
>.../Ed
>
>
> Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
> ewi...@ewilts.org
> Linkedin 
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Ed Wilts]

On Wed, May 5, 2010 at 2:57 PM, Victor Engle  wrote:

> So my question is how best to configure the DSSUs with the goal of
> optimized de-staging. I will have 6TB to configure as desired on the
> backup server. If I understand correctly, the more concurrent streams
> allowed to the DSSUs, the slower the de-staging because of interleaved
> backup streams.


The DSSU consists of a set of files with each file being a backup image and
you define the maximum size of each file within an image.  There is no
"interleaving".  When you destage, one image at a time goes to tape.

Watch your fragment sizes and watch your disk file system fragmentation...

   .../Ed


Ed Wilts, RHCE, BCFP, BCSD, SCSP, SCSE
ewi...@ewilts.org
Linkedin 
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Victor Engle]

Just a follow-up question...

In my environment, I'm going to be given access to some SAN storage
for use as staging areas. I have a tape library with 4 drives and I'm
backing up 13TB per week. It is tough getting this done within the
backup windows with the 4 tape drives so I want the DSSU space to ease
the load a bit. It seems to me that having the DSSU's should
essentially extend my backup window since all data written to DSSUs
can be de-staged outside of the backup window.

So my question is how best to configure the DSSUs with the goal of
optimized de-staging. I will have 6TB to configure as desired on the
backup server. If I understand correctly, the more concurrent streams
allowed to the DSSUs, the slower the de-staging because of interleaved
backup streams. So, for the 6TB of SAN storage it would seem to be
more efficient to create several smaller volumes, each with its own
filesystem and mountpoint rather than one big 6TB volume with 1
filesystem. Am I correct to assume that the interleaving blocks from
multiple streams is occurring at the filesystem level? Since the SAN
provisioned LUN is virtual and made up of several disks in a stripe
configuration I expect disk reads to be reasonably quick.

Thanks,
Vic


On Mon, Apr 26, 2010 at 4:34 PM, Martin, Jonathan  wrote:
>
>
> I started with a calculation like that, but I also incorporate destaging
> speed. I've never used anything higher than 16 concurrent writes, and that
> is for incremental backups. With incremental backups, I don’t care as much
> because even if the destaging speed is poor, the amount of data is so small
> that it will still be written to tape quickly.
>
>
>
> However for full backups, I’ve found that the more streams you have, the
> slower your destaging will be.  I created the graphic below to help explain
> it to management.  The issue isn’t fragmentation as much as it is sequential
> blocks of data. If you were to write one sequential stream to a raid group
> then reading it back would be a sequential operation.  And in my case, SATA
> disks read sequential data very quickly. But as soon as you multistream, the
> data “fragments” or becomes non-sequential (really, the file sizes are so
> large every file is fragmented).
>
>
>
> Image removed due to size limitation.
>
>
>
> So to find your “sweet” spot you will have to test.  I started by building
> my environment and then testing with a single stream, from client to disk,
> then disk to tape.  This was my best case scenario, and I was able to drive
> LTO3 with it.
>
>
>
> My configuration: Dell MD1000 w/ 12 500GB DATA disks in a Raid 5. PERC 5/e
> card with Adaptive Read Ahead and Write Back enabled. GPT Partition table w/
> 64KB block sizes.
>
>
>
> The 64KB block size and adaptive read ahead settings were key to my
> performance. I was going to test the same hardware with a linux ext2 file
> system and 4KB block sizes for comparison, but I never got the chance.
>
>
>
> Once you have a baseline you can run more streams. You also need to consider
> job length.  For example, I backup an NFS server with 16 streams to a DSSU
> that allows all 16 streams because 12 of those streams take less than an
> hour each.  I’ve also tested with 1TB SATA Disks and 300GB 15K SAS Disks.  I
> still rely on the 500GB SATA as my work horse because I can drive LTO3 if I
> need to. Most of my DSSUs are 8 streams and push 70MB/sec to tape.
>
>
>
> -Jonathan
>
>
>
>
>
>
>
> -----Original Message-
> From: Victor Engle [mailto:victor.en...@gmail.com]
> Sent: Monday, April 26, 2010 3:39 PM
> To: Martin, Jonathan
> Cc: veritas-bu@mailman.eng.auburn.edu
> Subject: Re: [Veritas-bu] Architectural question (staging)
>
>
>
> Thanks Martin. Good info. What criteria do you use to determine the
>
> number of concurrent jobs for a DSSU? Is it reasonable to determine
>
> the concurrent DSSU sessions based on the speed of the clients? For
>
> example, If I have a disk to which I can write 100MB/s and clients
>
> that can write 5MB/s then I would use 20 concurrent sessions?
>
>
>
> Thanks all for your responses!
>
>
>
> Vic
>
>
>
>
>
> On Mon, Apr 26, 2010 at 10:14 AM, Martin, Jonathan
>
>  wrote:
>
>> We use Disk Stage Storage Units (DSSU) almost exclusively for our backups.
>
>> As someone has already mentioned, you can stream many slow clients to a
>> DSSU
>
>> without impacting your speed significantly. To do this with tape you would
>
>> have to use multiplexing, which is a real performance killer come restore
>
>> time.  The DSSUs essentially allow you to “multiplex” to disk, then single
>
>> stream to tape. Regarding Ed’s speed issue below, I’ve

Re: [Veritas-bu] Architectural question (staging)

[Nicholas]

What kind of data are you sending to the staging area? Destaging doesn't 
use multiplex so as small are your images, as slow will be your 
destaging process.
I had many problems with this, mainly when dealing with Oracle's archive 
logs. Even when using 20 drives just for destaging, it was never enough.
We work with some T1 tape drives that can write something like 
120MB/s for backups, but when destaging this archive log area, it was 
running at 400KB/s

IMHO, the best thing is to put medium to large backups to staging, 
instead of putting small backups, and configuring the image size to a 
higher value.
This will increase a lot your throughput
regards,
Nick
Em 26/4/2010 11:55, judy_hinchcli...@administaff.com escreveu:
> I am in agreement with Ed,
> We could backup to disk ok, but getting it of disk to tape took longer
> and we could not get it finished before the next nights backups, and
> that was when we had SDLT tape drives.
> I now have LTO4 drives and backup straight to tape and I still cannot
> keep all the tape drives busy.  So just did not work for us.
> There are times it is needed.
> Some backups, like if you want to do exchange where you can restore just
> one email, needed the backup to be on disk, you could not do that from
> tape.  So you have to look at what kind of backups and restores you do
> and if you need the backup on disk to so the restore.
>
> -Original Message-
> From: veritas-bu-boun...@mailman.eng.auburn.edu
> [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Victor
> Engle
> Sent: Sunday, April 25, 2010 2:15 PM
> To: veritas-bu@mailman.eng.auburn.edu
> Subject: [Veritas-bu] Architectural question (staging)
>
> Hello List,
>
> Just wanted to get some opinions about whether disk staging units are
> worthwhile. My backup server has two BasicDisk staging units with the
> storage units configured such that the data goes to disk and is then
> moved to tape. I have a tape library with four LTO-3 drives connected
> via FC. So what I'm wondering is, since the LTO drives are reasonably
> fast, and since I'm writing the data ultimately to tape anyway, would
> it be better to just write directly to tape. The disk is just old
> fashioned spinning disk with no de-duplication so there are
> operational costs for the disks. All tape and disk storage units are
> local to the backup server. I'm thinking it would be better to add LTO
> drives and eliminate the disk for now and maybe later add a
> de-duplicating disk unit.
>
> Under what circumstances does it make sense to stage data on disk. I
> would appreciate hearing what your thoughts and experiences are with
> regard to disk staging.
>
> Thanks,
> Vic
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Victor Engle]

Excellent information, thanks. In theory, from what I am reading in this
thread, it is desirable to stage backups for clients that can't stream the
tape media.

What is the minimum speed for streaming an LTO3 drive?

Thanks,
Vic


On Tue, Apr 27, 2010 at 6:59 AM, WEAVER, Simon (external) <
simon.wea...@astrium.eads.net> wrote:

>  Have to agree with Ed here. I spent many, many months mucking around with
> SAN Media Servers, and configuring Disk Staging Units, that then write to
> tape!
>
> I came across 2 major problems:
>
> 1) Performance was slower than LTO3 drives (FC attached, with SSO)
> 2) Writing from disk to tape (using the DSSU config) was quick, but and
> this is an annoying BUT, it would randomly fail to write to tape!!
> 3) The time difference between writing to Disk and Tape was 2 mins. This
> difference was mainly where the Robot was picking the tape and mounting it !
>
> I have stopped going down this route, but although I have my DSSU's ready
> and a good 4TB storage available, if the drives went pearshaped, I could
> still do the backups to disk, short term !
>
> Simon
>
>  --
> *From:* veritas-bu-boun...@mailman.eng.auburn.edu [mailto:
> veritas-bu-boun...@mailman.eng.auburn.edu] *On Behalf Of *Ed Wilts
> *Sent:* Sunday, April 25, 2010 8:45 PM
> *To:* Victor Engle
> *Cc:* veritas-bu@mailman.eng.auburn.edu
> *Subject:* Re: [Veritas-bu] Architectural question (staging)
>
>  On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle wrote:
>
>> Just wanted to get some opinions about whether disk staging units are
>> worthwhile. My backup server has two BasicDisk staging units with the
>> storage units configured such that the data goes to disk and is then
>> moved to tape. I have a tape library with four LTO-3 drives connected
>> via FC. So what I'm wondering is, since the LTO drives are reasonably
>> fast, and since I'm writing the data ultimately to tape anyway, would
>> it be better to just write directly to tape. The disk is just old
>> fashioned spinning disk with no de-duplication so there are
>> operational costs for the disks. All tape and disk storage units are
>> local to the backup server. I'm thinking it would be better to add LTO
>> drives and eliminate the disk for now and maybe later add a
>> de-duplicating disk unit.
>>
>
> We worked with disk staging units for at least a year before we mostly gave
> up.  The biggest challenge we ran into was that destaging was too slow. Even
> though we proved to Symantec that we could read from those disk drives at
> over 100MB/sec, we could never destage even half that fast.  We had an open
> case with Symantec for a VERY long time before we agreed that it wasn't
> going to get fixed.
>
> Under what circumstances does it make sense to stage data on disk. I
>> would appreciate hearing what your thoughts and experiences are with
>> regard to disk staging.
>>
>
> There are times when DSSUs make sense.  1.  If you don't have a tape drive
> free but want to do a backup anyway - we still use DSSUs for things like
> small backups of Oracle archive logs. 2.  If you need to throttle your
> backups, especially across things like a bunch of virtual servers on the
> same physical server.  NBU only allows you to set the maximum jobs per
> client name, not per client.  DSSUs make an acceptable choke point for
> clusters.  3.  If you have small backups, but don't have a lot of them at
> once, multiplexing may not buy you enough performance boosts.  Use DSSUs to
> write those little jobs to disk and then destage them at once.
>
> If you currently multiplex, realize that your restores are going to be
> slower than if you don't multiplex.  All tapes created from a DSSU destage
> are non-multiplexed so your restores can go faster.
>
> DSSUs also give you a staging area for restores.  If your tapes go offsite,
> you may still be able to do a restore from the staging unit the next day (or
> longer) depending on how big your stagig units are.  NBU is smart enough to
> realize that if the same data is on both disk and tape and you kick off a
> restore, the restore will automatically come from disk.
>
> In general, I'd say that there is a place for DSSUs but it's not the great
> benefit we thought it was going to be.
>
>.../Ed
>
> This email (including any attachments) may contain confidential
> and/or privileged information or information otherwise protected
> from disclosure. If you are not the intended recipient, please
> notify the sender immediately, do not copy this message or any
> attachments and do not use it for any purpose or disclo

Re: [Veritas-bu] Architectural question (staging)

[WEAVER, Simon (external)]

Have to agree with Ed here. I spent many, many months mucking around
with SAN Media Servers, and configuring Disk Staging Units, that then
write to tape!
 
I came across 2 major problems:
 
1) Performance was slower than LTO3 drives (FC attached, with SSO)
2) Writing from disk to tape (using the DSSU config) was quick, but and
this is an annoying BUT, it would randomly fail to write to tape!! 
3) The time difference between writing to Disk and Tape was 2 mins. This
difference was mainly where the Robot was picking the tape and mounting
it !
 
I have stopped going down this route, but although I have my DSSU's
ready and a good 4TB storage available, if the drives went pearshaped, I
could still do the backups to disk, short term !
 
Simon



From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts
Sent: Sunday, April 25, 2010 8:45 PM
To: Victor Engle
Cc: veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)


On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle 
wrote:


Just wanted to get some opinions about whether disk staging
units are
worthwhile. My backup server has two BasicDisk staging units
with the
storage units configured such that the data goes to disk and is
then
moved to tape. I have a tape library with four LTO-3 drives
connected
via FC. So what I'm wondering is, since the LTO drives are
reasonably
fast, and since I'm writing the data ultimately to tape anyway,
would
it be better to just write directly to tape. The disk is just
old
fashioned spinning disk with no de-duplication so there are
operational costs for the disks. All tape and disk storage units
are
local to the backup server. I'm thinking it would be better to
add LTO
drives and eliminate the disk for now and maybe later add a
de-duplicating disk unit.



We worked with disk staging units for at least a year before we mostly
gave up.  The biggest challenge we ran into was that destaging was too
slow. Even though we proved to Symantec that we could read from those
disk drives at over 100MB/sec, we could never destage even half that
fast.  We had an open case with Symantec for a VERY long time before we
agreed that it wasn't going to get fixed.



Under what circumstances does it make sense to stage data on
disk. I
would appreciate hearing what your thoughts and experiences are
with
regard to disk staging.



There are times when DSSUs make sense.  1.  If you don't have a tape
drive free but want to do a backup anyway - we still use DSSUs for
things like small backups of Oracle archive logs. 2.  If you need to
throttle your backups, especially across things like a bunch of virtual
servers on the same physical server.  NBU only allows you to set the
maximum jobs per client name, not per client.  DSSUs make an acceptable
choke point for clusters.  3.  If you have small backups, but don't have
a lot of them at once, multiplexing may not buy you enough performance
boosts.  Use DSSUs to write those little jobs to disk and then destage
them at once.

If you currently multiplex, realize that your restores are going to be
slower than if you don't multiplex.  All tapes created from a DSSU
destage are non-multiplexed so your restores can go faster.

DSSUs also give you a staging area for restores.  If your tapes go
offsite, you may still be able to do a restore from the staging unit the
next day (or longer) depending on how big your stagig units are.  NBU is
smart enough to realize that if the same data is on both disk and tape
and you kick off a restore, the restore will automatically come from
disk.

In general, I'd say that there is a place for DSSUs but it's not the
great benefit we thought it was going to be.

   .../Ed


This email (including any attachments) may contain confidential
and/or privileged information or information otherwise protected
from disclosure. If you are not the intended recipient, please
notify the sender immediately, do not copy this message or any
attachments and do not use it for any purpose or disclose its
content to any person, but delete this message and any attachments
from your system. Astrium disclaims any and all liability if this
email transmission was virus corrupted, altered or falsified.
-o-
Astrium Limited, Registered in England and Wales No. 2449259
Registered Office:
Gunnels Wood Road, Stevenage, Hertfordshire, SG1 2AS, England
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Martin, Jonathan]

 

I started with a calculation like that, but I also incorporate destaging speed. 
I've never used anything higher than 16 concurrent writes, and that is for 
incremental backups. With incremental backups, I don’t care as much because 
even if the destaging speed is poor, the amount of data is so small that it 
will still be written to tape quickly.

 

However for full backups, I’ve found that the more streams you have, the slower 
your destaging will be.  I created the graphic below to help explain it to 
management.  The issue isn’t fragmentation as much as it is sequential blocks 
of data. If you were to write one sequential stream to a raid group then 
reading it back would be a sequential operation.  And in my case, SATA disks 
read sequential data very quickly. But as soon as you multistream, the data 
“fragments” or becomes non-sequential (really, the file sizes are so large 
every file is fragmented).

 

Image removed due to size limitation.

 

So to find your “sweet” spot you will have to test.  I started by building my 
environment and then testing with a single stream, from client to disk, then 
disk to tape.  This was my best case scenario, and I was able to drive LTO3 
with it.

 

My configuration: Dell MD1000 w/ 12 500GB DATA disks in a Raid 5. PERC 5/e card 
with Adaptive Read Ahead and Write Back enabled. GPT Partition table w/ 64KB 
block sizes.

 

The 64KB block size and adaptive read ahead settings were key to my 
performance. I was going to test the same hardware with a linux ext2 file 
system and 4KB block sizes for comparison, but I never got the chance.

 

Once you have a baseline you can run more streams. You also need to consider 
job length.  For example, I backup an NFS server with 16 streams to a DSSU that 
allows all 16 streams because 12 of those streams take less than an hour each.  
I’ve also tested with 1TB SATA Disks and 300GB 15K SAS Disks.  I still rely on 
the 500GB SATA as my work horse because I can drive LTO3 if I need to. Most of 
my DSSUs are 8 streams and push 70MB/sec to tape.

 

-Jonathan

 

 

 

-Original Message-
From: Victor Engle [mailto:victor.en...@gmail.com] 
Sent: Monday, April 26, 2010 3:39 PM
To: Martin, Jonathan
Cc: veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)

 

Thanks Martin. Good info. What criteria do you use to determine the

number of concurrent jobs for a DSSU? Is it reasonable to determine

the concurrent DSSU sessions based on the speed of the clients? For

example, If I have a disk to which I can write 100MB/s and clients

that can write 5MB/s then I would use 20 concurrent sessions?

 

Thanks all for your responses!

 

Vic

 

 

On Mon, Apr 26, 2010 at 10:14 AM, Martin, Jonathan

 wrote:

> We use Disk Stage Storage Units (DSSU) almost exclusively for our backups.

> As someone has already mentioned, you can stream many slow clients to a DSSU

> without impacting your speed significantly. To do this with tape you would

> have to use multiplexing, which is a real performance killer come restore

> time.  The DSSUs essentially allow you to “multiplex” to disk, then single

> stream to tape. Regarding Ed’s speed issue below, I’ve got data here that

> directly correlates number of concurrent streams written to a DSSU with the

> performance to tape.  You’ll need to balance this when setting this on the

> DSSU, but we get away with 8-12 streams on 14x500GB SATA Disks in a Raid-5

> and still drive LTO3.

> 

> 

> 

> Here we were also able to purchase a much smaller tape library.  I replaced

> a library with 8 x SDLT220 drives with a library with 2 x LTO3 drives (and

> disk storage.)  The disk storage is far more reliable than tapes and

> libraries. As Ed noted below you also benefit from having your backups on

> disk which makes restores very snappy.

> 

> 

> 

> -Jonathan

> 

> 

> 

> From: veritas-bu-boun...@mailman.eng.auburn.edu

> [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts

> Sent: Sunday, April 25, 2010 3:45 PM

> To: Victor Engle

> Cc: veritas-bu@mailman.eng.auburn.edu

> Subject: Re: [Veritas-bu] Architectural question (staging)

> 

> 

> 

> On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle 

> wrote:

> 

> Just wanted to get some opinions about whether disk staging units are

> worthwhile. My backup server has two BasicDisk staging units with the

> storage units configured such that the data goes to disk and is then

> moved to tape. I have a tape library with four LTO-3 drives connected

> via FC. So what I'm wondering is, since the LTO drives are reasonably

> fast, and since I'm writing the data ultimately to tape anyway, would

> it be better to just write directly to tape. The disk is just old

> fashioned spinning disk with no de-duplication so there are

> ope

Re: [Veritas-bu] Architectural question (staging)

[Victor Engle]

Thanks Martin. Good info. What criteria do you use to determine the
number of concurrent jobs for a DSSU? Is it reasonable to determine
the concurrent DSSU sessions based on the speed of the clients? For
example, If I have a disk to which I can write 100MB/s and clients
that can write 5MB/s then I would use 20 concurrent sessions?

Thanks all for your responses!

Vic


On Mon, Apr 26, 2010 at 10:14 AM, Martin, Jonathan
 wrote:
> We use Disk Stage Storage Units (DSSU) almost exclusively for our backups.
> As someone has already mentioned, you can stream many slow clients to a DSSU
> without impacting your speed significantly. To do this with tape you would
> have to use multiplexing, which is a real performance killer come restore
> time.  The DSSUs essentially allow you to “multiplex” to disk, then single
> stream to tape. Regarding Ed’s speed issue below, I’ve got data here that
> directly correlates number of concurrent streams written to a DSSU with the
> performance to tape.  You’ll need to balance this when setting this on the
> DSSU, but we get away with 8-12 streams on 14x500GB SATA Disks in a Raid-5
> and still drive LTO3.
>
>
>
> Here we were also able to purchase a much smaller tape library.  I replaced
> a library with 8 x SDLT220 drives with a library with 2 x LTO3 drives (and
> disk storage.)  The disk storage is far more reliable than tapes and
> libraries. As Ed noted below you also benefit from having your backups on
> disk which makes restores very snappy.
>
>
>
> -Jonathan
>
>
>
> From: veritas-bu-boun...@mailman.eng.auburn.edu
> [mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts
> Sent: Sunday, April 25, 2010 3:45 PM
> To: Victor Engle
> Cc: veritas-bu@mailman.eng.auburn.edu
> Subject: Re: [Veritas-bu] Architectural question (staging)
>
>
>
> On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle 
> wrote:
>
> Just wanted to get some opinions about whether disk staging units are
> worthwhile. My backup server has two BasicDisk staging units with the
> storage units configured such that the data goes to disk and is then
> moved to tape. I have a tape library with four LTO-3 drives connected
> via FC. So what I'm wondering is, since the LTO drives are reasonably
> fast, and since I'm writing the data ultimately to tape anyway, would
> it be better to just write directly to tape. The disk is just old
> fashioned spinning disk with no de-duplication so there are
> operational costs for the disks. All tape and disk storage units are
> local to the backup server. I'm thinking it would be better to add LTO
> drives and eliminate the disk for now and maybe later add a
> de-duplicating disk unit.
>
> We worked with disk staging units for at least a year before we mostly gave
> up.  The biggest challenge we ran into was that destaging was too slow. Even
> though we proved to Symantec that we could read from those disk drives at
> over 100MB/sec, we could never destage even half that fast.  We had an open
> case with Symantec for a VERY long time before we agreed that it wasn't
> going to get fixed.
>
> Under what circumstances does it make sense to stage data on disk. I
> would appreciate hearing what your thoughts and experiences are with
> regard to disk staging.
>
> There are times when DSSUs make sense.  1.  If you don't have a tape drive
> free but want to do a backup anyway - we still use DSSUs for things like
> small backups of Oracle archive logs. 2.  If you need to throttle your
> backups, especially across things like a bunch of virtual servers on the
> same physical server.  NBU only allows you to set the maximum jobs per
> client name, not per client.  DSSUs make an acceptable choke point for
> clusters.  3.  If you have small backups, but don't have a lot of them at
> once, multiplexing may not buy you enough performance boosts.  Use DSSUs to
> write those little jobs to disk and then destage them at once.
>
> If you currently multiplex, realize that your restores are going to be
> slower than if you don't multiplex.  All tapes created from a DSSU destage
> are non-multiplexed so your restores can go faster.
>
> DSSUs also give you a staging area for restores.  If your tapes go offsite,
> you may still be able to do a restore from the staging unit the next day (or
> longer) depending on how big your stagig units are.  NBU is smart enough to
> realize that if the same data is on both disk and tape and you kick off a
> restore, the restore will automatically come from disk.
>
> In general, I'd say that there is a place for DSSUs but it's not the great
> benefit we thought it was going to be.
>
>    .../Ed
>
> ___
> Veritas-bu maillist  -  veritas...@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Carl Elkins]

Lightner, Jeff wrote:
> What we do here is backup to dedupe devices (Data Domain) then vault to
> tape.  This has the benefit mentioned below that the restores for recent
> backups (we keep 30 days on DD) are fast because there are no tape
> operations involved.

We see a similar benefit for restore operations - most of ours come 
directly from the staging areas : there's the additional benefit of 
keeping the tape drives streaming more of the time, if your staging area 
is fast enough (don't know if you're seeing this), as compared to more 
variable throughput from networked clients.

>
> -Original Message-
> From: veritas-bu-boun...@mailman.eng.auburn.edu

[snip]

> --
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu


-- 
Carl Elkins, Team Leader, Storage & Backups,
Wellcome Trust Sanger Institute,
Wellcome Trust Genome Campus,
Hinxton, Cambridge, UK



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Lightner, Jeff]

What we do here is backup to dedupe devices (Data Domain) then vault to
tape.  This has the benefit mentioned below that the restores for recent
backups (we keep 30 days on DD) are fast because there are no tape
operations involved.

-Original Message-
From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of
judy_hinchcli...@administaff.com
Sent: Monday, April 26, 2010 10:56 AM
To: victor.en...@gmail.com; veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)

I am in agreement with Ed,
We could backup to disk ok, but getting it of disk to tape took longer
and we could not get it finished before the next nights backups, and
that was when we had SDLT tape drives.
I now have LTO4 drives and backup straight to tape and I still cannot
keep all the tape drives busy.  So just did not work for us.
There are times it is needed. 
Some backups, like if you want to do exchange where you can restore just
one email, needed the backup to be on disk, you could not do that from
tape.  So you have to look at what kind of backups and restores you do
and if you need the backup on disk to so the restore.

-Original Message-
From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Victor
Engle
Sent: Sunday, April 25, 2010 2:15 PM
To: veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] Architectural question (staging)

Hello List,

Just wanted to get some opinions about whether disk staging units are
worthwhile. My backup server has two BasicDisk staging units with the
storage units configured such that the data goes to disk and is then
moved to tape. I have a tape library with four LTO-3 drives connected
via FC. So what I'm wondering is, since the LTO drives are reasonably
fast, and since I'm writing the data ultimately to tape anyway, would
it be better to just write directly to tape. The disk is just old
fashioned spinning disk with no de-duplication so there are
operational costs for the disks. All tape and disk storage units are
local to the backup server. I'm thinking it would be better to add LTO
drives and eliminate the disk for now and maybe later add a
de-duplicating disk unit.

Under what circumstances does it make sense to stage data on disk. I
would appreciate hearing what your thoughts and experiences are with
regard to disk staging.

Thanks,
Vic
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu


___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
 
Proud partner. Susan G. Komen for the Cure.
 
Please consider our environment before printing this e-mail or attachments.
--
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential 
information and is for the sole use of the intended recipient(s). If you are 
not the intended recipient, any disclosure, copying, distribution, or use of 
the contents of this information is prohibited and may be unlawful. If you have 
received this electronic transmission in error, please reply immediately to the 
sender that you have received the message in error, and delete it. Thank you.
--
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[judy_hinchcliffe]

I am in agreement with Ed,
We could backup to disk ok, but getting it of disk to tape took longer
and we could not get it finished before the next nights backups, and
that was when we had SDLT tape drives.
I now have LTO4 drives and backup straight to tape and I still cannot
keep all the tape drives busy.  So just did not work for us.
There are times it is needed. 
Some backups, like if you want to do exchange where you can restore just
one email, needed the backup to be on disk, you could not do that from
tape.  So you have to look at what kind of backups and restores you do
and if you need the backup on disk to so the restore.

-Original Message-
From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Victor
Engle
Sent: Sunday, April 25, 2010 2:15 PM
To: veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] Architectural question (staging)

Hello List,

Just wanted to get some opinions about whether disk staging units are
worthwhile. My backup server has two BasicDisk staging units with the
storage units configured such that the data goes to disk and is then
moved to tape. I have a tape library with four LTO-3 drives connected
via FC. So what I'm wondering is, since the LTO drives are reasonably
fast, and since I'm writing the data ultimately to tape anyway, would
it be better to just write directly to tape. The disk is just old
fashioned spinning disk with no de-duplication so there are
operational costs for the disks. All tape and disk storage units are
local to the backup server. I'm thinking it would be better to add LTO
drives and eliminate the disk for now and maybe later add a
de-duplicating disk unit.

Under what circumstances does it make sense to stage data on disk. I
would appreciate hearing what your thoughts and experiences are with
regard to disk staging.

Thanks,
Vic
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu


___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Martin, Jonathan]

We use Disk Stage Storage Units (DSSU) almost exclusively for our
backups. As someone has already mentioned, you can stream many slow
clients to a DSSU without impacting your speed significantly. To do this
with tape you would have to use multiplexing, which is a real
performance killer come restore time.  The DSSUs essentially allow you
to "multiplex" to disk, then single stream to tape. Regarding Ed's speed
issue below, I've got data here that directly correlates number of
concurrent streams written to a DSSU with the performance to tape.
You'll need to balance this when setting this on the DSSU, but we get
away with 8-12 streams on 14x500GB SATA Disks in a Raid-5 and still
drive LTO3.

 

Here we were also able to purchase a much smaller tape library.  I
replaced a library with 8 x SDLT220 drives with a library with 2 x LTO3
drives (and disk storage.)  The disk storage is far more reliable than
tapes and libraries. As Ed noted below you also benefit from having your
backups on disk which makes restores very snappy. 

 

-Jonathan

 

From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Ed Wilts
Sent: Sunday, April 25, 2010 3:45 PM
To: Victor Engle
Cc: veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Architectural question (staging)

 

On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle 
wrote:

Just wanted to get some opinions about whether disk staging units are
worthwhile. My backup server has two BasicDisk staging units with the
storage units configured such that the data goes to disk and is then
moved to tape. I have a tape library with four LTO-3 drives connected
via FC. So what I'm wondering is, since the LTO drives are reasonably
fast, and since I'm writing the data ultimately to tape anyway, would
it be better to just write directly to tape. The disk is just old
fashioned spinning disk with no de-duplication so there are
operational costs for the disks. All tape and disk storage units are
local to the backup server. I'm thinking it would be better to add LTO
drives and eliminate the disk for now and maybe later add a
de-duplicating disk unit.


We worked with disk staging units for at least a year before we mostly
gave up.  The biggest challenge we ran into was that destaging was too
slow. Even though we proved to Symantec that we could read from those
disk drives at over 100MB/sec, we could never destage even half that
fast.  We had an open case with Symantec for a VERY long time before we
agreed that it wasn't going to get fixed.

Under what circumstances does it make sense to stage data on
disk. I
would appreciate hearing what your thoughts and experiences are
with
regard to disk staging.


There are times when DSSUs make sense.  1.  If you don't have a tape
drive free but want to do a backup anyway - we still use DSSUs for
things like small backups of Oracle archive logs. 2.  If you need to
throttle your backups, especially across things like a bunch of virtual
servers on the same physical server.  NBU only allows you to set the
maximum jobs per client name, not per client.  DSSUs make an acceptable
choke point for clusters.  3.  If you have small backups, but don't have
a lot of them at once, multiplexing may not buy you enough performance
boosts.  Use DSSUs to write those little jobs to disk and then destage
them at once.

If you currently multiplex, realize that your restores are going to be
slower than if you don't multiplex.  All tapes created from a DSSU
destage are non-multiplexed so your restores can go faster.

DSSUs also give you a staging area for restores.  If your tapes go
offsite, you may still be able to do a restore from the staging unit the
next day (or longer) depending on how big your stagig units are.  NBU is
smart enough to realize that if the same data is on both disk and tape
and you kick off a restore, the restore will automatically come from
disk.

In general, I'd say that there is a place for DSSUs but it's not the
great benefit we thought it was going to be.

   .../Ed

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[William Brown]

DSSUs are OK but not as useful as e.g. AdvancedDisk pools for staging.  The 
algorithm for how it selects items to delete is very different.  I think you 
need to turn the question on its head.  Disk staging is not useful if your 
backups  (including incremental) can always stream the drive.  Ours cannot, 
Windows VMs are just too slow.  Disk staging allows the slow backups to be 
decoupled from the tape drives.  You don't have to dream up multiplexing levels 
that are different in full backups and incrementals; I doubt even if we set MPX 
to 32 the Windows VM incrementals could stream the LTO3/4, and LTO5 started 
shipping.

Using staging, disk does not care how slow the data arrives.  When it destages, 
you get proper use of the tape drives.  More importantly, the drives are not 
hogged by slow show-shine jobs, leaving other jobs queuing and in some cases 
hitting the end of the window. With SSO shared drives this can be painful.  So, 
what I'm saying is don't just think about your fast clients, think how to 
prevent the slow ones being a pain.

I'd even suggest considering sending FULL backups direct to tape and the INCR 
via staging.  With INCR the time taken to tee up the tape drive can be out of 
proportion to the backup duration, and it allocates the drive before it even 
starts to decide what files to backup...which may be very few.  Backups to disk 
start running so much faster, no mount time.  So for a backup that runs for < 5 
minutes tape is painful.

But more broadly, identify the slow backups that hog the drives, and if the 
total size will fit your staging, do it.  Destaging to tape makes the tape 
drives sing (OK, poetic licence there).  But you can use fewer drives and that 
means less $$ on annual maintenance.

William D L Brown


-Original Message-
From: veritas-bu-boun...@mailman.eng.auburn.edu 
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Victor Engle
Sent: 25 April 2010 20:15
To: veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] Architectural question (staging)

Hello List,

Just wanted to get some opinions about whether disk staging units are
worthwhile. My backup server has two BasicDisk staging units with the
storage units configured such that the data goes to disk and is then
moved to tape. I have a tape library with four LTO-3 drives connected
via FC. So what I'm wondering is, since the LTO drives are reasonably
fast, and since I'm writing the data ultimately to tape anyway, would
it be better to just write directly to tape. The disk is just old
fashioned spinning disk with no de-duplication so there are
operational costs for the disks. All tape and disk storage units are
local to the backup server. I'm thinking it would be better to add LTO
drives and eliminate the disk for now and maybe later add a
de-duplicating disk unit.

Under what circumstances does it make sense to stage data on disk. I
would appreciate hearing what your thoughts and experiences are with
regard to disk staging.

Thanks,
Vic
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu


---
This e-mail was sent by GlaxoSmithKline Services Unlimited 
(registered in England and Wales No. 1047315), which is a 
member of the GlaxoSmithKline group of companies. The 
registered address of GlaxoSmithKline Services Unlimited 
is 980 Great West Road, Brentford, Middlesex TW8 9GS.
---

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Architectural question (staging)

[Ed Wilts]

On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle wrote:

> Just wanted to get some opinions about whether disk staging units are
> worthwhile. My backup server has two BasicDisk staging units with the
> storage units configured such that the data goes to disk and is then
> moved to tape. I have a tape library with four LTO-3 drives connected
> via FC. So what I'm wondering is, since the LTO drives are reasonably
> fast, and since I'm writing the data ultimately to tape anyway, would
> it be better to just write directly to tape. The disk is just old
> fashioned spinning disk with no de-duplication so there are
> operational costs for the disks. All tape and disk storage units are
> local to the backup server. I'm thinking it would be better to add LTO
> drives and eliminate the disk for now and maybe later add a
> de-duplicating disk unit.
>

We worked with disk staging units for at least a year before we mostly gave
up.  The biggest challenge we ran into was that destaging was too slow. Even
though we proved to Symantec that we could read from those disk drives at
over 100MB/sec, we could never destage even half that fast.  We had an open
case with Symantec for a VERY long time before we agreed that it wasn't
going to get fixed.

Under what circumstances does it make sense to stage data on disk. I
> would appreciate hearing what your thoughts and experiences are with
> regard to disk staging.
>

There are times when DSSUs make sense.  1.  If you don't have a tape drive
free but want to do a backup anyway - we still use DSSUs for things like
small backups of Oracle archive logs. 2.  If you need to throttle your
backups, especially across things like a bunch of virtual servers on the
same physical server.  NBU only allows you to set the maximum jobs per
client name, not per client.  DSSUs make an acceptable choke point for
clusters.  3.  If you have small backups, but don't have a lot of them at
once, multiplexing may not buy you enough performance boosts.  Use DSSUs to
write those little jobs to disk and then destage them at once.

If you currently multiplex, realize that your restores are going to be
slower than if you don't multiplex.  All tapes created from a DSSU destage
are non-multiplexed so your restores can go faster.

DSSUs also give you a staging area for restores.  If your tapes go offsite,
you may still be able to do a restore from the staging unit the next day (or
longer) depending on how big your stagig units are.  NBU is smart enough to
realize that if the same data is on both disk and tape and you kick off a
restore, the restore will automatically come from disk.

In general, I'd say that there is a place for DSSUs but it's not the great
benefit we thought it was going to be.

   .../Ed
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu