Re: [Veritas-bu] Compression of catalog images

2008-07-16 Thread Dean 
The impact of compression during backup windows may be significant when you
first turn it on. It may have a lot of compressing to do. I mitigated this
by setting compression on for images older than 1 year, then the next day
set it to 6 months, then 3 months the next day, then 1 month, so the initial
"surge" in compression was spread over several days.

If I recall correctly, decompression of the catalog image occurs before you
initate the restore. The catalog image gets decompressed when you browse it
using the restore GUI.

This of course means that doing large searches for files using the GUI (or
command line) can cause your catalog to grow significantly in a short period
of time.

If you are restore from something like RMAN, I guess the decompression would
occur when RMAN starts the restore for each image. But any delay should be
negligible since the catalog images are so small - just containing one file.

The degree of the impact will of course be highly dependent on how much
processing power you have available on your master server, the I/O
throughput to your catalog, and the number of files in the backup you are
trying to restore from.

We compress catalog entries older than about 30 days. Not that we restore
from them often, but when we do, the delay while the image is decompressed
is not noticeable. (But then, the backup/restore GUI is never "snappy" - I
guess I am used to the spinning hourglass).

- Dean

On Thu, Jul 17, 2008 at 9:49 AM, Justin Piszcz <[EMAIL PROTECTED]>
wrote:

>
>
> On Wed, 16 Jul 2008, Abhishek Dhingra1 wrote:
>
> > Justin,
> >
> >  What is the impact of doing the restore, does it take more time.if yes,
> > then hw much,
> >
> > We have around 1000 clients in our environment, with no free window.
> >
> > As per the document , compression run after the each backup session, as
> we
> > dont have any free window, if we configure the compression at what time
> > compression will run.
> >
> > Any suggestion will be appreciated.
>
> Abhishek,
>
> Best to do test restores, it can impact it quite a bit if you are restoring
> many small files, up to 10-30 minutes on slower disks & I/O subsystems.
>
> You also need to have enough space to decompress the compressed images.
>
> It depends on the I/O / where the images are stored.
> It depends on how many files you are restoring.
>
> Justin.
>
>
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Image compression algorithm or enthusiastic operator?

2008-07-16 Thread Tim Hoke 
That's certainly not the normal NetBackup format.  Normally files are in the
ctime directories, have names just like the .f file and end in .f.Z.
 NetBackup uses the native compress on UNIX.
It'd be interesting to see what's in the files you list below.  I'd also be
curious if a bpdbm -consistency check would report them as errors.

FWIW, it looks like some of your STREAMS files have been mucked with too.
 Notice the old dates on some of those files.  I'm pretty sure you only need
the STREAMS and STREAMS.lck files.  Be careful changing those, if you remove
them, your next incremental may become a full (So only clean them up when
your next backup is scheduled to be a full).

-Tim

On Wed, Jul 16, 2008 at 11:42 AM, <[EMAIL PROTECTED]> wrote:

> I found this in the netbackup/db/images/ directory for a client
> of mine.
>
> Did I get an enthusiastic operator compressing stuff or is this part of
> the catalog compression algorithm (the XXX.tar.gz) files on top?
>
> The directory is far too big for the number of files I think we're
> tracking here.
>
> > ls -l
> total 16367638
> -rw-r--r--   1 root other  44146 Dec 31  2007 102.tar.gz
> -rw-r--r--   1 root other  55176 Dec 31  2007 103.tar.gz
> -rw-r--r--   1 root other  47607 Dec 31  2007 104.tar.gz
> -rw-r--r--   1 root other  61527 Dec 31  2007 105.tar.gz
> -rw-r--r--   1 root other  35792 Dec 31  2007 106.tar.gz
> -rw-r--r--   1 root other1543518284 Dec 31  2007 107.tar.gz
> -rw-r--r--   1 root other1446773287 Dec 31  2007 108.tar.gz
> -rw-r--r--   1 root other2962613248 Dec 31  2007 109.5.tar.gz
> -rw-r--r--   1 root other2426976416 Dec 31  2007 109.9.tar.gz
> drw-r-xr-x   3 root other   2048 Oct  8  2007 11
> drw-r-xr-x   3 root other   4096 Oct  8  2007 110100
> 
> drw-r-xr-x   2 root root1024 Oct  8  2007 112300
> drw-r-xr-x   2 root other   1024 Oct  8  2007 112400
> -rw---   1 root other309 Jul 11 07:22 STREAMS
> -rw-rw-rw-   1 root other 16 Jul 27  2006
> STREAMS.ALT_NAME.26477
> -rw---   1 root other   4253 Jun  2  2006 STREAMS.alt
> -rw---   1 root other 15 Jul 11 07:22 STREAMS.lck
> -rw---   1 root other 17 Nov 16  2005 STREAMS.wl.lck
>
> -M
>
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Compression of catalog images

2008-07-16 Thread Justin Piszcz 


On Wed, 16 Jul 2008, Abhishek Dhingra1 wrote:

> Justin,
>
>  What is the impact of doing the restore, does it take more time.if yes,
> then hw much,
>
> We have around 1000 clients in our environment, with no free window.
>
> As per the document , compression run after the each backup session, as we
> dont have any free window, if we configure the compression at what time
> compression will run.
>
> Any suggestion will be appreciated.

Abhishek,

Best to do test restores, it can impact it quite a bit if you are restoring
many small files, up to 10-30 minutes on slower disks & I/O subsystems.

You also need to have enough space to decompress the compressed images.

It depends on the I/O / where the images are stored.
It depends on how many files you are restoring.

Justin.


___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Allocation Failure when doing bpimmedia NBU 5.0MP6

2008-07-16 Thread Tim Hoke 
Check out this TechNote:

http://seer.support.veritas.com/docs/273884.htm

There's an internal timeout which gets hit when querying a large catalog.

-Tim

On Wed, Jul 16, 2008 at 2:30 PM, Jim VandeVegt <[EMAIL PROTECTED]> wrote:

> I am getting an allocation failure (only error reported) when attempting to
> run bpimmedia against individual media IDs.  Sometimes the command reports
> some image data before dumping the error, sometimes it does not. It usually
> takes several minutes before the error dumps out.
>
> My environment is not particularly big. Images database is 16 GB as
> reported by du. Usually run about a 1000 jobs per night.
>
> Solaris 8 master (Sun V240) running NBU 5.0MP6. I am executing the command
> on the server as root.
>
> I have also received similar results when querying the same media from the
> Catalog section of the GUI.
>
> I did a Google search for this and all I found was a reference to a bug
> fixed in V3.2. The bug was bprd would reach 256MB in size during a long
> restore and dump the error. (Sun patch 108261)
>
> I tried right after a complete reboot ... no help.
>
> Does not seem to affect all media, which is strange.
>
> Would something under the guise of ulimit be at work here?  Or maybe
> /etc/system entries?
>
> Some outputs:
> # ulimit -Ha
> core file size (blocks) unlimited
> data seg size (kbytes)  unlimited
> file size (blocks)  unlimited
> open files  4096
> pipe size (512 bytes)   10
> stack size (kbytes) unlimited
> cpu time (seconds)  unlimited
> max user processes  4091
> virtual memory (kbytes) unlimited
>
> # ulimit -Sa
> core file size (blocks) unlimited
> data seg size (kbytes)  unlimited
> file size (blocks)  unlimited
> open files  512
> pipe size (512 bytes)   10
> stack size (kbytes) 8192
> cpu time (seconds)  unlimited
> max user processes  4091
> virtual memory (kbytes) unlimited
>
> --
> Jim VandeVegt
> VandeVegt @t yahoo.com, Jim.VandeVegt @t PhysiciansMutual.com
> Eliminate the IRS and put the fair consumption tax in place.
> Visit http://www.fairtax.org/ 
> "Quid quid latine dictum sit, altum videtur."
>
>
>
>
> ___
> Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

[Veritas-bu] Allocation Failure when doing bpimmedia NBU 5.0MP6

2008-07-16 Thread Jim VandeVegt 
I am getting an allocation failure (only error reported) when attempting to run 
bpimmedia against individual media IDs.  Sometimes the command reports some 
image data before dumping the error, sometimes it does not. It usually takes 
several minutes before the error dumps out.

My environment is not particularly big. Images database is 16 GB as reported by 
du. Usually run about a 1000 jobs per night.

Solaris 8 master (Sun V240) running NBU 5.0MP6. I am executing the command on 
the server as root.

I have also received similar results when querying the same media from the 
Catalog section of the GUI.

I did a Google search for this and all I found was a reference to a bug fixed 
in V3.2. The bug was bprd would reach 256MB in size during a long restore and 
dump the error. (Sun patch 108261)

I tried right after a complete reboot ... no help.

Does not seem to affect all media, which is strange.

Would something under the guise of ulimit be at work here?  Or maybe 
/etc/system entries?

Some outputs:
# ulimit -Ha
core file size (blocks) unlimited
data seg size (kbytes)  unlimited
file size (blocks)  unlimited
open files  4096
pipe size (512 bytes)   10
stack size (kbytes) unlimited
cpu time (seconds)  unlimited
max user processes  4091
virtual memory (kbytes) unlimited

# ulimit -Sa
core file size (blocks) unlimited
data seg size (kbytes)  unlimited
file size (blocks)  unlimited
open files  512
pipe size (512 bytes)   10
stack size (kbytes) 8192
cpu time (seconds)  unlimited
max user processes  4091
virtual memory (kbytes) unlimited

 

Jim VandeVegt
VandeVegt @t yahoo.com, Jim.VandeVegt @t PhysiciansMutual.com
Eliminate the IRS and put the fair consumption tax in place.
Visit http://www.fairtax.org/
"Quid quid latine dictum sit, altum videtur."


  ___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

[Veritas-bu] Image compression algorithm or enthusiastic operator?

2008-07-16 Thread Mark.Donaldson 
I found this in the netbackup/db/images/ directory for a client
of mine.  

Did I get an enthusiastic operator compressing stuff or is this part of
the catalog compression algorithm (the XXX.tar.gz) files on top?

The directory is far too big for the number of files I think we're
tracking here.

> ls -l
total 16367638
-rw-r--r--   1 root other  44146 Dec 31  2007 102.tar.gz
-rw-r--r--   1 root other  55176 Dec 31  2007 103.tar.gz
-rw-r--r--   1 root other  47607 Dec 31  2007 104.tar.gz
-rw-r--r--   1 root other  61527 Dec 31  2007 105.tar.gz
-rw-r--r--   1 root other  35792 Dec 31  2007 106.tar.gz
-rw-r--r--   1 root other1543518284 Dec 31  2007 107.tar.gz
-rw-r--r--   1 root other1446773287 Dec 31  2007 108.tar.gz
-rw-r--r--   1 root other2962613248 Dec 31  2007 109.5.tar.gz
-rw-r--r--   1 root other2426976416 Dec 31  2007 109.9.tar.gz
drw-r-xr-x   3 root other   2048 Oct  8  2007 11
drw-r-xr-x   3 root other   4096 Oct  8  2007 110100

drw-r-xr-x   2 root root1024 Oct  8  2007 112300
drw-r-xr-x   2 root other   1024 Oct  8  2007 112400
-rw---   1 root other309 Jul 11 07:22 STREAMS
-rw-rw-rw-   1 root other 16 Jul 27  2006
STREAMS.ALT_NAME.26477
-rw---   1 root other   4253 Jun  2  2006 STREAMS.alt
-rw---   1 root other 15 Jul 11 07:22 STREAMS.lck
-rw---   1 root other 17 Nov 16  2005 STREAMS.wl.lck

-M

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Compression of catalog images

2008-07-16 Thread Abhishek Dhingra1 
Justin,

  What is the impact of doing the restore, does it take more time.if yes, 
then hw much,

We have around 1000 clients in our environment, with no free window.

As per the document , compression run after the each backup session, as we 
dont have any free window, if we configure the compression at what time 
compression will run.

Any suggestion will be appreciated.


Abhishek Dhingra

IBM Global Services, Delhi,
Email : [EMAIL PROTECTED]
Mobile : +91-9818675370



Justin Piszcz <[EMAIL PROTECTED]> 
16/07/08 01:31 PM

To
Abhishek Dhingra1/India/[EMAIL PROTECTED]
cc
veritas-bu@mailman.eng.auburn.edu
Subject
Re: [Veritas-bu] Compression of catalog images






It's broken with 6.0MP5 (catalog compression) upgrade to 6.0MP6 or later 
before doing that! It works pretty well, compressed a ~100-120GiB catalog 
to ~25-35GiB for us.

Justin.

On Wed, 16 Jul 2008, Abhishek Dhingra1 wrote:

> Hi All,
>
>   We are using Netbackup 6.0 MP5, our netbackup catalog is around 300 GB
> and more are  Infinite retention backups. I am Planning of compressing 
the
> images , Need to know if any one of you using this feature, how much
> feasible is compressing the images and pros and cons of the same.
>
> Also want to know the steps in performing the compression.
>
> Abhishek Dhingra
>
> IBM Global Services, Delhi,
> Email : [EMAIL PROTECTED]
> Mobile : +91-9818675370

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] Compression of catalog images

2008-07-16 Thread Justin Piszcz 
It's broken with 6.0MP5 (catalog compression) upgrade to 6.0MP6 or later 
before doing that! It works pretty well, compressed a ~100-120GiB catalog 
to ~25-35GiB for us.

Justin.

On Wed, 16 Jul 2008, Abhishek Dhingra1 wrote:

> Hi All,
>
>   We are using Netbackup 6.0 MP5, our netbackup catalog is around 300 GB
> and more are  Infinite retention backups. I am Planning of compressing the
> images , Need to know if any one of you using this feature, how much
> feasible is compressing the images and pros and cons of the same.
>
> Also want to know the steps in performing the compression.
>
> Abhishek Dhingra
>
> IBM Global Services, Delhi,
> Email : [EMAIL PROTECTED]
> Mobile : +91-9818675370
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu