[Veritas-bu] Improving Duplication Performance from DataDomain Restorers
Here's my set up. I have a Sparc T2000 with 16 1.2Ghz cores and 16Gb of RAM running Solaris 10. I have three DataDomain restorers, two DD460s and a DD565. These are connected to a dedicated backup gigabit backup network and jumbo frames are implemented on the DataDomains and the T2000 interfaces that talk to the backup subnet. I write all of my backups to disk storage units mounted from the DataDomains or to a VTL on the DD565. I keep these for four weeks. I then duplicate the images from the DataDomains to LTO4 tape with retention levels on the duplicates ranging from two months to infinity (for certain archival data) and offsite them with Iron Mountain. I've implemented the system tuning recommendations in the DataDomain white paper on best practices with DataDomain restorers and NetBackups. I've tuned the NetBackup parameters as follows: NET_BUFFER_SZ=262144 NUMBER_DATA_BUFFERS=256 NUMBER_DATA_BUFFERS_DISK=256 SIZE_DATA_BUFFERS=262144 SIZE_DATA_BUFFERS_DISK=262144 To take advantage of the increased value of NET_BUFFER_SZ I've increased tcp_recv_hiwat and tcp_xmit_hiwat to 262144 bytes from the default Solaris 10 value of 49152 bytes. The result of this is that when things are going well, as they are this morning, I can get between 120 and 150 megabytes per second throughput to two LTO4 drives, when things aren't going well I can see drive throughput drop to kilobytes per second (as measured by iostat). The latter condition seems to be caused by duplicating smaller backup images, bpdm has to be called, it opens the backup image, reads it, sends the data to bptm and bptm writes it to tape. When I'm duplicating the smaller images I can watch the logs and see each bpdm process being spawned and then each bptm being spawned and I can watch my poor LTO4 tape drives idling at 4 kilobytes per second, which kills me and makes me wonder if PETLTO4TD (People for the Ethical Treatment of LTO4 Tape Drives) are going to come and take my tape drives away. Since all of my primary backups are written to a disk based storage unit multiplexing is set to 1. Is there a way to multiplex duplication from disk storage units? Failing that is there some way of writing to tape asynchronously so that the the buffers on the tape drives could be filled before the write actually happen. Are there tuning steps that I haven't taken that could improve the duplication performance? Any help will be, as it has been in the past, greatly appreciated. Thank You, Jamie Jamison Network Systems Administrator ZymoGenetics ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] My Experiences with DataDomain Restorers
I've been using DataDomain restorers, the 460 and 565 series for almost three years now and here's my opinion of them, good and bad. Good: DeDuplication - DataDomain's de-duplication claims are accurate and the deduplication performance is impressive. I'm seeing compression ratios of 4:1 (ORACLE policy type backups) to 17.1 (general filesystem backups of systems with both STANDARD and MS-WINDOWS-NT policy types. For hot catalog backups I'm seeing a compression ratio of 77:1. Replication - Replication also works well, which makes implementing a DR plan for your backup system much easier. I write hot catalog backups and my DR info to a DSU on my primary restorer once a day. Implementing a DR plan for NetBackup becomes a lot easier with this kind of technology because it takes care of replicating all of your backup data and your catalogs and DR info to your remote site. Performance - Backup performance is very good and restores are wicked fast. Before I got my restorers set up I was running a StorageTek L180 library with eight LTO2 tape drives 24x7. I ran my primary backups and then duplicated them to tape and kept the duplicated copies onsite with a three week retention period. I was very close to running out of slots in the library for backup tapes and onsite duplication and I was getting to a point where having even one drive go down was seriously impacting my schedule. Installing two 460 series restorers, with two at the DR site for replication solved this problem. I write primary backups to the restorer with a one month retention period and then duplicate them to tape with a longer retention period for offsite vaulting. There are still performance issues related to the number of streams that the restorer can handle, if you have streams open for reading data from the restorer for duplication to tape then you can't have as many write streams open for backups, but these are minor. The most I've had to do is stop a running duplication job during the backup window and then let duplication catch up once the backups are done. Being able to restore from a DSU with a long retention period is awesome, it's like having NetApp snapshot restores for all of your data. The Oracle and Exchange administrators I work with love this. Installing the two DataDomain restorers allowed me to hold off on upgrading my tape library for eighteen months. Field engineering support - The DataDomain field engineers I have worked with are knowledgable, efficient and friendly. They reallly know the equipment and know the ins and outs of NetBackup and how best to configure it for use with the DataDomain equipment.DataDomain contracts out their routine technical support to Glasshouse Technologies, who have been OK so far. Ease of installation and configuration - Configuring a restorer takes about 15 minutes. There is a menu driven configuration utility at the CLI that runs you through all of the steps and once that's done you mount the restorer filesystems as NFS volumes or CIFS shares on your NetBackup master or media server, configure these filesystems as disk storage units and start using the system. I have not used the Open Storage Option yet but am looking forward to it. Really the hardest part about configuring a restorer is getting it into a rack. User Interface - The GUI is very good and the CLI is superb. You can have multiple CLI sessions via ssh and the CLI supports tab completion, command line history and if you enter a command without any arguments will tell you what the possible arguments are. There's a CS term to describe this but I don't know what it is, but as an example if you want to see all of the arguments to the command replication You type replication at the command line. One of the arguments for replication is show. If you want to see all of the arguments for replication show you type replication show at the command line and it shows you replication show history, replication show config, replication show performance and replication show stats. I'm a CLI guy and I love that I can quickly check on the status of the system by connecting to it with ssh and running a handful of commands instead of having to, as you do with a NetApp filer, connect with a web interface and put up with a GUI because the CLI is crippled. The online documentation in the CLI is also superb with the help system showing good and relevant examples for each command. I've rarely had to RTFM with my restorers. Bad: FLAMING RESTORERS OF DEATH! - Last year we had one of our DDR460 restorers catch fire. Well, actually it didn't catch fire, according to the DataDomain tech support people the restorers are built from UL listed fire resistant materials, so what actually happened is that the system midplane that the disk drives are connected to developed a short circuit, heated to 950 degrees Celsius and melted. I found out about this when I didn't get my morning status e-mail from the restorer in question. I tried pinging it and getting
[Veritas-bu] ZFS backup problems
I have two servers that I've configured with ZFS filesystems but I haven't had a chance to test them with NetBackup yet. Do ZFS filesystems that are mounted via the /etc/vfstab versus the ZFS mounting facility file back up properly with the ALL_LOCAL_DRIVES directive? Jamie ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Problems Uninstalling BMR 6.5.2
I upgraded my Solaris 10 NetBackup master to 6.5.2A by doing a clean install of 6.5 and then upgrading to 6.5.2A and everything went well up until the point where I wanted to start backups. nbemm would not start and nothing I could do would make it start nor would it run in console mode. I tried rolling back to 6.5 and was able to successfully uninstall patches until I got to uninstalling BMR. When I run NB.update.uninstall to uninstall BMR I get an error message that says: Running preuninstall script. See /usr/openv/pack/pack.history for more details. /usr/openv/pack/NB_BMR_6.5.2/save/VrtsNB_BMR_6.5.2.preuninstall: Running. Hardware/OS Type=Solaris/Solaris10 Downgrading NBDB: /usr/openv/db/bin/nbdb_upgrade -dbn BMRDB -downgrade ERROR: /usr/openv/db/bin/nbdb_upgrade -dbn BMRDB -downgrade command failed. Unable to downgrade database to original level. Refer to the log file in /usr/openv/netbackup/logs/nbdb for more information. Rerun Vrts_pack.uninstall once the problem has been resolved. Pack uninstall failure. Exiting NB_update.uninstall Then if I try to manually run the command to downgrade the BMR database I get the following error message: [EMAIL PROTECTED] on (pts/6) at 00:10:36, Fri Aug 01 /usr/openv/pack/NB_BMR_6.5.2/save 174 /usr/openv/db/bin/nbdb_upgrade -dbn BMRDB -downgrade Verifying the running version of BMRDB ... Verification Failed. So at this point I'm stuck because I can't finish the rollback to 6.5 from 6.5.2A. Fortunately I was able to fall back to my current 6.0 environment and start backups so I'm not totally screwed but I do need to get 6.5 running. Is there a way I can just do a complete and total uninstall of 6.5 instead of just a rollback from the current, broken 6.5.2A install to see if I can get that running? As always thanks in advance for any help, advice or commiseration you can send my way. Jamie Jamison Network Systems Administrator ZymoGenetics ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Backing up network drives in Windows
I have several policies that back up network drives through Windows and I use the UNC path for them. You have to enter this manually but it insures that if the drive letter is changed the filesystem is still backed up. Jamie Jamison Network Systems Administrator Zymogenetics - Original Message - From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: veritas-bu@mailman.eng.auburn.edu veritas-bu@mailman.eng.auburn.edu Sent: Tue May 20 10:00:06 2008 Subject: Veritas-bu Digest, Vol 25, Issue 42 Send Veritas-bu mailing list submissions to veritas-bu@mailman.eng.auburn.edu To subscribe or unsubscribe via the World Wide Web, visit http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of Veritas-bu digest... Today's Topics: 1. Backup Network drives (Jerry Rioux) -- Message: 1 Date: Tue, 20 May 2008 12:29:04 -0400 From: Jerry Rioux [EMAIL PROTECTED] Subject: [Veritas-bu] Backup Network drives To: veritas-bu@mailman.eng.auburn.edu Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1 I'm adding a Windows backup policy and the network drives do not show up as a selection in the file list. I have checked the backup network drives on the attributes list. What am I missing? Nbu 6.5 Windows master server. Thanks On 5/20/08, Dave Carpe [EMAIL PROTECTED] wrote: Linux on x64 is referring to the AMD Opteron and Intel EM-64T processors, so x86_64 would be correct. Itanuium needs a completely different version and is 64 bit only. David K. Carpe Principal Systems Engineer Symantec Corporation Office: 646.487.6012 Mobile: 908.963.6818 Home Office: 973.940-1805 email: [EMAIL PROTECTED] From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff Lightner Sent: Tuesday, May 20, 2008 10:51 AM To: WEAVER, Simon (external); veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] looking for x64 NBU 5.1 Client What do you mean by x64. Linux x86_64? Itanium? Something else? If it is Linux x86_64 bit you can install the 2.4 kernel client (32 bit) and add compatibility libraries to make it work properly. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of WEAVER, Simon (external) Sent: Tuesday, May 20, 2008 10:26 AM To: veritas-bu@mailman.eng.auburn.edu Subject: [Veritas-bu] looking for x64 NBU 5.1 Client All Although I have managed to find 5.1 MP5 for x64, I cannot find a client. I have an x64 on CD, but I am getting alot of problems with the volume manager service refusing to start. Is there a way to get the software, without the need to contact Symantec? Thanks, Si This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Astrium disclaims any and all liability if this email transmission was virus corrupted, altered or falsified. - Astrium Limited, Registered in England and Wales No. 2449259 REGISTERED OFFICE:- Gunnels Wood Road, Stevenage, Hertfordshire, SG1 2AS, England -- CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you. -- -- Sent from Gmail for mobile | mobile.google.com -- ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu End of Veritas-bu Digest, Vol 25, Issue 42 ** ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Veritas-bu Digest, Vol 24, Issue 60 - Changing media server for restore
Today's Topics: 1. Changing media server for restore (Paiz, Eduardo) -- Message: 1 Date: Thu, 24 Apr 2008 11:11:32 -0400 From: Paiz, Eduardo [EMAIL PROTECTED] Subject: [Veritas-bu] Changing media server for restore To: veritas-bu@mailman.eng.auburn.edu Message-ID: [EMAIL PROTECTED] om Content-Type: text/plain; charset=us-ascii I'm running NBU 6.0 MP5. A couple of weeks ago we retired one of our Media servers and brought a new one online (Different host names). I'm attempting to restore some files from one of my host, but the restore is looking for the old media server? How can I make NBU use the new media server for the restore? Thank you in advance, Eduardo J. Paiz -- You should be able to repair this by opening the NetBackup Management/Host Properties/Master Server pane. In the Master Server pane click on General Server. There is a dialog box for Media Host Override. Entering the name of the original media server and the new server that you are trying to restore from should solve your problem. Jamie Jamison Network Systems Administrator ZymoGenetics ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] MS-Exchange Server Backup Problems With MS-Exchange policy type.
Date: Thu, 17 Apr 2008 10:57:12 +0100 From: James Pattinson [EMAIL PROTECTED] Subject: [Veritas-bu] Backup of exchange using wrong name To: veritas-bu@mailman.eng.auburn.edu Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi All I'm backing up a customer's machine over a dedicated backup network with our own naming convention, which doesn't always relate to the actual name of the machine. This has been working OK for Filesystem and SQL backups, but I am unable to get an exchange backup working in this way. Example Exchange server is called exch We back it up as cust-exch-b and a filesystem backup works, but exchange backup doesn't work, we get this in the bpbkar log: 10:26:37.515: [5140.964] 2 tar_backup_tfi::setupFileDirectives: TAR - backup filename = Microsoft Information Store:\ 10:26:37.530: [5140.5648] 4 tar_base::keepaliveThread: INF - keepalive thread is active with an interval of 60 seconds 10:26:38.593: [5140.964] 2 ov_log::V_GlobalLog: INF - BEDS_Init() Enter InitFlags:0x0 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: C: 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: D: 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Terminal Services 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Windows Network 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Web Client Network 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: \\cust-exch-b\Microsoft Exchange Mailboxes 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: \\cust-exch-b\Microsoft Exchange Public Folders 10:26:43.171: [5140.964] 2 ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Shadow?Copy?Components 10:26:43.171: [5140.964] 4 dos_backup::V_VerifyFileList: INF - unable to determine UBS type for:Microsoft Information Store: 10:26:43.171: [5140.964] 4 dos_backup::V_VerifyFileList: INF - UBS Local Type for 'Microsoft Information Store:\' -- 10:26:43.171: [5140.964] 4 dos_backup::V_Initialize: INF - Exchange Backup - Disabling Open File Backups, Archive Bit processing, TIR and Job Estimating. 10:26:43.171: [5140.964] 4 backup_create: INF - NetBackup Temp Directory: 'D:\Program Files\Veritas\\NetBackup\Temp' 10:26:43.171: [5140.964] 2 tar_base::V_vTarMsgW: ERR - object not found for file system backup: Microsoft Information Store: So it looks like I'm missing a bit of configuration on the Exchange agent to tell it to use a certain name when talking to Exchange... Can anyone help? Thanks! James --- There are two peculiarities with MS Exchange Public Folder, Mailbox and Information store backups that are not well documented, or at least were not well documented in NetBackup 4.5 and 5.x. The first is that the client name of your server in the Host Properties/Client Properties/Client Name dialog for the Exchange server needs to be the NetBIOS name for the server. If the NetBIOS name is not the same as the FQDN then change it in this dialog box. In your case change the client name to exch. In the Host Properties/Client Properties/Universal Settings dialog box change the Use Specified Network Interface to the hostname or IP address of your backup subnet NIC. The other thing that you need to do for Exchange backups and restores to work properly is set the name of the mailbox for message level backups and restores. This is set in Host Properties/Client Properties/Windows Client/Exchange. You can also enable single instance backup for attachments here which reduces the amount of data you back up by storing a single instance of a message attachment and then for subsequent mailbox backups that contain that attachment set a pointer to the attachment. You'll have to have your local Exchange guru create this mailbox on the Exchange server if it doesn't exist. (I'm lucky, I work with a wicked smart Exchange/Mail/MS products guy who knew exactly what to do as soon as he saw this dialog box.). Another thing you can do when backing up mailboxes, and some people might think that I'm completely insane for doing this, is use a wildcard in your backup selections window on the policy and allow multiple streams. So in backup selections you might have an entry that looked like this: Microsoft Exchange Mailboxes:\* Doing this causes NetBackup to spawn a separate job for each mailbox on your Exchange server. I set this up after a couple of incidents where NetBackup was backing up the MS-Exchange mailboxes, hit some weirdness in one of the last mailboxes that needed to be backed up and decided to choke, puke and fall all over itself. This caused the entire backup to fail. Now if I have a mailbox with weirdness in it that NetBackup doesn't like that mailbox backup fails without affecting the backups of the other
[Veritas-bu] NetBackup 5.x to 6.x upgrade
Message: 2 Date: Thu, 13 Mar 2008 15:32:04 +0100 From: WALLEBROEK Bart [EMAIL PROTECTED] Subject: Re: [Veritas-bu] Lets hear about your upgrade experience! 5.x To: veritas-bu@mailman.eng.auburn.edu Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=us-ascii It would be much nicer to build the Sol 10 env with a fresh install of 6.5.1 and just import the 5.0 catalog. We are plannig to go to 6.5.1 the third quarter of this year. What we planned however is to place a brand new HP cluster next to the existing Master Server and install 6.5.1 on this new cluster. But then we are stuck with the conversion of the catalog. Isn't there a way (Symantec tool for example) to convert the catalog to the new db format and import it ton the new nbu server so not going through the normal upgrade ? Best Regards, Bart WALLEBROEK Backup admin Swift -- I upgraded to 6.0MP4 last year and upgraded my NetBackup master server from a V880 running Solaris 9 to a T2000 (which is just the cutest piece of hardware I've ever seen from Sun) running Solaris 10. To avoid bringing a lot of cruft over which had accumulated on the old backup Master over the years while getting the catalog upgraded to the new db format I did the following. 1) Ran the NetBackup utilities to check catalog consistency and worked with my VAR (DataLink) and Symantec to make sure that I was good to go for the upgrade. 2) Backed up my catalog (twice). 3) Stopped my NetBackup services on the old NetBackup master. 4) Created a new SAN volume, mounted it on the old NetBackup master and copied my entire /usr/openv/ hierarchy over to it. 5) Rebooted the old NetBackup master with the mount point for /usr/openv changed to point at the new /usr/openv duplicate volume. 6) Installed NetBackup 6.0 and ran the upgrade scripts to convert the catalog db to the new format. 7) Installed the NetBackup maintenance packs to bring the system up to MP5. 8) Tested to make sure that I could back up and restore files from each policy type from disk based storage units and tape based storage units. 9) Tested duplication to tape from disk based storage units and restore from those duplicated copies. 10) Reverted back to MP4 after finding out that Symantec broke NDMP backups in MP5 (how that MP ever got out of the lab escapes me). This was time consuming but gave me an easy fall back. If everything had failed I could have changed the mount point for /usr/openv to point back at the old mount point, rebooted my master server and I would have been back at status quo ante. Once I had 6.0MP4 up, running and tested on the old NetBackup master I let it run for a week to check performance and while it was doing so finalized my 6.0MP4 install on the new NetBackup master. When I did the transition to the new server I did the following. 1) Backed up the catalog, now converted to the new db format, on the old NetBackup master. 2) Shut the old NetBackup master down and changed the hostname and IP address of the new NetBackup master to that of the old NetBackup master. 3) Rebooted the new NetBackup master, checked that it came up with the correct hostname and IP address. 4) Started NetBackup and ran bprecover on my catalog backup tape. 5) Tested as above. I don't know if this was good practice, or if it was too laborious. I'm a suspenders and belt kind of guy when it comes to doing upgrades like this and since I'm long past my days of cowboy'ing it when I do upgrades like to have as many fallback options as reasonably possible. This did require some extra SAN storage for a few weeks while I tested things, and as I've mentioned was perhaps too laborious but other than that it did allow me to do a clean upgrade and gave me an easy fallback option at each step along the way. I don't know if this would work in your environment, mine is pretty small compared to a lot of others, but I hope it helps. Jamie Jamison Network Systems Administrator ZymoGenetics, Seattle ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] (no subject)
On Thu, Feb 21, 2008 at 10:09 AM, [EMAIL PROTECTED] wrote: I have a question. I am currently at 5.1 MP6 on Solaris. Does anyone have any recommendations on whether I should upgrade to 6.0 or 6.5.1? What are the pros cons for going directly to 6.5? Thanks, Chris. The pros of going to 6.5.1 directly from 5.1MP6 are that you're only doing one upgrade and that NetBackup 6.0 is garbage, the single worst version of the program I have ever worked with. NetBackup 6.0 is the Windows Vista of backup software. It has lots of features that sound really neat (hot catalog backups) but lots of those features don't work (hot catalog backups failing with type 41 errors, the pempersist problem, NDMP backup failures with 6.0MP5 and I'm sure we'll find more). Working with 6.0 makes me feel like an unpaid beta tester for Symantec, or worse, someone who was stupid enough to give Symantec money for the privilege of beta testing their software. If you go to any version of 6.0 plan on spending a lot of time on the phone with technical support and on installing super duper secret point patches from Symantec. An engineer with the VAR I work with said, the first time I called with in with a problem with 6.0 that 6.5.1 was incredibly solid and that it made the other versions of NetBackup look like ...they were made out of Tinker Toys by a bunch of retarded monkeys. I wish I had known this before I jumped to 6.0MP4 from 5.1MP5. Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] LTO2 restore problem on LTO4 tape drive
I'm having problems restoring a cumulative incremental standard policy backup stored on an LTO2 tape on an LTO4 drive. The density settings for the LTO2 tapes on my system is HCART, for LTO4 it's HCART3. I've run restores where I've set one of the LTO4 drives as density HCART in the device manager, restarted LTID and then created a storage unit for the LTO4 drive with the HCART density setting. I've also been able to run restores by setting the write protect tab on the LTO2 tape and then using vmchange to change the density to HCART3. I've been trying to restore from two tapes from a backup in November and I keep getting messages like this in the status window. 2/19/2008 12:59:46 PM - begin Restore 2/19/2008 12:59:48 PM - 1 images required 2/19/2008 12:59:48 PM - media K00443 required 2/19/2008 12:59:52 PM - restoring image spike.zgi.com_1195784616 2/19/2008 12:59:54 PM - connecting 2/19/2008 12:59:56 PM - connected; connect time: 00:00:02 2/19/2008 12:59:56 PM - requesting resource K00443 2/19/2008 12:59:56 PM - granted resource K00443 2/19/2008 12:59:56 PM - granted resource rmt-3cbn 2/19/2008 12:59:58 PM - Warning bptm(pid=546) media id K00443 not found in Media Manager, mount request will most likely occur 2/19/2008 12:59:58 PM - started process bptm (546) 2/19/2008 12:59:58 PM - mounting K00443 I don't know why I'm getting the media id not found in media manager error. The tape shows up if you do a bpmedialist or a vmquery. What I see is that the tape loads into the proper drive, does nothing for a while and then then NetBackup says that it is done with the tape (in this case K00443) and needs to load the next tape, which it identifies as K00443 (this is a one tape restore, I'm only interested in restoring the cumulative incremental backups from two days) and downs the tape drive. Does anyone have any suggestions on how to deal with this? Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Type 84 errors on NDMP backups on 6.0MP5
I upgraded to 6.0MP5 a few weeks ago to fix the notorious pempersist problem where NetBackup refused to run any of my scheduled policies or let me manually start backups. I had been running 6.0MP4 because MP5 had a bug that caused all of my NDMP backups to fail and to get things running I had to install MP5 and then install some super duper secret binaries for bptm and bpdbm. This fixed the problem with jobs not scheduling although I am still seeing the occasional type 41 error on my hot catalog backups, which is a symptom of the pempersist problem. This weekend I started getting type 84 errors on some of my NDMP backups with the error message: Error bptm(pid=13699) FREEZING media id XX, too many data blocks written, check tape/driver block size configuration error. Searching for this on Google produced the following web page: http://seer.entsupport.symantec.com/docs/295172.htm ETrack: 117380 Description: NDMP backup using TIR - positioning error - bptm does not advance expected_block_pos[TWIN_INDEX] if bytes_this_buf == 0 Has anyone else had any experience with this? I'm becoming increasingly frustrated with NetBackup 6.0. There are nice new features that I love, such as hot catalog backups and the ability to queue vault jobs, but for every feature I like there's a bug that I really hate, such as the NDMP problems in 6.0MP5, the pempersist problem in every 6.0 release and now this. It's especially annoying since I'm not using TIR in any of my NDMP policies. Indeed as far as I can tell it's not even an option for an NDMP policy type backup. Looking at the webpage listed above is depressing since the page was apparently last updated on the 23rd of January, 2008, yet contains this sentence This issue is currently being considered by Symantec Corporation to be addressed in a forthcoming Maintenance Pack or version of the product. The fix for this issue is expected to be released in the fourth quarter of 2007. I have this nightmare that I'm going to have to restore some crucial bit of corporate data and I'm not going to be able to. Then Symantec will post an eTrack notice saying Oh yeah, we found this bug in the version of NetBackup that you're running that causes it to expire all of your backup images, run 'rm -rf' on all of your disk based storage units, relabel all of the tapes in your library and then overwrite your catalog with zeros. Don't worry though, we're working on a fix that should be out at some date that's well in the past. Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Max Fragment Size for Disk and LTO4 Based Storage Units
So I just got a SpectraLogic T950 tape library, and after some initial teething pains it's up and running and it's wicked fast, but being an impatient little monkey of a NetBackup administrator I'm wondering if I could make it wicked faster. My primary backups go to DataDomain restorers and then are duplicated to tape for offsite storage with Iron Mountain. When I set all of this up I used 2048 megabytes as the maximum fragment size in both the disk based and tape based storage units. This was based on my predecessor's experience with what was the best trade-off between backup speed and file restoration speed with LTO Gen 2 drives. But I'm wondering if I can improve backup performance even more by increasing the fragment size for my disk and LTO4 based storage units without degrading my current file restoration performance. I thought I would send this to the list to find out what people were using as the frag sizes on LTO4 and on disk based storage units such as the DataDomains. Also if I change the fragment size on my tape based storage units how does NetBackup handle tapes that have backups written to them with different fragment sizes. Can this cause problems? Any feedback will be greatly appreciated. Thank You, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] NetBackup 6.0MP4 not expiring old backup images
I use DataDomain restorers as my primary backup system and then duplicate the images from the restorers to tape. The backup images residing on the restorers have a retention period of 1 month and I duplicate them to tape using vaulting policies that create secondary copies with either a two month or four month retention with the disk based image remaining the primary copy. For the last month I have noticed a problem with NetBackup not deleting images off of my restorers after they've expired. I can select a specific backup image and run bpimagelist against it and NetBackup tells where the files are and cheerfully informs me that the disk based image should have expired three weeks ago. When I check the files listing from bpimagelist I find that the disk based files still exist, even though they should be dead and gone. There is an option in vaulting to delete disk based images X number of hours after they are duplicated to tape and I could set this to 720 hours (1 month) and take care of the problem that way. However I shouldn't need to enable this and really don't want to. NetBackup knows that these images have a one month retention and should damn well be expiring them without me having to tell it to. When I started diving further into this problem I noticed that NetBackup was also not deleting some old tape based images either. I can get backup IDs out of the /usr/openv/netbackup/db/images directory and run bpimagelist against them and again, NetBackup cheerfully informs me that these backups should have expired weeks ago. Has anyone else had problems like this with NetBackup 6.0? Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Slow offline backup problem on NetBackup 6.0MP4 Solaris 10 resolved
I posted a query to the list a few weeks ago asking if anyone else had seen problems with offline catalog backups taking excessive amounts of time a Sun T2000 running Solaris 10 and with NetBackup 6.0MP4. Several people e-mailed me informing me that Veritas recommends using the hot catalog backup functionality instead of offline catalog backups, which I already knew and which was completely unhelpful as I was concerned about why an offline catalog backup that used to take about an hour and 15 minutes was now taking over five hours without any increase in the size of the catalog. Ignoring this sort of drastic change in a system's behavior is incredibly bad practice and I wanted to get to the bottom of this and not just assume that because I could run hot catalog backups that everything was going to be OK. I did get one very helpful e-mail from Don Klebba who had had a similar problem. Don found that disabling TCP fusion and changing some of his TCP parameters with ndd fixed the problem. Armed with the information that Don gave me (Thanks Don!) I found the following Symantec tech note. http://seer.entsupport.symantec.com/docs/284421.htm which explains the TCP fusion bug and how to fix it; you can either install the latest Sun patch bundle, which also fixes the T2000's problem with crashing when jumbo frames are enabled on the network interfaces, or disable TCP fusion with mdb and then edit your /etc/system file so that on the next reboot your system will no longer be using TCP fusion. If it hadn't been for Don's e-mail I wouldn't have found this because the tech note is about catalog backups failing with type 11 (a system call failed) errors. My catalog backups weren't failing, they had just become incredibly slow. I was able to take the downtime to install the patch bundle and my offline catalog backups went back to taking an hour and 15 minutes. Again I'd like to say thanks to Don Klebba and point out that when this sort of thing happens it's a good idea to figure out why it's happening and not just assume that because you have an alternative solution (in this case hot catalog backups) that things are going to be OK. I've seen some system administrators get bitten because they made that sort of assumption and end up having their careers suddenly interrupted with an RGE (resume generating event). ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Incredibly Slow Offline Catalog Backups In NetBackup 6.0MP4
Ever since I upgraded to NetBackup 6.0MP4 I've noticed that my offline catalog backups are incredibly and painfully slow. When I was running 5.1MP5 it took between 1 and 2 hours to back up my approximately 150Gb catalog. After the upgrade the offline catalog backups take well over four hours to complete, this despite having a faster server (Sun T2000 v. Sun V880) and faster HBAs. What gives? Is this a bug or a feature? Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] LTO Generation 4 tape throughput with on-drive encryption
I'm researching the purchase of a new library with LTO generation 4 tape drives and am interested in using the on-drive encryption to encrypt my backup tapes so that if a box of tapes ever falls off of the Iron Mountain truck I'm not having to explain things to the board of directors and legal, update my resume and/or both. The spec sheets for the LTO-4 drives that I've seen claim throughput of up to 120Mbps, but as we all know the devil is in the details and for all I know that throughput could have consisted of writing extremely large files consisting of nothing but the letter e to tape, without using encryption. Has anyone upgraded to LTO gen 4 yet who is also using the on-drive encryption and if so what kind of throughput do you see on average. Any real-world, real-life information will be greatly appreciated. Thank You, Jamie Jamison ZymoGenetics, Seattle ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Vaulting Question
I'm currently backing all of my systems up to DataDomain restorers with a one month retention period for the copy on the restorer and then duplicating from the restorer, using NetBackup Vault, to tape with a six month retention and sending the tapes offsite to Iron Mountain. Recently we decided to change the retention level for certain data from six months to two months and I am trying to figure out how to make this work with vaulting. If I create multiple profiles within one vault, one with a six month retention and another with a two month retention I run into the problem of scheduling, I can't run more than one profile at a time. I could create a new vault, but then I'm wondering how the eject and catalog backup steps work. I only want to eject tapes once a week and run one catalog backup for off-siting. Any help that anyone can provide on this will be greatly appreciated. Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Backing Up Network Drives in Win2003
Due to some problems with the NDMP implementation in our NAS head I'm forced (temporarily I hope) to back up NAS volumes by mounting them on a server and creating a policy that allows metwork drives to be backed up. On UNIX this works just fine, NFS mount the volumes, set up the policy and watch it run. However on Windows it's not so easy. The Windows guy I work with (who is very, very sharp) has mapped two of the NAS volumes to drive letters on a Win2003 host. I've tried backing this thing up and every time I do I get type 71 errors (none of the files in the list exist). I've tried backing up using both the mapped drive letters and the UNC path and every attempt has failed. A bpdir -M from the command line of my master server does not show the drive letters of the mapped network drives. I don't know if this is relevant or not. Any help will be greatly appreciated. Thank You, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Type 1 Error on Duplication Caused by Image Already Existing
I've seen this error before on duplication jobs when I was running tests and manually duplicated an existing image. But what would cause this to happen with images that were automatically duplicated? TCB1:duplicate.log.1_en failed batch dup: 03:12:51 INF - Duplicate of backup id patches.zgi.com_1169867358 failed, the entity already exists (226). Is this a catalog glitch I should worry about? Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] DataDomain ddr460 restorer storage unit setup practices
We are in the process of implementing DDR460 restorers in our NetBackup environment. Backups will go to a DDR460 restorer with a three week retention and from there will be duplicated to tape and also replicated to DDR460 restorers at a remote DR site. Right now I have one storage unit created on each of my DDR460s with the maximum concurrent job parameter set to 16 jobs. On my tape based storage units I have maximum concurrent drives used for backup set to 6 and maximum multiplexing per drive set to 6., so in theory things are roughly equivalent. I have a higher concurrent job number set on my disk storage units, based on DataDomain's recommendation to make up for the fact that we don't multiplex backups to the DDR460. Is anyone else using these and if so do you have any suggestions? So far the technology is working pretty well and now I'm ready to do a full-up test to see how well it handles our entire backup load. Thank You and Happy Holidays Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Queue v. Requeue
I'm running NetBackup 5.1MP5 on a Sun V880 server with 4CPUs and 8Gb of RAM. My tape storage is IBM Ultrium LTO Gen 2 drives on a fiber connected L180 library. Lately I'm seeing a lot of jobs requeue when I watch my backups kick off. Other jobs just queue up and sit there but some jobs queue and then requeue a few times before finally starting. What I see when I look at the status for the job is something like this: 12/13/2006 6:50:22 PM - mounting K00696 12/13/2006 6:50:22 PM - end writing 12/13/2006 6:50:23 PM - Info bpsched(pid=17014) Re-queueing job due to busy resources (134) 12/13/2006 6:50:32 PM - mounting K00696 12/13/2006 6:50:34 PM - end writing 12/13/2006 6:50:34 PM - Info bpsched(pid=17014) Re-queueing job due to busy resources (134) 12/13/2006 6:50:39 PM - mounting K00696 12/13/2006 6:50:40 PM - end writing 12/13/2006 6:50:40 PM - Info bpsched(pid=17014) Re-queueing job due to busy resources (134) This confuses me. I assume that NetBackup is actually trying to mount this tape and then finding that all of the drives are in use. Is there a communications breakdown here? Why isn't NetBackup just queueing these jobs until a drive and tape are available. I haven't seen any failures yet, but it concerns me and I don't understand why some jobs start and sit in the queue until a drive is ready and then start and others requeue. Thank You, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Type 24 (socket read failures) on NetBackup vault duplication jobs
I've had three duplication jobs fail out with type 1 errors in the last two days. When I look at the detailed status for each job I see the following: 10/30/2006 4:44:55 AM - positioned K01306; position time: 00:00:00 10/30/2006 6:44:56 AM - Error bpduplicate(pid=10559) host kira.zgi.com backup id maude_1162174269 read failed, socket read failed (23). 10/30/2006 6:44:58 AM - Error bpduplicate(pid=10559) Duplicate of backup id maude_1162174269 failed, socket read failed (23). 10/30/2006 6:45:01 AM - Error bpduplicate(pid=10559) host kira.zgi.com backup id maude_1162174271 read failed, socket read failed (23). 10/30/2006 6:47:57 AM - end Duplicate; elapsed time: 02:35:01 Does anyone have any idea what might be causing this? The Veritas Troubleshooter information is less than helpful for dealing with type 24 errorsIt's not an issue with that particular piece of media, I've seen similar errors on other duplication jobs using other tapes and it's not a drive issue and the image files do exist and there aren't any smoking guns in the bpdbm logs. I'd really like to stamp this out because I hate stuff like this so any help or info that anyone can provide will be greatly appreciated. Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Determining Number of Additional Client Licenses in NetBackup
I'm auditing my NetBackup environment to make sure that I have the number of licenses I need for clients and I'm finding the bpminlicense command somewhat unhelpful. When I run the command bpminlicense -list_keys -verbose I get output that looks like this file version= 0x0304 time added = 0x3c4bc801 Sun Jan 20 23:49:21 2002 hostname= kira product ID = 6 NetBackup Enterprise Server serial number = 3219 key version = 0x0304 count = 0 server platform = 0 Any platform client platform = 2 Any Windows platform server tier = 10 NetBackup Enterprise Server client tier = 0 No tier license type= 1 Permanent Site ID = 0 Any site Feature ID = 22 Additional clients + file version= 0x0304 time added = 0x3c4bc7ce Sun Jan 20 23:48:30 2002 hostname= kira product ID = 6 NetBackup Enterprise Server serial number = 4730 key version = 0x0304 count = 0 server platform = 0 Any platform client platform = 1 Any UNIX platform server tier = 10 NetBackup Enterprise Server client tier = 0 No tier license type= 1 Permanent Site ID = 0 Any site Feature ID = 22 Additional clients + How many additional client licenses do I have and how many of them come with Server? I'm assuming that 22 is a code for additional clients and not the number of clients. Any enlightenment will be greatly appreciated. Thank You, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] What Is the Command for Re-reading the bp.conf File Without Restarting the NetBackup Service
There is a command you can run that will re-read bp.conf and as long as certain changes have not been made, will dynamically update your running NetBackup configuration to match the new file. Can anyone tell me what this is? Thanks, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] NDMP restore problem - more information - NT Alternate Data Stream
OK, I looked at the logs for bptm and when I try to do an NDMP restore to my NetApp filer this is the kind of error I see: -BEGIN ERROR LOG-- 10:57:24.191 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 RESTORE: Fri Aug 4 10:57:24 2006 : We have read 2712480 KB from the backup. 11:02:24.193 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 RESTORE: Fri Aug 4 11:02:24 2006 : We have read 5422265 KB from the backup. 11:03:00.851 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 RESTORE: Cannot restore files with NT streams : /vol/volW5a/Users/XXX/X-RESTORED.mpg 11:03:00.851 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 RESTORE: No files were created. 11:03:07.936 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 RESTORE: RESTORE IS ABORTED 11:03:08.330 [14265] 2 data_halt_callback: data halted--internal error 11:03:08.331 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 Dump aborted. 11:03:08.331 [14265] 2 mover_halt_callback: mover halted--internal error 11:03:08.331 [14265] 4 apu.zgi.com: NDMP_LOG_NORMAL 0 MoveletInput: Internal Error. 11:03:08.331 [14265] 2 read_data_ndmp: restore done--notify_halted is 3 11:03:08.451 [14265] 4 ndmp_wait_for_msg: notify_mover_halted already arrived 11:03:08.860 [14265] 2 read_data: read_data_ndmp returned -98 11:03:08.860 [14265] 2 set_job_details: Done 11:03:08.862 [14265] 16 read_data: cannot restore from filepath /vol/volW5a, NDMP_HALT_INTERNAL_ERROR occurred -END ERROR LOG- My NetApp Filer is running Data OnTap 7.01RC. My NetBackup media/master server is a Sun V880 running Solaris 8 with NetBackup 5.1, MP4. I have checked files with the LADS (List Alternate Data Streams) tool and have been able to successfully restore files that do not contain NT Alternate Data Streams. Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] NDMP restore problem - NDMP_HALT_INTERNAL_ERROR Type 5 on Solaris Server with 5.1 MP4
I'm trying to restore a file from an NDMP volume and it fails with 'status 5 - NDMP_HALT_INTERNAL_ERROR'. When this happens nothing shows in the console of my NetApp filer and the message I get on the backup/restore console is WRN - Cannot restore from filepath /vol/volW5a, NDMP_HALT_INTERNAL_ERROR occurred. I've forced the restore from another drive and see the same error message. I am ordering back another tape with the same file on it to see if I can restore from it before I submit a ticket. Does anyone have any suggestions? Thank You, Jamie Jamison ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
