Re: [Samba] WinNT offline file attribut
Illtud Daniel wrote: > "John E. Malmberg" wrote: > >> On Thu, 25 Apr 2002, Illtud Daniel wrote: > > >> The question that none of the Windows based HSM vendors would give >> me an answer on was: Is there any way to make sure that a copy of >> all files shelved and unshelved exists on the storage robot, and >> how do I restore things when the real disk fails. I would think >> that question should be easy to answer. > > Now I'm confused. It may be because that before coming to this > thread my HSM terminology was different to yours. I use 'migrated' > for when a file is written to tape (or optical, whatever). 'fetched' > for getting it back and writing it to the stub file, 'purged' for > removing the file from the extended volume and replacing it with a > stub file. You purge only migrated files (for obvious reasons), and > a file open on the stub will trigger a fetch. Ok, your terms are more precise than mine. > To answer your questions with regard to OTG DX2000: How do you know > that a copy of all files shelved exists on the robot? Assuming you > mean 'shelved' to be 'migrated & purged' and 'unshelved' to mean > 'migrated but not purged', then you don't know, you trust. You can > run tape reports to list what's on each tape, but bugs > notwithstanding, if a file's been migrated, then it should be on the > tapes. DX can backup its internal database (stub file -> tape > location) to file which you can stick somewhere safe. In event of > distaster, you can rebuild the stubs by just restoring from this file. > You can read about this on OTG's website: > http://www.otg.com/KnowledgeBase/default.htm try 'dxdrivedump.exe' - > that knowledge base will give you a lot of info on how DX does stuff. Yes, I would want to make sure that all files exist on a tape after they have have been in existance for a period of time. HSM software should be doing disk space management in the background, not necessarily shelving and purging files on demand. HSM directories are by nature usually read only archives, so the HSM software should be able to migrate but not purge in the background so that if it needs free space in a hurry, it does not have to do a migrate and purge to get it. >> >> I am sure that there is interest. Because of the caching issue, >> unless you have enough files so that they all do not fit on the >> disk, you may not notice a performance problem. > > :) We'll force them to tape, don't worry! The disk is 1.44TB, so yes, > we'll have a job filling it, but you can set policies to ensure that > migrating starts when your disk reaches 0.01% full, or to migrate as > soon as a file appears on the volume, and to force a purge (stubify) > of all files. Yes you can force a purge, but if the HSM software is smart, it will do anything it can to avoid going to the robot to retreive a file, so even if it claims to have replaced a file with a stub, the file may still be resident on the disk. So when a file is opened, the HSM software just swaps the stub with the actual file that is still on the disk becasue the migration process did not really delete the file. So unless you cause the HSM software to overwrite the files that had been "purged", you will not see the performance impact. Virtual memory works the same way. When my process is over it's quota, some pages get moved from my address space and put on the free or modified lists to be used by other processes. If my process accesses those pages, if the page is still on the free list or the modified list, I get the same page back instead of having to go to the disk and wait for it. > By the way, we're moving to ADICs AMASS because we've had problems > with DX2000 which have never been resolved (including loss of > hundreds of gigs of content, which isn't fun). An open HSM solution > would be a godsend. Unfortunately the number of users that need an HSM system is small, and with the size of hard drives shrinking fast, getting smaller. The cost of the robot hardware is the gating factor. Now using a gzip archive instead of a robot, may make it more economically feasable. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
Re: [Samba] WinNT offline file attribut
"John E. Malmberg" wrote: > > On Thu, 25 Apr 2002, Illtud Daniel wrote: [] > > but I'm fairly sure that the > > mac file sharing services certainly aren't aware of any offline > > attributes. > > There are two ways to handle HSM, swap part of the file or swap all of the > file. OTG DX2000 swaps the whole lot, leaving an (empty) stub file. > This would make Windows files happy. I am not sure about the Mac's, as > they may expect a different file format, or the resource fork may be a > factor. The resource fork is handled by the Mac file sharing application, which may put it a seperate file/directory (like CAP, netatalk, SFM) or in a different file stream (like MacServerIP can optionally do - v useful, in that it keeps everything tidy, big headache in that *nobody* handles NTFS streams properly, not MS, not OTG, not nobody I've found). Keeping them in seperate files/directories mean that you can set your shelving policy not to touch these (tiny) files, which takes away some mac fileserving headaches. > The question that none of the Windows based HSM vendors would give me an > answer on was: Is there any way to make sure that a copy of all files > shelved and unshelved exists on the storage robot, and how do I restore > things when the real disk fails. I would think that question should be > easy to answer. Now I'm confused. It may be because that before coming to this thread my HSM terminology was different to yours. I use 'migrated' for when a file is written to tape (or optical, whatever). 'fetched' for getting it back and writing it to the stub file, 'purged' for removing the file from the extended volume and replacing it with a stub file. You purge only migrated files (for obvious reasons), and a file open on the stub will trigger a fetch. To answer your questions with regard to OTG DX2000: How do you know that a copy of all files shelved exists on the robot? Assuming you mean 'shelved' to be 'migrated & purged' and 'unshelved' to mean 'migrated but not purged', then you don't know, you trust. You can run tape reports to list what's on each tape, but bugs notwithstanding, if a file's been migrated, then it should be on the tapes. DX can backup its internal database (stub file -> tape location) to file which you can stick somewhere safe. In event of distaster, you can rebuild the stubs by just restoring from this file. You can read about this on OTG's website: http://www.otg.com/KnowledgeBase/default.htm try 'dxdrivedump.exe' - that knowledge base will give you a lot of info on how DX does stuff. > > I'll be trying samba (and netatalk) on a HSM'd volume on Monday, so > > I should be able to report back, if there's interest. > > I am sure that there is interest. Because of the caching issue, unless > you have enough files so that they all do not fit on the disk, you may not > notice a performance problem. :) We'll force them to tape, don't worry! The disk is 1.44TB, so yes, we'll have a job filling it, but you can set policies to ensure that migrating starts when your disk reaches 0.01% full, or to migrate as soon as a file appears on the volume, and to force a purge (stubify) of all files. By the way, we're moving to ADICs AMASS because we've had problems with DX2000 which have never been resolved (including loss of hundreds of gigs of content, which isn't fun). An open HSM solution would be a godsend. -- Illtud Daniel [EMAIL PROTECTED] Uwch Ddadansoddwr Systemau Senior Systems Analyst Llyfrgell Genedlaethol Cymru National Library of Wales Yn siarad drosof fy hun, nid LlGC - Speaking personally, not for NLW -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
Re: [Samba] WinNT offline file attribut
On Thu, 25 Apr 2002, Illtud Daniel wrote: > "John E. Malmberg" wrote: > > > Does Windows NT/2000 do any tests when serving shelved files, or does it > > just assume that the client will do the right thing? > > I have OTG's DX2000 HSM software running on NT. We can serve > HSM'd directories with the normal windows sharing to PCs > or SFM/MacServerIP for Mac clients. There are some issues > with the Macs tending to want to access the icon information > (thus fetching the files), but I'm fairly sure that the > mac file sharing services certainly aren't aware of any offline > attributes. There are two ways to handle HSM, swap part of the file or swap all of the file. If the HSM software leaves the part of the file most frequently accessed on the disk, then Windows clients may be able to access the portion of the file for it's icon and be happy. This requires that the HSM software either statistically know what parts of the file are frequently accessed or to make assumptions. Eitehr way is overhead. This would make Windows files happy. I am not sure about the Mac's, as they may expect a different file format, or the resource fork may be a factor. > > Really? I hope not! We've got ADIC engineers coming in on Monday > to install AMASS (HSM on Solaris) on a Solaris 8 box here, and > we've indicated to them that we'll be using samba and netatalk > to serve the shelved directory to PCs & Macs, and they haven't > objected. I'll call my ADIC tech contact and see if he can > clarify. How HSM's work is proprietary, and I have never been sucessful in getting information as to their algorithms, but it appears that they leave only a stub file on the real disk. How much is left is the real question. For the non-PC uses that I am familiar with, there does not seem to be much reason to leave any part of the file on the disk once it is shelved. So that means with out considering a product like SAMBA, a UNIX based HSM program would not need to contain the algorithms to just swap part of a file. Also if the data that has been shelved has not been overwritten on the disk, and the file is requested to be unshelved, I would expect that the HSM software would know to just use the data. The question that none of the Windows based HSM vendors would give me an answer on was: Is there any way to make sure that a copy of all files shelved and unshelved exists on the storage robot, and how do I restore things when the real disk fails. I would think that question should be easy to answer. It should be possible to get close to a current restore by putting in a new disk, and letting the HSM restore the files as needed with a minimal number of commands. > I'll be trying samba (and netatalk) on a HSM'd volume on Monday, so > I should be able to report back, if there's interest. I am sure that there is interest. Because of the caching issue, unless you have enough files so that they all do not fit on the disk, you may not notice a performance problem. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
