On Jan 21, 2008, at 3:17 PM, Adam Bazinet wrote:
I'm basically going ahead with your suggestion and plan on using RLS/DRS to achieve the kind of file caching that I want. However, I'm struggling with how I should use these tools to meet our needs based solely on the documentation and the limited amount I've been able to play around with them.
I'm hoping to get Rob Schuler to chime in with some suggestions, as I'm not a particularly active RLS/DRS user myself.
As a prerequisite to replicating files with DRS, do they already need to have LFN entries in an RLS somewhere? In other words, once the RLS is bootstrapped with at least one LFN, perhaps it is possible to use DRS to replicate it. But it is also possible to have DRS transfer files/make entries in an RLS de novo, from scratch?
I don't know, but I suspect that the assumption is that there is no algorithmic way to convert your PFN into an LFN. The LFN conceptually holds metadata regarding the file. It's an interesting idea, though, that you could provide some command/script that you could run against a file and extract an LFN. That would work if there's enough metadata in the file itself to regenerate the LFN, but I think it was probably developed under the assumption that that's not possible.
So, one simple question is: do I need an RLS installation on each remote resource, or can I get away with the single RLS installation on the Grid server keeping track of the locations of files on a modest (~10) number of these resources?
I believe you can get away with a single RLS server, and can add more later if you find that you need them.
The main reason I'm interested in this is because we are running into the situation where we are staging files in needed by jobs (or job batches) over and over again to the same resource, and sometimes duplicate files get staged unnecessarily, wasting bandwidth and disk space. Here's a more technical question, though. What if two different jobs need the same logical file, but need it named two different things?
This is a place where you may find yourself writing a little wrapper around the RLS/DRS functionality. I think you could use DRS to get the first copy/name at a location. Subsequently if you search the catalog and find that there's an LFN->PFN mapping that suits you in terms of host but not in filename you could add a symlink and make another entry in the catalog to show that LFN->PFN map. Part of the difficulty here is the co-scheduling of storage resources and compute resources. If you're using DRS, you have to get the file there before your job hits the queue, while if you use GRAM's staging you don't get the benefits of DRS. So it's not clear to me whether you're better off with DRS + a small script, or RLS + GRAM + a larger script, where the larger script does some of the things DRS would have done for you.
Either way, I suspect you're going to want the client to run some RLS/ DRS queries/transfers before the job is submitted at all.
Charles
