Hey Ricky,
You can just NFS mount the files to all resource manager nodes like you mentioned or you can use cas-pge's filestaging support to stage the files locally using the filemgr... see <fileStagine> element in this file:
it will stage all the files in the given metadata fields to the given dir and will update the metadata fields provided with the new location of the files. For example in that given pge-config.xml file if:
InputFiles = /path/to/product.one,/path/to/product.two
And you have the following configuration (note: force just says that if the files is already staged there, stage it again):
<fileStaging dir="/my/pge/workspace/input" force="true">
<stageFiles metadataKey="InputFiles" />
</fileStaging>Then the files would be staged to /my/pge/workspace/input and InputFiles would now be:
InputFiles = /my/pge/workspace/input/product.one,/my/pge/workspace/input/product.two
-brian
On Jul 20, 2012, at 09:35 AM, "Nguyen, Ricky" <[email protected]> wrote:
Hi folks,
I was thinking of using Resource Manager to execute my PGE tasks on various worker nodes (batch stubs) in hopes of improving the run time of my workflows. So my first question is, how will a batch stub obtain files from the File Manager? I'm only familiar with the local use case: PGE (without resmgr) will just use "$FileLocation/$Filename" returned by filemgr to access the file locally. My guess is that all the nodes should mount the filemgr archive dir at the same NFS location (as in https://cwiki.apache.org/OODT/getting-products-from-a-remote-filemanager.html). Or does the PushPull component have a role here?
Thanks,
Ricky
---------------------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain confidential
or legally privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please
contact the sender by reply e-mail and destroy all copies of this original message.
---------------------------------------------------------------------
