Thanks Bill. I think the issue would be with the fact that I am trying to inject the property (base path) from an external framework into Load/StoreFunc. An analogy could be a fancy random generator that resides in a framework has to be used to setLocation. However, "random generator logic" can not be placed within a Load/StoreFunc. So a user that uses, say CustomPigStorage passes his script to the framework and the framework now needs to alter the i/o locations.
1. User submits script -> Framework.getService().submitPigScript(File script); 2. Script is using a Load/StoreFunc 3. Framework has to parse the script to determine the user provided locations? I understand when you say it could be done dynamically within Funcs, but I guess I am not very clear on how we could alter i/o locations from outside of Pig. Pardon me if I am not explaining it well enough! Julien, this is not for testing or for injecting data. On Mon, Feb 11, 2013 at 7:47 PM, Bill Graham <[email protected]> wrote: > You can do it dynamically in the StoreFunc/LoadFunc, but not before the LP > is built (since the LP really isn't yet involved with the physical > location). Why would you need to change the location that early in the > process? > > > On Mon, Feb 11, 2013 at 6:30 PM, Julien Le Dem <[email protected]> wrote: > > > Is it for testing or something totally different? > > you can also look at o.a.p.builtin.mock.Storage for injecting data > > Julien > > > > > > On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi < > [email protected]>wrote: > > > >> That would work in the case when it's a static path? The problem here is > >> my > >> base path will vary entirely for each different Pig job, and I would not > >> like the client to set that. Rather my framework plugs-in the base path > >> and > >> I am guessing that can happen only after parsing/buildLP? > >> > >> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[email protected]> > >> wrote: > >> > >> > We've done this before by overriding relativeToAbsolutePath > >> > and setLocation in the LoadFunc, or the correspoding methods in > >> StoreFunc. > >> > > >> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi < > >> [email protected] > >> > >wrote: > >> > > >> > > Hey, > >> > > > >> > > I wanted to run an idea by you guys. I have a use-case where I try > >> > > injecting load/store paths into the script. So if a user says A = > load > >> > > 'input'; I would like to add a base path to it and make it A = load > >> > > 'base_path/foo/bar/input'. > >> > > > >> > > I would like to achieve this programatically, and 1 way I can of > >> doing it > >> > > is by allowing setter on LOLoad/LOStore that can modify the > FileSpec. > >> Is > >> > > there a cleaner or better way to approach this? > >> > > > >> > > In the future I would probably be meddling with more than just > >> Load/Store > >> > > (for eg, disabling rmf or mkdir commands etc) but this is something > I > >> am > >> > > looking at currently. > >> > > > >> > > Thanks, > >> > > Prashant > >> > > > >> > > >> > > >> > > >> > -- > >> > *Note that I'm no longer using my Yahoo! email address. Please email > me > >> at > >> > [email protected] going forward.* > >> > > >> > > > > > > > -- > *Note that I'm no longer using my Yahoo! email address. Please email me at > [email protected] going forward.* >
