Then I would point the same direction as Bill: override relativeToAbsolutePath That's more or less what HCatalog does: You use a loader/storer that abstracts out where the data is. and based on the requested location you point to the right files. Out of curiosity: Why does your framework need to parse first and then modify the plan ? Are you trying to cache some intermediary results? Are you trying to change the logical or the physical plan? Julien
On Tue, Feb 12, 2013 at 12:24 AM, Prashant Kommireddi <[email protected]>wrote: > Thanks Bill. I think the issue would be with the fact that I am trying to > inject the property (base path) from an external framework into > Load/StoreFunc. An analogy could be a fancy random generator that resides > in a framework has to be used to setLocation. However, "random generator > logic" can not be placed within a Load/StoreFunc. So a user that uses, say > CustomPigStorage passes his script to the framework and the framework now > needs to alter the i/o locations. > > > 1. User submits script -> Framework.getService().submitPigScript(File > script); > 2. Script is using a Load/StoreFunc > 3. Framework has to parse the script to determine the user provided > locations? > > I understand when you say it could be done dynamically within Funcs, but I > guess I am not very clear on how we could alter i/o locations from outside > of Pig. > Pardon me if I am not explaining it well enough! > > Julien, this is not for testing or for injecting data. > > > On Mon, Feb 11, 2013 at 7:47 PM, Bill Graham <[email protected]> wrote: > >> You can do it dynamically in the StoreFunc/LoadFunc, but not before the LP >> is built (since the LP really isn't yet involved with the physical >> location). Why would you need to change the location that early in the >> process? >> >> >> On Mon, Feb 11, 2013 at 6:30 PM, Julien Le Dem <[email protected]> >> wrote: >> >> > Is it for testing or something totally different? >> > you can also look at o.a.p.builtin.mock.Storage for injecting data >> > Julien >> > >> > >> > On Mon, Feb 11, 2013 at 6:17 PM, Prashant Kommireddi < >> [email protected]>wrote: >> > >> >> That would work in the case when it's a static path? The problem here >> is >> >> my >> >> base path will vary entirely for each different Pig job, and I would >> not >> >> like the client to set that. Rather my framework plugs-in the base path >> >> and >> >> I am guessing that can happen only after parsing/buildLP? >> >> >> >> On Mon, Feb 11, 2013 at 5:55 PM, Bill Graham <[email protected]> >> >> wrote: >> >> >> >> > We've done this before by overriding relativeToAbsolutePath >> >> > and setLocation in the LoadFunc, or the correspoding methods in >> >> StoreFunc. >> >> > >> >> > On Mon, Feb 11, 2013 at 2:10 PM, Prashant Kommireddi < >> >> [email protected] >> >> > >wrote: >> >> > >> >> > > Hey, >> >> > > >> >> > > I wanted to run an idea by you guys. I have a use-case where I try >> >> > > injecting load/store paths into the script. So if a user says A = >> load >> >> > > 'input'; I would like to add a base path to it and make it A = load >> >> > > 'base_path/foo/bar/input'. >> >> > > >> >> > > I would like to achieve this programatically, and 1 way I can of >> >> doing it >> >> > > is by allowing setter on LOLoad/LOStore that can modify the >> FileSpec. >> >> Is >> >> > > there a cleaner or better way to approach this? >> >> > > >> >> > > In the future I would probably be meddling with more than just >> >> Load/Store >> >> > > (for eg, disabling rmf or mkdir commands etc) but this is >> something I >> >> am >> >> > > looking at currently. >> >> > > >> >> > > Thanks, >> >> > > Prashant >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > *Note that I'm no longer using my Yahoo! email address. Please email >> me >> >> at >> >> > [email protected] going forward.* >> >> > >> >> >> > >> > >> >> >> -- >> *Note that I'm no longer using my Yahoo! email address. Please email me at >> [email protected] going forward.* >> > >
