Hi,
Not sure if I get it. I want to have modular pig script that process data
file based on the metadata file. Not sure if thats possible.
John
On Mon, Dec 7, 2015 at 12:18 AM, Andrew Oliver <acoli...@gmail.com> wrote:
> I wrote a simple shell script to generate the individual pig scripts and a
> control script that executed them when it found a pig script and file to
> load... Worked okay.
> On Dec 6, 2015 5:47 PM, "John Smith" <lenov...@gmail.com> wrote:
>
> > Hi,
> >
> > I would like to ask you if there is any possibility to write pig flow
> using
> > control (metadata file).
> >
> > As a source I want to use one json (metadata) file and another csv file
> > with user data.
> >
> > Based on the description insde json file i want to process csv file
> >
> > For example:
> > JSON
> > ...
> > fields : [ {name: name, encryption: yes ... }, {...} .... ]}
> >
> > ...
> > CSV
> > name, age, insurance, ....
> > John S, 39, yes
> >
> >
> > Pig script reads json file and based on the file it processes CSV eg. if
> > there is flag for the first field "name" saying encryption: yes it will
> > somehow encrypt the value inside column Name (csv file).
> >
> >
> > Is it possible to control flow/ of pig script using another metadata
> file?
> >
> >
> > Thank you!
> >
>
