Query on Using Pig standalone basis

2010-08-26 Thread Somdip
Hi All, Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea? How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management i

Query on Using Pig standalone basis

2010-08-26 Thread Somdip
Hi All, Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea? How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management i

Re: Query on Using Pig standalone basis

2010-08-26 Thread Jeff Zhang
It is not recommended using Pig standalone mode to process big files if you concern the performance. I believe perl's performance will be better than pig in local mode. On Thu, Aug 26, 2010 at 5:42 PM, Somdip wrote: > Hi All, > > Just wanted to know has anyone used Pig on a standalone basis in

Re: Query on Using Pig standalone basis

2010-08-26 Thread Ravi Phulari
Somdip, I will be surprised to know if anyone is using Pig on a standalone basis in production environment . Pig uses Hadoop's power to process huge data. HDFS for storing large data and MapReduce for using distributed system for processing huge data. You can use Pig in standalone mode for learn