Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-08 Thread Todd Lipcon
On Sat, Aug 7, 2010 at 9:18 PM, Alex Luya alexander.l...@gmail.com wrote: Does it(hadoop-lzo) only work for hadoop 0.20,not work for 0.21 or 0.22? I don't know that anyone has tested it against 0.21 or trunk, but I don't see any reasons it won't work just fine -- the APIs are pretty stable

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-07 Thread Alex Luya
Does it(hadoop-lzo) only work for hadoop 0.20,not work for 0.21 or 0.22? On Friday, August 06, 2010 09:05:47 am Todd Lipcon wrote: On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett bdenn...@gmail.com wrote: Hi Josh, No real pain points... just trying to investigate/research the best way to

Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Bobby Dennett
We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular,

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Arun C Murthy
Please take questions on Cloudera Distro to their internal lists. On Aug 5, 2010, at 3:52 PM, Bobby Dennett wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Josh Patterson
Bobby, We're working hard to make compression easier, the biggest hurdle currently is the licensing issues around the LZO codec libs (GPL, which is not compatible with ASF bsd-style license). Outside of making the changes to the mapred-site.xml file, with your setup would do you view as the

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Bobby Dennett
Hi Josh, No real pain points... just trying to investigate/research the best way to create the necessary libraries and jar files to support LZO compression in Hadoop. In particular, there are the 2 repositories to build from and I am trying to find out if one should be used over the other. For

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Todd Lipcon
On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett bdenn...@gmail.com wrote: Hi Josh, No real pain points... just trying to investigate/research the best way to create the necessary libraries and jar files to support LZO compression in Hadoop. In particular, there are the 2 repositories to build