Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Arun C Murthy
Please take questions on Cloudera Distro to their internal lists. On Aug 5, 2010, at 3:52 PM, Bobby Dennett wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your tho

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Josh Patterson
Bobby, We're working hard to make compression easier, the biggest hurdle currently is the licensing issues around the LZO codec libs (GPL, which is not compatible with ASF bsd-style license). Outside of making the changes to the mapred-site.xml file, with your setup would do you view as the bigge

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Bobby Dennett
Hi Josh, No real pain points... just trying to investigate/research the "best" way to create the necessary libraries and jar files to support LZO compression in Hadoop. In particular, there are the 2 "repositories" to build from and I am trying to find out if one should be used over the other. For

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-05 Thread Todd Lipcon
On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett wrote: > Hi Josh, > > No real pain points... just trying to investigate/research the "best" > way to create the necessary libraries and jar files to support LZO > compression in Hadoop. In particular, there are the 2 "repositories" > to build from and

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-07 Thread Alex Luya
Does it(hadoop-lzo) only work for hadoop 0.20,not work for 0.21 or 0.22? On Friday, August 06, 2010 09:05:47 am Todd Lipcon wrote: > On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett wrote: > > Hi Josh, > > > > No real pain points... just trying to investigate/research the "best" > > way to create t

Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1

2010-08-08 Thread Todd Lipcon
On Sat, Aug 7, 2010 at 9:18 PM, Alex Luya wrote: > Does it(hadoop-lzo) only work for hadoop 0.20,not work for 0.21 or 0.22? > I don't know that anyone has tested it against 0.21 or trunk, but I don't see any reasons it won't work just fine -- the APIs are pretty stable between 0.20 and above.