Re: changes to compression interfaces in 0.15?

2008-02-21 Thread Ted Dunning
The principles are pretty simple: If the semantics change significantly, then the name should change. Conversely, if the name doesn't change the semantics shouldn't change. That is, unless the changes fix seriously broken old semantics or extend old semantics in a way that old calls don't change

Re: changes to compression interfaces in 0.15?

2008-02-21 Thread Pete Wyckoff
ooking at the >> INCOMPATIBLE section (which btw - i did carefully before sending my >> mail). >> >> (fortunately i escaped the worst case - but i think this is a real >> call to action) >> >> >> -Original Message- >> From: Arun C Murthy [mailt

Re: changes to compression interfaces in 0.15?

2008-02-21 Thread Arun C Murthy
ail). (fortunately i escaped the worst case - but i think this is a real call to action) -Original Message- From: Arun C Murthy [mailto:[EMAIL PROTECTED] Sent: Thu 2/21/2008 11:21 AM To: core-user@hadoop.apache.org Subject: Re: changes to compression interfaces in 0.15? Joydeep,

RE: changes to compression interfaces in 0.15?

2008-02-21 Thread Joydeep Sen Sarma
worst case - but i think this is a real call to action) -Original Message- From: Arun C Murthy [mailto:[EMAIL PROTECTED] Sent: Thu 2/21/2008 11:21 AM To: core-user@hadoop.apache.org Subject: Re: changes to compression interfaces in 0.15? Joydeep, On Feb 20, 2008, at 5:06 PM, Joydeep

Re: changes to compression interfaces in 0.15?

2008-02-21 Thread Arun C Murthy
Joydeep, On Feb 20, 2008, at 5:06 PM, Joydeep Sen Sarma wrote: Hi developers, In migrating to 0.15 - i am noticing that the compression interfaces have changed: - compression type for sequencefile outputs used to be set by: SequenceFile.setCompressionType() - now it seem

RE: changes to compression interfaces in 0.15?

2008-02-20 Thread Joydeep Sen Sarma
adoop.apache.org Subject: Re: changes to compression interfaces in 0.15? I filed a JIRA for this issue: https://issues.apache.org/jira/browse/HADOOP-2866 - Aaron Aaron Kimball wrote: > +1 > > Ted Dunning wrote: >> Actually, it might just be good to have a warning spit out if you use

Re: changes to compression interfaces in 0.15?

2008-02-20 Thread Aaron Kimball
I filed a JIRA for this issue: https://issues.apache.org/jira/browse/HADOOP-2866 - Aaron Aaron Kimball wrote: +1 Ted Dunning wrote: Actually, it might just be good to have a warning spit out if you use ANY unknown key that starts with mapred.* or any of the other hadoop-specific parameters.

Re: changes to compression interfaces in 0.15?

2008-02-20 Thread Aaron Kimball
+1 Ted Dunning wrote: Actually, it might just be good to have a warning spit out if you use ANY unknown key that starts with mapred.* or any of the other hadoop-specific parameters. That way mis-spellings would be caught as well as deprecations. If you want to set a value and not get a warning

Re: changes to compression interfaces in 0.15?

2008-02-20 Thread Ted Dunning
Actually, it might just be good to have a warning spit out if you use ANY unknown key that starts with mapred.* or any of the other hadoop-specific parameters. That way mis-spellings would be caught as well as deprecations. If you want to set a value and not get a warning, just pick a different

RE: changes to compression interfaces in 0.15?

2008-02-20 Thread Joydeep Sen Sarma
long. But still .. -Original Message- From: Jason Venner [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 20, 2008 6:13 PM To: core-user@hadoop.apache.org Subject: Re: changes to compression interfaces in 0.15? I agree. I am in the midst of combing through the config files for 16 to see

Re: changes to compression interfaces in 0.15?

2008-02-20 Thread Jason Venner
I agree. I am in the midst of combing through the config files for 16 to see what changes i have to retrofit into our jobs. Support in the tools to inform of the use of depreciated or outright removed keys would be wonderful. Aaron Kimball wrote: As a general follow-up suggestion : Is there a m

Re: changes to compression interfaces in 0.15?

2008-02-20 Thread Aaron Kimball
As a general follow-up suggestion : Is there a mechanism to output a warning when the user sets deprecated JobConf keys? Given that you can set any arbitrary key name and it will simply be ignored, this might be a good idea. - Aaron Joydeep Sen Sarma wrote: In addition: - "mapred.o

RE: changes to compression interfaces in 0.15?

2008-02-20 Thread Joydeep Sen Sarma
In addition: - "mapred.output.compression.type" is now replaced with "mapred.map.output.compression.type" - the old implementation of the Java interface setMapOutputCompressorClass() used to turn on map compression on automatically as side-effect, the 0.15 one doesn't. Looks lik