Re: changes to compression interfaces in 0.15?

[Aaron Kimball]

+1

Ted Dunning wrote:

Actually, it might just be good to have a warning spit out if you use ANY
unknown key that starts with mapred.* or any of the other hadoop-specific
parameters.

That way mis-spellings would be caught as well as deprecations.

If you want to set a value and not get a warning, just pick a different
prefix.

On 2/20/08 6:13 PM, "Jason Venner" <[EMAIL PROTECTED]> wrote:

I agree. I am in the midst of combing through the config files for 16 to
see what changes i have to retrofit into our jobs.
Support in the tools to inform of the use of depreciated or outright
removed keys would be wonderful.

Aaron Kimball wrote:

As a general follow-up suggestion : Is there a mechanism to output a
warning when the user sets deprecated JobConf keys? Given that you can
set any arbitrary key name and it will simply be ignored, this might
be a good idea.

- Aaron

Joydeep Sen Sarma wrote:

In addition:

-          "mapred.output.compression.type" is now replaced with
"mapred.map.output.compression.type"

-          the old implementation of the Java interface
setMapOutputCompressorClass() used to turn on map compression on
automatically as side-effect, the 0.15 one doesn't. Looks like one has
to call setCompressMapOutput() separately.

Aargh.

________________________________

From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Joydeep Sen
Sarma
Sent: Wednesday, February 20, 2008 5:06 PM
To: core-user@hadoop.apache.org
Subject: changes to compression interfaces in 0.15?

Hi developers,

In migrating to 0.15 - i am noticing that the compression interfaces
have changed:

-          compression type for sequencefile outputs used to be set by:
SequenceFile.setCompressionType()

-          now it seems to be set using:
sequenceFileOutputFormat.setOutputCompressionType()

The change is for the better - but would it be possible to:

-          remove old/dead interfaces. That would have been a
straightforward hint for applications to look for new interfaces.
(hadoop-default.xml also still has setting for old conf variable:
io.seqfile.compression.type)

-          if possible - document changed interfaces in the release
notes (there's no way we can find this out by looking at the long list
of Jiras).

As u can imagine - this causes a very subtle and harmful regression in
behavior of existing apps. It does not causes failures - and in our case
- switched from BLOCK to RECORD compression - meaning - there's no
compression at all pretty much. I caught this by *pure* chance and now I
am living in absolute fear of what else lurks out there.

i am not sure how updated the wiki is on the compression stuff (my
responsibility to update it) - but please do consider the impact of
changing interfaces on existing applications. (maybe we should have a
JIRA tag to mark out bugs that change interfaces).

As always - thanks for all the fish (err .. working code),

Joydeep