> To maintain backward compat, we cannot remove old apis - the standard
> procedure is to deprecate them for the next release and remove them
> in subsequent releases.
you've got to be kidding.
we didn't maintain backwards compatibility. my app broke. Simple and
straightforward. and the old interfaces are not deprecated (to quote 0.15.3 on
a 'deprecated' interface:
/**
* Set the compression type for sequence files.
* @param job the configuration to modify
* @param val the new compression type (none, block, record)
*/
static public void setCompressionType(Configuration job,
CompressionType val) {
)
I (and i would suspect any average user willing to recompile code) would much
much rather that we broke backwards compatibility immediately rather than
maintain carry over defunct apis that insidiously break application behavior.
and of course - this does not address the point that the option strings
themselves are depcreated. (remember - people set options explicitly from xml
files and streaming. not everyone goes through java apis)).
--
as one of my dear professors once said - put ur self in the other person's
shoe. consider that u were in my position and that a production app suddenly
went from consuming 100G to 1TB. and everything slowed down drastically. and it
did not give any sign that anything was amiss. everything looked golden on the
ourside. what would be ur reaction if u find out after a week that the system
was full and numerous processes had to be re-run? how would you have figured
that was going to happen by looking at the INCOMPATIBLE section (which btw - i
did carefully before sending my mail).
(fortunately i escaped the worst case - but i think this is a real call to
action)
-----Original Message-----
From: Arun C Murthy [mailto:[EMAIL PROTECTED]
Sent: Thu 2/21/2008 11:21 AM
To: core-user@hadoop.apache.org
Subject: Re: changes to compression interfaces in 0.15?
Joydeep,
On Feb 20, 2008, at 5:06 PM, Joydeep Sen Sarma wrote:
> Hi developers,
>
> In migrating to 0.15 - i am noticing that the compression interfaces
> have changed:
>
> - compression type for sequencefile outputs used to be set
> by:
> SequenceFile.setCompressionType()
>
> - now it seems to be set using:
> sequenceFileOutputFormat.setOutputCompressionType()
>
>
Yes, we added SequenceFileOutputFormat.setOutputCompressionType and
deprecated the old api. (HADOOP-1851)
>
> The change is for the better - but would it be possible to:
>
> - remove old/dead interfaces. That would have been a
> straightforward hint for applications to look for new interfaces.
> (hadoop-default.xml also still has setting for old conf variable:
> io.seqfile.compression.type)
>
To maintain backward compat, we cannot remove old apis - the standard
procedure is to deprecate them for the next release and remove them
in subsequent releases.
> - if possible - document changed interfaces in the release
> notes (there's no way we can find this out by looking at the long list
> of Jiras).
>
Please look at the INCOMPATIBLE CHANGES section of CHANGES.txt,
HADOOP-1851 is listed there. Admittedly we can do better, but that is
a good place to look for when upgrading to newer releases.
>
> i am not sure how updated the wiki is on the compression stuff (my
> responsibility to update it) - but please do consider the impact of
Please use the forrest-based docs (on the hadoop website - e.g.
mapred_tutorial.html) rather than the wiki as the gold-standard. The
reason we moved away from the wiki is precisely this - harder to
maintain docs per release etc.
> changing interfaces on existing applications. (maybe we should have a
> JIRA tag to mark out bugs that change interfaces).
>
>
Again, CHANGES.txt and INCOMPATIBLE CHANGES section for now.
Arun
>
>
> As always - thanks for all the fish (err .. working code),
>
>
>
> Joydeep
>
>
>
