On Oct 23, 2008, at 11:02 AM, Dhruba Borthakur wrote:
Hi Sanjay,
Ok, let me understand the terminology.
-- Versions 1.0 and versions 2.0 are "major" releases, maybe occur
once every year
-- Versions 1.2 and 1.3 are minor releases, maybe occurs once
every 2 months
-- Versions 2.4.5 and 2.4.6 are "dot" release.
Is my understanding correct?
Yes.
>don't you want disk format compatibility to be for both major and
minor release changes as I stated earlier?
Disk formats can change for both major and minor releases but not for
dot release. And the hadoop code should handle the required disk
format upgrades and downgrades transparently. This issue, by itself,
should have no impact on pre-existing FileSystem APIs.
Yes.
How about the following stronger requirement on disk formats:
- disk formats should be forward AND backward compatible for minor
releases (ie 1.2 to 1.3)
this can be done though the versioning techniques used by protocol
buffers, thrift, hessian
No roll back is needed, olver version can ignore the fields
they don't know about.
- disk formats can be changed in a non-backward compatible way for
major releases (eg. 1.9 to 2.) but that a roll back is possible.
So for example adding fields to the fsImage or edits log records is
forward and backward compatible and can be done on any minor release.
But the changes like we did when we added CRCs could only be done on
major releases.
I think this stronger requirement will make Hadoop admin's life a lot
easier.
sanjay
Does this make sense?
thanks,
dhruba
On Wed, Oct 22, 2008 at 9:04 AM, Sanjay Radia <[EMAIL PROTECTED]>
wrote:
>
>
>
> On Oct 20, 2008, at 11:44 PM, Dhruba Borthakur wrote:
>>
>> 3. HDFS disk format can change from 1.x to 1.y release and is
>> transparent to user-application. A cluster when rolling back to 1.x
>> from 1,y will revert back to the old disk format.
>
> Dhruba don't you want disk format compatibility to be for both
major and
> minor release changes as I stated earlier?
> I don't think it is acceptable to break data format across major
releases.
>
>
> Q. do we want disk upgrade to be supported across arbitrary number
of
> releases? For example we allowed the crc upgrade for
> approx 4 releases in the past.
>
>
>
