On Dec 6, 2010, at 10:40 AM, Chris Douglas wrote:
This question is backwards. If the assertion is that a part of the
framework's development should be arrested, that claim requires a
discussion and vote. The PMC should not have to weigh in on allowing
code to change. -C
Agreed.
Arresting development on SequenceFile is preposterous. There are
several petabytes of data sitting on it all over for several reasons,
including legacy. Stopping development on it is unreasonable. Apache
Hadoop is volunteer driven, volunteers should be allowed to contribute
as they see fit.
+1
Arun
On Mon, Dec 6, 2010 at 9:16 AM, Owen O'Malley <[email protected]>
wrote:
On Dec 1, 2010, at 11:11 AM, Owen O'Malley wrote:
All,
We really need some guidance on the general direction for the
project.
Please comment and/or vote. If no one cares, then I'll probably
commit it to
Yahoo's internal branch.
-- Owen
The question is how the Hadoop project wants to move forward.
It was motivated by Doug's veto of HADOOP-6685, which was based on
his
personal decisions about how the project should go forward and not
on
anything that had been decided by the PMC.
These decisions are much more important to MapReduce, which is a
framework, than HDFS which is a client/server model.
1. Should Hadoop include a user-facing library of useful code?
There has been a suggestion that user-facing library code, such as
SequenceFile, TFile, DistCp, etc. should be deprecated and that
Hadoop
should allow third party projects like Avro to supply the user-
facing
library code that makes Hadoop usable. I think it is critical that
we keep
those components as part of Hadoop and extend them as the
framework evolves.
Users depend heavily on SequenceFile for storing their data in
Hadoop and
they should not be deprecated as Doug has suggested.
2. Should MapReduce support non-Writables through the pipeline out
of the
box?
There has also been a discussion about whether we should support
non-Writables natively. There is already library code in Avro that
lets
users use Avro types in a custom MapReduce API. A general
MapReduce API that
encompasses all of the serialization frameworks and does not lock
users into
a particular one is much more powerful.
Furthermore, making it convenient for the users, by including the
plugins
in the default configuration and class path, will enable the use
of Avro,
Thrift and ProtoBuf objects by people who would rather not focus on
serialization. Avro and Writables should not be the only first class
serializations that Hadoop supports by default.
3. Should a framework dependency on ProtoBuf be allowed?
Doug has added several framework dependences on Avro. The question
is
whether it is acceptable to use the ProtoBuf library in the
framework. Avro
is good for uses where there are a lot of objects of the same
type. ProtoBuf
is better for small number of objects. The question is whether
Avro, JSON,
and XML should be the only serialization libraries that are
acceptable to
use in the framework.
