For 1.0 - complete Owl?
http://wiki.apache.org/pig/Metadata
Russell Jurney
rjur...@cloudstenography.com
On Jun 23, 2009, at 4:40 PM, Alan Gates wrote:
I don't believe there's a solid list of want to haves for 1.0. The
big issue I see is that there are too many interfaces that are still
shifting, such as:
1) Data input/output formats. The way we do slicing (that is, user
provided InputFormats) and the equivalent outputs aren't yet solid.
They are still too tied to load and store functions. We need to
break those out and understand how they will be expressed in the
language. Related to this is the semantics of how Pig interacts with
non-file based inputs and outputs. We have a suggestion of moving
to URLs, but we haven't finished test driving this to see if it will
really be what we want.
2) The memory model. While technically the choices we make on how
to represent things in memory are internal, the reality is that
these changes may affect the way we read and write tuples and bags,
which in turn may affect our load, store, eval, and filter functions.
3) SQL. We're working on introducing SQL soon, and it will take it
a few releases to be fully baked.
4) Much better error messages. In 0.2 our error messages made a
leap forward, but before we can claim to be 1.0 I think they need to
make 2 more leaps: 1) they need to be written in a way end users
can understand them instead of in a way engineers can understand
them, including having sufficient error documentation with suggested
courses of action, etc.; 2) they need to be much better at tying
errors back to where they happened in the script, right now if one
of the MR jobs associated with a Pig Latin script fails there is no
way to know what part of the script it is associated with.
There are probably others, but those are the ones I can think of off
the top of my head. The summary from my viewpoint is we still have
several 0.x releases before we're ready to consider 1.0. It would
be nice to be 1.0 not too long after Hadoop is, which still gives us
at least 6-9 months.
Alan.
On Jun 22, 2009, at 10:58 AM, Dmitriy Ryaboy wrote:
I know there was some discussion of making the types release (0.2)
a "Pig 1"
release, but that got nixed. There wasn't a similar discussion on
0.3.
Has the list of want-to-haves for Pig 1.0 been discussed since?