Integration with Owl is something we want for 1.0. I am hopeful that
by Pig's 1.0 Owl will have flown the coop and become either a
subproject or found a home in Hadoop's common, since it will hopefully
be used by multiple other subprojects.
Alan.
On Jun 23, 2009, at 11:42 PM, Russell Jurney wrote:
For 1.0 - complete Owl?
http://wiki.apache.org/pig/Metadata
Russell Jurney
rjur...@cloudstenography.com
On Jun 23, 2009, at 4:40 PM, Alan Gates wrote:
I don't believe there's a solid list of want to haves for 1.0. The
big issue I see is that there are too many interfaces that are
still shifting, such as:
1) Data input/output formats. The way we do slicing (that is, user
provided InputFormats) and the equivalent outputs aren't yet
solid. They are still too tied to load and store functions. We
need to break those out and understand how they will be expressed
in the language. Related to this is the semantics of how Pig
interacts with non-file based inputs and outputs. We have a
suggestion of moving to URLs, but we haven't finished test driving
this to see if it will really be what we want.
2) The memory model. While technically the choices we make on how
to represent things in memory are internal, the reality is that
these changes may affect the way we read and write tuples and bags,
which in turn may affect our load, store, eval, and filter functions.
3) SQL. We're working on introducing SQL soon, and it will take it
a few releases to be fully baked.
4) Much better error messages. In 0.2 our error messages made a
leap forward, but before we can claim to be 1.0 I think they need
to make 2 more leaps: 1) they need to be written in a way end
users can understand them instead of in a way engineers can
understand them, including having sufficient error documentation
with suggested courses of action, etc.; 2) they need to be much
better at tying errors back to where they happened in the script,
right now if one of the MR jobs associated with a Pig Latin script
fails there is no way to know what part of the script it is
associated with.
There are probably others, but those are the ones I can think of
off the top of my head. The summary from my viewpoint is we still
have several 0.x releases before we're ready to consider 1.0. It
would be nice to be 1.0 not too long after Hadoop is, which still
gives us at least 6-9 months.
Alan.
On Jun 22, 2009, at 10:58 AM, Dmitriy Ryaboy wrote:
I know there was some discussion of making the types release (0.2)
a "Pig 1"
release, but that got nixed. There wasn't a similar discussion on
0.3.
Has the list of want-to-haves for Pig 1.0 been discussed since?
