To be clear, going to 1.0 is not about having a certain set of
features. It is about stability and usability. When a project
declares itself 1.0 it is making some guarantees regarding the
stability of its interfaces (in Pig's case this is Pig Latin, UDFs,
and command line usage). It is also declaring itself ready for the
world at large, not just the brave and the free. New features can
come in as experimental once we're 1.0, but the semantics of the
language and UDFs can't be shifting (as we've done the last several
releases and will continue to do for a bit I think).
With that in mind, further comments inlined.
On Jun 24, 2009, at 10:18 AM, Dmitriy Ryaboy wrote:
Alan, any thoughts on performance baselines and benchmarks?
Meaning do we need to reach a certain speed before 1.0? I don't think
so. Pig is fast enough now that many people find it useful. We want
to continue working to shrink the gap between Pig and MR, but I don't
see this as a blocker for 1.0.
I am a little surprised that you think SQL is a requirement for 1.0,
since
it's essentially an overlay, not core functionality.
If we were debating today whether to go 1.0, I agree that we would not
wait for SQL. But given that we aren't (at least I wouldn't vote for
it now) and that SQL will be in soon, it will need to stabilize.
What about the storage layer rewrite (or is that what you referred
to with
your first bullet-point)?
To be clear, the Zebra (columnar store stuff) is not a rewrite of the
storage layer. It is an additional storage option we want to
support. We aren't changing current support for load and store.
Also, the subject of making more (or all) operators nestable within a
foreach comes up now and then.. would you consider this important
for 1.0,
or something that can wait?
This would be an added feature, not a semantic change in Pig Latin.
Integration with other languages (a-la PyPig)?
Again, this is a new feature, not a stability issue.
The Roadmap on the Wiki is still "as of Q3 2007".... makes it hard
for an
outside contributor to know where to jump :-).
Agreed. Olga has given me the task of updating this soon. I'm going
to try to get to that over the next couple of weeks. This discussion
will certainly provide input to that update.
Alan.
