I agree that API refactoring should be done in smaller steps. This could
be done in multiple releases also. Maybe for this release we could have
some goal like just making some splits eg API and impl. For the rest we
can continue on them also but they should not hold back the release.
Also +1 to the idea of having support of different Apache projects like
solr/hcatalog/Cassandra etc. But again we can have some specific ones
that we will make in this sprint.
I also want to know when do we want to make the next release ? Will it
be linked to the things that we want to fix or do we have a set
time-frame ? I feel that the next release should not take much long.
On 15-09-2012 15:25, Matthias Friedrich wrote:
Hi,
On Friday, 2012-09-14, Josh Wills wrote:
I like the idea of having themes for releases. In my head, the theme of
this release could be either
a) Hacking the new MSCRPlanner code, esp. to add the ability to fuse
different MSCR jobs into a single instance that it enables, or
b) data access/integration points-- things like solr, hcatalog, hbase,
cassandra, jdbc, etc. as input and output sources for Crunch pipelines, or
c) API refactoring-- the crunch-api/crunch-impl/crunch-lib split, or
I would see c) as part of a larger mission for improving documentation
and usability. An immediate benefit would be that we don't have to
provide javadoc for each and every class, only for those packages that
are client-facing. Higher perceived quality with less work for us.
I wouldn't make it a separate release though, perhaps we can do this
in a series of smaller steps, starting with the crunch-api split.
Refactorings like this usually turn into long frustrating monster
tasks that prevent other progress. I'd really like to avoid that.
But, before spending any more time on this, I think we should all
agree that this is what we really want. Somehow I got the impression
that you're not fully convinced that this refactoring is necessary or
even a good idea. To me it would feel like people trying to rearrange
the furniture in my living room. Let's discuss this here before we
produce any more patches.
d) working on a PStream API that would let people apply DoFns to streams
and would build on top of things like WalMart's mupd8 or Storm or whatever.
Of course, this is in addition to whatever fixes and new lib functions we
want to add over time. I don't want anything heavyweight, but those are
some of the larger-scale things that we'll need to tackle as a community,
and I would think of completing each of those big things as corresponding
to a release.
Sounds good to me.
Regards,
Matthias
