To support different versions of spark, wouldn't it be better to have a single code base that has adapters for different versions of spark? It seems like that would be better than maintaining several active branches with semi-duplicate code.
I do think it would be better to keep the geode spark connector in a separate repository with a separate release cycle, for all of the reasons outlined on this thread (don't bloat the geode codebase, modularity, etc.). But I think there is also value in keeping it in the apache community and managing it through the apache process. I'm not sure how "just put it on github" would work out. Maybe it's just a matter of making it through the pain of the restrictive incubation process until we can split this code out. And in the mean time keeping it as loosely coupled as possible. -Dan On Tue, Jul 7, 2015 at 11:57 AM, John Blum <[email protected]> wrote: > +1 - Bingo, that tis the question. > > Part of the answer lies in having planned, predictable and a consistent > cadence of releases. > > E.g. the *Spring Data* project <http://projects.spring.io/spring-data/> > [0] > is an umbrella project managing 12 individual modules (e.g. SD... JPA, > Mongo, Redis, Neo4j, GemFire, Cassandra, etc, dubbed the "release train") > which all are at different versions and all have different external, > critical (driver) dependencies. The only dependen(cy|cies) all SD modules > have in common is the version of the core *Spring Framework* and the > version of Spring* Data Commons*. Otherwise individual modules upgrade > their "driver" dependencies at different cycles, possibly in different > "release train", but only when the current release train is released > (~every 4 weeks). See SD Wiki > < > https://github.com/spring-projects/spring-data-commons/wiki/Release-planning > > > [1] > for more details. > > [0] - http://projects.spring.io/spring-data/ > [1] - > > https://github.com/spring-projects/spring-data-commons/wiki/Release-planning > > > On Tue, Jul 7, 2015 at 11:21 AM, Gregory Chase <[email protected]> wrote: > > > More important than easy to develop is easy to pick up and use. > > > > Improving the new user experience is something that needs attention from > > Geode. How we develop and provide Spark integration needs to take this > > into account. > > > > Once we are able to provide official releases, how can a user know and > make > > sure they are getting the correct plug-in version, and have relatively up > > to date support for latest Geode and Spark versions? > > > > That to me is the requirement we should be designing for first in our > > development process. > > > > On Tue, Jul 7, 2015 at 10:47 AM, Roman Shaposhnik <[email protected]> > > wrote: > > > > > On Tue, Jul 7, 2015 at 10:34 AM, Anilkumar Gingade < > [email protected]> > > > wrote: > > > > Agree...And thats the point...The connector code needs to catch up > with > > > > spark release train; if its part of Geode then the Geode releases > needs > > > to > > > > happen as often as Spark release (along with other planned Geode > > > release)... > > > > > > I don't think this is a realistic goal to have that many actively > > > supported branches > > > of Geode Spark connector. > > > > > > Look, I've been around Hadoop ecosystem for years. Nowhere the problem > of > > > integration with upstream is as present as in Hadoop ecosystem > > > (everything depends > > > on everything else and everything evolves like crazy). I haven't seen a > > > single > > > project in that ecosystem that would be able to support a blanket > > statement > > > like the above. May be Geode has resources that guys depending on > > something > > > like HBase simply don't have. > > > > > > Thanks, > > > Roman. > > > > > > > > > > > -- > > Greg Chase > > > > Director of Big Data Communities > > http://www.pivotal.io/big-data > > > > Pivotal Software > > http://www.pivotal.io/ > > > > 650-215-0477 > > @GregChase > > Blog: http://geekmarketing.biz/ > > > > > > -- > -John > 503-504-8657 > john.blum10101 (skype) >
