Agree...And thats the point...The connector code needs to catch up with spark release train; if its part of Geode then the Geode releases needs to happen as often as Spark release (along with other planned Geode release)...
Even if the connector code is compatible with latest Spark; the previous connector release/version still has to be maintained as end users are still working with older versions (most of the enterprise customers). I like to get a closure on this topic/thread; how do we do that... Thanks, -Anil. On Tue, Jul 7, 2015 at 9:55 AM, Anthony Baker <aba...@pivotal.io> wrote: > Given the rate of change, it doesn’t seem like we should be trying to add > (and maintain) support for every single Spark release. We’re early in the > lifecycle of the Spark connector and too much emphasis on > backwards-compatibility will be a drag on our ongoing development, > particularly since the Spark community is valuing rapid evolution over > stability. > > (apologies if I have misconstrued the state of Spark) > > Anthony > > > > On Jul 6, 2015, at 11:22 PM, Qihong Chen <qc...@pivotal.io> wrote: > > > > The problem is caused by multiple major dependencies and different > release > > cycles. Spark Geode Connector depends on two products: Spark and Geode > (not > > counting other dependencies), and Spark moves much faster than Geode, and > > some features/code are not backward compatible. > > > > Our initial connector implementation depends on Spark 1.2 in before the > > last week of March 15. Then Spark 1.3 was released on the last week of > > March, and some connector feature doesn't work with Spark 1.3, then we > > moved on, and now support Spark 1.3 (but not 1.2 any more, we did create > > tag). Two weeks ago, Spark 1.4 was released, and it breaks our connector > > code again. > > > > Therefore, for each Geode release, we probably need multiple Connector > > releases, and probably need to maintain last 2 or 3 Connector releases, > for > > example, we need to support both Spark 1.3 and 1.4 with the current Geode > > code. > > > > The question is how to support this with single source repository? > > > > Thanks, > > Qihong > >
