>>What I am suggesting is that we need to maintain backward compatibility with a defined set of 1.x version clients when Drill 2.0 version is out.
I'm asking you to be concrete on why. There is definitely a cost to maintaining this compatibility. What are the real costs if we don't? -- Jacques Nadeau CTO and Co-Founder, Dremio On Wed, Apr 6, 2016 at 9:21 AM, Neeraja Rentachintala < [email protected]> wrote: > Jacques > can you elaborate on what you mean by 'internal' implementation changes but > maintain external API. > I thought that changes that are being discussed here are the Drill client > library changes. > What I am suggesting is that we need to maintain backward compatibility > with a defined set of 1.x version clients when Drill 2.0 version is out. > > Neeraja > > On Tue, Apr 5, 2016 at 12:12 PM, Jacques Nadeau <[email protected]> > wrote: > > > Thanks for bringing this up. BI compatibility is super important. > > > > The discussions here are primarily about internal implementation changes > as > > opposed to external API changes. From a BI perspective, I think (hope) > > everyone shares the goal of having zero (to minimal) changes in terms of > > ODBC and JDBC behaviors in v2. The items outlined in DRILL-4417 are also > > critical to strong BI adoption as numerous patterns right now are > > suboptimal and we need to get them improved. > > > > In terms of your request of the community, it makes sense to have a > > strategy around this. It sounds like you have a bunch of considerations > > that should be weighed but your presentation doesn't actually share what > > the concrete details. To date, there has been no formal consensus or > > commitment to any particular compatibility behavior. We've had an > informal > > "don't change wire compatibility within a major version". If we are going > > to have a rich dialog about pros and cons of different approaches, we > need > > to make sure that everybody has the same understanding of the dynamics. > For > > example: > > > > Are you saying that someone has packaged the Apache Drill drivers in > their > > BI solution? If so, what version? Is this the Apache release artifact or > a > > custom version? Has someone certified them? Did anyone commit a > particular > > compatibility pattern to a BI vendor on behalf of the community? > > > > To date, I'm not aware of any of these types of decisions being discussed > > in the community so it is hard to evaluate how important they are versus > > other things. Knowing that DRILL-4417 is outstanding and critical to the > > best BI experience, I think we should be very cautious of requiring > > long-term support of the existing (internal) implementation. Guaranteeing > > ODBC and JDBC behaviors should be satisfactory for the vast majority of > > situations. Anything beyond this needs to have a very public cost/benefit > > tradeoff. In other words, please expose your thinking 100x more so that > we > > can all understand the ramifications of different strategies. > > > > thanks! > > Jacques > > > > > > > > > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Tue, Apr 5, 2016 at 10:01 AM, Neeraja Rentachintala < > > [email protected]> wrote: > > > > > Sorry for coming back to this thread late. > > > I have some feedback on the compatibility aspects of 2.0. > > > > > > We are working with a variety of BI vendors to certify Drill and > provide > > > native connectors for Drill. Having native access from BI tools helps > > with > > > seamless experience for the users with performance and functionality. > > This > > > work is in progress and they are (and will be) working with 1.x > versions > > of > > > Drill as part of the development because thats what we have now. Some > of > > > these connectors will be available before 2.0 and some of them can come > > in > > > post 2.0 as certification is a long process. We don't want to be in a > > > situation where the native connectors are just released by certain BI > > > vendor and the connector is immediately obsolete or doesn't work > because > > we > > > have 2.0 release out now. > > > So the general requirement should be that we maintain backward > > > compatibility with certain number of prior releases. This is very > > important > > > for the success of the project and adoption by eco system. I am happy > to > > > discuss further. > > > > > > -Neeraja > > > > > > On Tue, Apr 5, 2016 at 8:44 AM, Jacques Nadeau <[email protected]> > > wrote: > > > > > > > I'm going to take this as lazy consensus. I'll create the branch. > > > > > > > > Once created, all merges to the master (1.x branch) should also go to > > the > > > > v2 branch unless we have a discussion here that they aren't > applicable. > > > > When committing, please make sure to commit to both locations. > > > > > > > > thanks, > > > > Jacques > > > > > > > > > > > > -- > > > > Jacques Nadeau > > > > CTO and Co-Founder, Dremio > > > > > > > > On Sat, Mar 26, 2016 at 7:26 PM, Jacques Nadeau <[email protected]> > > > > wrote: > > > > > > > > > Re Compatibility: > > > > > > > > > > I actually don't even think 1.0 clients work with 1.6 server, do > > they? > > > > > > > > > > I would probably decrease the cross-compatibility requirement > > burden. A > > > > > nice goal would be cross compatibility across an extended series of > > > > > releases. However, given all the things we've learned in the last > > year, > > > > we > > > > > shouldn't try to maintain more legacy than is necessary. As such, I > > > > propose > > > > > that we consider the requirement of 2.0 to be: > > > > > > > > > > 1.lastX works with 2.firstX. (For example, if 1.8 is the last minor > > > > > release of the 1.x series, 1.8 would work with 2.0.) > > > > > > > > > > This simplifies testing (we don't have to worry about things like > > does > > > > 1.1 > > > > > work with 2.3, etc) and gives people an upgrade path as they > desire. > > > This > > > > > also allows us to decide what pieces of the compatibility shim go > in > > > the > > > > > 2.0 server versus the 1.lastX client. (I actually lean towards > > > allowing a > > > > > full break between v1 and v2 server/client but understand that that > > > level > > > > > or coordination is hard in many organizations since analysts are > > > separate > > > > > from IT). Hopefully, what I'm proposing can be a good compromise > > > between > > > > > progress and deployment ease. > > > > > > > > > > Thoughts? > > > > > > > > > > Re: Branches/Dangers > > > > > > > > > > Good points on this Julian. > > > > > > > > > > How about this: > > > > > > > > > > - small fixes and enhancements PRs should be made against v1 > > > > > - new feature PRs should be made against v2 > > > > > - v2 should continue to always pass all precommit tests during its > > life > > > > > - v2 becomes master in two months > > > > > > > > > > I definitely don't want to create instability in the v2 branch. > > > > > > > > > > The other option I see is we can only do bug fix releases and > branch > > > the > > > > > current master into a maintenance branch and treat master as v2. > > > > > > > > > > Other ideas? > > > > > > > > > > > > > > > -- > > > > > Jacques Nadeau > > > > > CTO and Co-Founder, Dremio > > > > > > > > > > On Sat, Mar 26, 2016 at 6:07 PM, Julian Hyde <[email protected]> > > wrote: > > > > > > > > > >> Do you plan to be doing significant development on both the v1 and > > v2 > > > > >> branches, and if so, for how long? I have been bitten badly by > that > > > > pattern > > > > >> in the past. Developers put lots of unrelated, destabilizing > changes > > > > into > > > > >> v2, it look longer than expected to stabilize v2, product > management > > > > lost > > > > >> confidence in v2 and shifted resources back to v1, and v2 never > > caught > > > > up > > > > >> with v1. > > > > >> > > > > >> One important question: Which branch will you ask people to target > > for > > > > >> pull requests? v1, v2 or both? If they submit to v2, and v2 is > > broken, > > > > how > > > > >> will you know whether the patches are good? > > > > >> > > > > >> My recommendation is to choose one of the following: (1) put a > > strict > > > > >> time limit of say 2 months after which v2 would become the master > > > branch > > > > >> (and v1 master would become a maintenance branch), or (2) make v2 > > > > focused > > > > >> on a particular architectural feature; create multiple independent > > > > feature > > > > >> branches with breaking API changes if you need to. > > > > >> > > > > >> Julian > > > > >> > > > > >> > > > > >> > On Mar 26, 2016, at 1:41 PM, Paul Rogers <[email protected]> > > > > wrote: > > > > >> > > > > > >> > Hi All, > > > > >> > > > > > >> > 2.0 is a good opportunity to enhance our ZK information. See > > > > >> DRILL-4543: Advertise Drill-bit ports, status, capabilities in > > > > ZooKeeper. > > > > >> This change will simplify YARN integration. > > > > >> > > > > > >> > This enhancement will change the “public API” in ZK. To Parth’s > > > point, > > > > >> we can do so in a way that old clients work - as long as a > Drill-bit > > > > uses > > > > >> default ports. > > > > >> > > > > > >> > I’ve marked this JIRA as a candidate for 2.0. > > > > >> > > > > > >> > Thanks, > > > > >> > > > > > >> > - Paul > > > > >> > > > > > >> >> On Mar 24, 2016, at 4:11 PM, Parth Chandra <[email protected]> > > > > wrote: > > > > >> >> > > > > >> >> What's our proposal for backward compatibility between 1.x and > > 2.x? > > > > >> >> My thoughts: > > > > >> >> Optional - Allow a mixture of 1.x and 2.x drillbits in a > > cluster. > > > > >> >> Required - 1.x clients should be able to talk to 2.x drillbits. > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau < > > > [email protected]> > > > > >> wrote: > > > > >> >> > > > > >> >>> There are some changes that either have reviews pending or are > > in > > > > >> progress > > > > >> >>> that would require breaking changes to Drill core. > > > > >> >>> > > > > >> >>> Examples Include: > > > > >> >>> DRILL-4455 (arrow integration) > > > > >> >>> DRILL-4417 (jdbc/odbc/rpc changes) > > > > >> >>> DRILL-4534 (improve null performance) > > > > >> >>> > > > > >> >>> I've created a new 2.0.0 release version in JIRA and moved > these > > > > >> tasks to > > > > >> >>> that umbrella. > > > > >> >>> > > > > >> >>> I'd like to propose a new v2 release branch where we can start > > > > >> >>> incorporating these changes without disrupting v1 stability > and > > > > >> >>> compatibility. > > > > >> >>> > > > > >> >>> > > > > >> >>> -- > > > > >> >>> Jacques Nadeau > > > > >> >>> CTO and Co-Founder, Dremio > > > > >> >>> > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > >
