I agree with Paul. Great points. I would also add the partners aspect to it. Majority of Drill users use it in conjunction with a BI tool.
-Neeraja On Tue, Apr 12, 2016 at 3:34 PM, Paul Rogers <[email protected]> wrote: > Hi Jacques, > > My two cents… > > The unfortunate reality is that enterprise customers move slowly. There is > a delay in the time it takes for end users to upgrade to a new release. > When a third-party tool must also upgrade, the delay becomes even longer. > > At a high level, we need to provide a window of time in which old/new > clients work with old/new servers. I may have a 1.6 client. The cluster > upgrades to 1.8. I need time to upgrade my client to 1.8 — especially if I > have to wait for the vendor to provide a new package. > > If I connect to two clusters, I may upgrade my client to 1.8 for one, but > I still need to connect to 1.6 for the other if they upgrade on different > schedules. > > This is exactly why we need to figure out a policy: how do we give users a > sufficient window of time to complete upgrades, even across the 1.x/2.x > boundary? > > The cost of not providing such a window? Broken production systems, > unpleasant escalations and unhappy customers. > > Thanks, > > - Paul > > > On Apr 12, 2016, at 3:14 PM, Jacques Nadeau <[email protected]> wrote: > > > >>> What I am suggesting is that we need to maintain backward > compatibility with > > a defined set of 1.x version clients when Drill 2.0 version is out. > > > > I'm asking you to be concrete on why. There is definitely a cost to > > maintaining this compatibility. What are the real costs if we don't? > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Wed, Apr 6, 2016 at 9:21 AM, Neeraja Rentachintala < > > [email protected]> wrote: > > > >> Jacques > >> can you elaborate on what you mean by 'internal' implementation changes > but > >> maintain external API. > >> I thought that changes that are being discussed here are the Drill > client > >> library changes. > >> What I am suggesting is that we need to maintain backward compatibility > >> with a defined set of 1.x version clients when Drill 2.0 version is out. > >> > >> Neeraja > >> > >> On Tue, Apr 5, 2016 at 12:12 PM, Jacques Nadeau <[email protected]> > >> wrote: > >> > >>> Thanks for bringing this up. BI compatibility is super important. > >>> > >>> The discussions here are primarily about internal implementation > changes > >> as > >>> opposed to external API changes. From a BI perspective, I think (hope) > >>> everyone shares the goal of having zero (to minimal) changes in terms > of > >>> ODBC and JDBC behaviors in v2. The items outlined in DRILL-4417 are > also > >>> critical to strong BI adoption as numerous patterns right now are > >>> suboptimal and we need to get them improved. > >>> > >>> In terms of your request of the community, it makes sense to have a > >>> strategy around this. It sounds like you have a bunch of considerations > >>> that should be weighed but your presentation doesn't actually share > what > >>> the concrete details. To date, there has been no formal consensus or > >>> commitment to any particular compatibility behavior. We've had an > >> informal > >>> "don't change wire compatibility within a major version". If we are > going > >>> to have a rich dialog about pros and cons of different approaches, we > >> need > >>> to make sure that everybody has the same understanding of the dynamics. > >> For > >>> example: > >>> > >>> Are you saying that someone has packaged the Apache Drill drivers in > >> their > >>> BI solution? If so, what version? Is this the Apache release artifact > or > >> a > >>> custom version? Has someone certified them? Did anyone commit a > >> particular > >>> compatibility pattern to a BI vendor on behalf of the community? > >>> > >>> To date, I'm not aware of any of these types of decisions being > discussed > >>> in the community so it is hard to evaluate how important they are > versus > >>> other things. Knowing that DRILL-4417 is outstanding and critical to > the > >>> best BI experience, I think we should be very cautious of requiring > >>> long-term support of the existing (internal) implementation. > Guaranteeing > >>> ODBC and JDBC behaviors should be satisfactory for the vast majority of > >>> situations. Anything beyond this needs to have a very public > cost/benefit > >>> tradeoff. In other words, please expose your thinking 100x more so that > >> we > >>> can all understand the ramifications of different strategies. > >>> > >>> thanks! > >>> Jacques > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Jacques Nadeau > >>> CTO and Co-Founder, Dremio > >>> > >>> On Tue, Apr 5, 2016 at 10:01 AM, Neeraja Rentachintala < > >>> [email protected]> wrote: > >>> > >>>> Sorry for coming back to this thread late. > >>>> I have some feedback on the compatibility aspects of 2.0. > >>>> > >>>> We are working with a variety of BI vendors to certify Drill and > >> provide > >>>> native connectors for Drill. Having native access from BI tools helps > >>> with > >>>> seamless experience for the users with performance and functionality. > >>> This > >>>> work is in progress and they are (and will be) working with 1.x > >> versions > >>> of > >>>> Drill as part of the development because thats what we have now. Some > >> of > >>>> these connectors will be available before 2.0 and some of them can > come > >>> in > >>>> post 2.0 as certification is a long process. We don't want to be in a > >>>> situation where the native connectors are just released by certain BI > >>>> vendor and the connector is immediately obsolete or doesn't work > >> because > >>> we > >>>> have 2.0 release out now. > >>>> So the general requirement should be that we maintain backward > >>>> compatibility with certain number of prior releases. This is very > >>> important > >>>> for the success of the project and adoption by eco system. I am happy > >> to > >>>> discuss further. > >>>> > >>>> -Neeraja > >>>> > >>>> On Tue, Apr 5, 2016 at 8:44 AM, Jacques Nadeau <[email protected]> > >>> wrote: > >>>> > >>>>> I'm going to take this as lazy consensus. I'll create the branch. > >>>>> > >>>>> Once created, all merges to the master (1.x branch) should also go to > >>> the > >>>>> v2 branch unless we have a discussion here that they aren't > >> applicable. > >>>>> When committing, please make sure to commit to both locations. > >>>>> > >>>>> thanks, > >>>>> Jacques > >>>>> > >>>>> > >>>>> -- > >>>>> Jacques Nadeau > >>>>> CTO and Co-Founder, Dremio > >>>>> > >>>>> On Sat, Mar 26, 2016 at 7:26 PM, Jacques Nadeau <[email protected]> > >>>>> wrote: > >>>>> > >>>>>> Re Compatibility: > >>>>>> > >>>>>> I actually don't even think 1.0 clients work with 1.6 server, do > >>> they? > >>>>>> > >>>>>> I would probably decrease the cross-compatibility requirement > >>> burden. A > >>>>>> nice goal would be cross compatibility across an extended series of > >>>>>> releases. However, given all the things we've learned in the last > >>> year, > >>>>> we > >>>>>> shouldn't try to maintain more legacy than is necessary. As such, I > >>>>> propose > >>>>>> that we consider the requirement of 2.0 to be: > >>>>>> > >>>>>> 1.lastX works with 2.firstX. (For example, if 1.8 is the last minor > >>>>>> release of the 1.x series, 1.8 would work with 2.0.) > >>>>>> > >>>>>> This simplifies testing (we don't have to worry about things like > >>> does > >>>>> 1.1 > >>>>>> work with 2.3, etc) and gives people an upgrade path as they > >> desire. > >>>> This > >>>>>> also allows us to decide what pieces of the compatibility shim go > >> in > >>>> the > >>>>>> 2.0 server versus the 1.lastX client. (I actually lean towards > >>>> allowing a > >>>>>> full break between v1 and v2 server/client but understand that that > >>>> level > >>>>>> or coordination is hard in many organizations since analysts are > >>>> separate > >>>>>> from IT). Hopefully, what I'm proposing can be a good compromise > >>>> between > >>>>>> progress and deployment ease. > >>>>>> > >>>>>> Thoughts? > >>>>>> > >>>>>> Re: Branches/Dangers > >>>>>> > >>>>>> Good points on this Julian. > >>>>>> > >>>>>> How about this: > >>>>>> > >>>>>> - small fixes and enhancements PRs should be made against v1 > >>>>>> - new feature PRs should be made against v2 > >>>>>> - v2 should continue to always pass all precommit tests during its > >>> life > >>>>>> - v2 becomes master in two months > >>>>>> > >>>>>> I definitely don't want to create instability in the v2 branch. > >>>>>> > >>>>>> The other option I see is we can only do bug fix releases and > >> branch > >>>> the > >>>>>> current master into a maintenance branch and treat master as v2. > >>>>>> > >>>>>> Other ideas? > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Jacques Nadeau > >>>>>> CTO and Co-Founder, Dremio > >>>>>> > >>>>>> On Sat, Mar 26, 2016 at 6:07 PM, Julian Hyde <[email protected]> > >>> wrote: > >>>>>> > >>>>>>> Do you plan to be doing significant development on both the v1 and > >>> v2 > >>>>>>> branches, and if so, for how long? I have been bitten badly by > >> that > >>>>> pattern > >>>>>>> in the past. Developers put lots of unrelated, destabilizing > >> changes > >>>>> into > >>>>>>> v2, it look longer than expected to stabilize v2, product > >> management > >>>>> lost > >>>>>>> confidence in v2 and shifted resources back to v1, and v2 never > >>> caught > >>>>> up > >>>>>>> with v1. > >>>>>>> > >>>>>>> One important question: Which branch will you ask people to target > >>> for > >>>>>>> pull requests? v1, v2 or both? If they submit to v2, and v2 is > >>> broken, > >>>>> how > >>>>>>> will you know whether the patches are good? > >>>>>>> > >>>>>>> My recommendation is to choose one of the following: (1) put a > >>> strict > >>>>>>> time limit of say 2 months after which v2 would become the master > >>>> branch > >>>>>>> (and v1 master would become a maintenance branch), or (2) make v2 > >>>>> focused > >>>>>>> on a particular architectural feature; create multiple independent > >>>>> feature > >>>>>>> branches with breaking API changes if you need to. > >>>>>>> > >>>>>>> Julian > >>>>>>> > >>>>>>> > >>>>>>>> On Mar 26, 2016, at 1:41 PM, Paul Rogers <[email protected]> > >>>>> wrote: > >>>>>>>> > >>>>>>>> Hi All, > >>>>>>>> > >>>>>>>> 2.0 is a good opportunity to enhance our ZK information. See > >>>>>>> DRILL-4543: Advertise Drill-bit ports, status, capabilities in > >>>>> ZooKeeper. > >>>>>>> This change will simplify YARN integration. > >>>>>>>> > >>>>>>>> This enhancement will change the “public API” in ZK. To Parth’s > >>>> point, > >>>>>>> we can do so in a way that old clients work - as long as a > >> Drill-bit > >>>>> uses > >>>>>>> default ports. > >>>>>>>> > >>>>>>>> I’ve marked this JIRA as a candidate for 2.0. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> > >>>>>>>> - Paul > >>>>>>>> > >>>>>>>>> On Mar 24, 2016, at 4:11 PM, Parth Chandra <[email protected]> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>> What's our proposal for backward compatibility between 1.x and > >>> 2.x? > >>>>>>>>> My thoughts: > >>>>>>>>> Optional - Allow a mixture of 1.x and 2.x drillbits in a > >>> cluster. > >>>>>>>>> Required - 1.x clients should be able to talk to 2.x drillbits. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau < > >>>> [email protected]> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> There are some changes that either have reviews pending or are > >>> in > >>>>>>> progress > >>>>>>>>>> that would require breaking changes to Drill core. > >>>>>>>>>> > >>>>>>>>>> Examples Include: > >>>>>>>>>> DRILL-4455 (arrow integration) > >>>>>>>>>> DRILL-4417 (jdbc/odbc/rpc changes) > >>>>>>>>>> DRILL-4534 (improve null performance) > >>>>>>>>>> > >>>>>>>>>> I've created a new 2.0.0 release version in JIRA and moved > >> these > >>>>>>> tasks to > >>>>>>>>>> that umbrella. > >>>>>>>>>> > >>>>>>>>>> I'd like to propose a new v2 release branch where we can start > >>>>>>>>>> incorporating these changes without disrupting v1 stability > >> and > >>>>>>>>>> compatibility. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Jacques Nadeau > >>>>>>>>>> CTO and Co-Founder, Dremio > >>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
