I opened hbase-21002 to start the scripts and assembly. Mike
On Thu, Aug 2, 2018, 19:29 Stack <st...@duboce.net> wrote: > Up in https://issues.apache.org/jira/browse/HBASE-20934 I created an > hbase-connectors repo. I put some form on it using the v19 patch from > HBASE-15320 "HBase connector for Kafka Connect". It builds and tests > pass. Here are some remaining TODOs: > > * Figure how to do start scripts: e.g. we need to start up the kafka > proxy. It wants some hbase jars, conf dir, and others on the CLASSPATH > (Depend on an HBASE_HOME and then source bin/hbase?) > * Can any of the connectors make-do with the shaded client? > * Make connectors standalone or have them share conf, bin, etc? > * Need to do an assembly. Not done. > * Move over REST and thrift next. Mapreduce after? > > The poms could do w/ a review. Hacked them over from hbase-thirdparty. > > File issues and apply patches up in JIRA if your up for any of the above. > > Thanks, > S > > On Wed, Jul 25, 2018 at 10:46 PM Stack <st...@duboce.net> wrote: > > > > > > On Tue, Jul 24, 2018 at 10:01 PM Misty Linville <mi...@apache.org> > wrote: > >> > >> I like the idea of a separate connectors repo/release vehicle, but I'm a > >> little concerned about the need to release all together to update just > one > >> of the connectors. How would that work? What kind of compatibility > >> guarantees are we signing up for? > >> > > > > I hate responses that begin "Good question" -- so fawning -- but, ahem, > good question Misty (in the literal, not flattering, sense). > > > > I think hbase-connectors will be like hbase-thirdparty. The latter > includes netty, pb, guava and a few other bits and pieces so yeah, > sometimes a netty upgrade or an improvement on our patch to pb will require > us releasing all though we are fixing one lib only. Usually, if bothering > to make a release, we'll check for fixes or updates we can do in the other > bundled components. > > > > On the rate of releases, I foresee a flurry of activity around launch as > we fill missing bits and address critical bug fixes, but that then it will > settle down to be boring, with just the occasional update. Thrift and REST > have been stable for a good while now (not saying this is a good thing). > Our Sean just suggested moving mapreduce to connectors too -- an > interesting idea -- and this has also been stable too (at least until > recently with the shading work). We should talk about the Spark connector > when it comes time. It might not be as stable as the others. > > > > On the compatibility guarantees, we'll semver it so if an incompatible > change in a connector or if the connectors have to change to match a new > version of hbase, we'll make sure the hbase-connector version number is > changed appropriately. On the backend, what Mike says; connectors use HBase > Public APIs (else they can't be moved to the hbase-connector repo). > > > > S > > > > > > > > > > > >> > >> On Tue, Jul 24, 2018, 9:41 PM Stack <st...@duboce.net> wrote: > >> > >> > Grand. I filed https://issues.apache.org/jira/browse/HBASE-20934. > Let me > >> > have a go at making the easy one work first (the kafka proxy). Lets > see how > >> > it goes. I'll report back here. > >> > S > >> > > >> > On Tue, Jul 24, 2018 at 2:43 PM Sean Busbey <bus...@apache.org> > wrote: > >> > > >> > > Key functionality for the project's adoption should be in the > project. > >> > > Please do not suggest we donate things to Bahir. > >> > > > >> > > I apologize if this is brisk. I have had previous negative > experiences > >> > > with folks that span our communities trying to move work I spent a > lot > >> > > of time contributing to within HBase over to Bahir in an attempt to > >> > > bypass an agreed upon standard of quality. > >> > > > >> > > On Tue, Jul 24, 2018 at 3:38 PM, Artem Ervits < > artemerv...@gmail.com> > >> > > wrote: > >> > > > Why not just donating the connector to http://bahir.apache.org/ ? > >> > > > > >> > > > On Tue, Jul 24, 2018, 12:51 PM Lars Francke < > lars.fran...@gmail.com> > >> > > wrote: > >> > > > > >> > > >> I'd love to have the Kafka Connector included. > >> > > >> > >> > > >> @Mike thanks so much for the contribution (and your planned ones) > >> > > >> > >> > > >> I'm +1 on adding it to the core but I'm also +1 on having a > separate > >> > > >> repository under Apache governance > >> > > >> > >> > > >> On Tue, Jul 24, 2018 at 6:01 PM, Josh Elser <els...@apache.org> > >> > wrote: > >> > > >> > >> > > >> > +1 to the great point by Duo about use of non-IA.Public classes > >> > > >> > > >> > > >> > +1 for Apache for the governance (although, I wouldn't care if > we > >> > use > >> > > >> > Github PRs to try to encourage more folks to contribute), a > repo > >> > with > >> > > the > >> > > >> > theme of "connectors" (to include Thrift, REST, and the like). > Spark > >> > > too > >> > > >> -- > >> > > >> > I think we had suggested that prior, but it could be a mental > >> > > invention > >> > > >> of > >> > > >> > mine.. > >> > > >> > > >> > > >> > > >> > > >> > On 7/24/18 10:16 AM, Hbase Janitor wrote: > >> > > >> > > >> > > >> >> Hi everyone, > >> > > >> >> > >> > > >> >> I'm the author of the patch. A separate repo for all the > >> > connectors > >> > > is > >> > > >> a > >> > > >> >> great idea! I can make whatever changes necessary to the > patch to > >> > > help. > >> > > >> >> > >> > > >> >> I have several other integration type projects like this > planned. > >> > > >> >> > >> > > >> >> Mike > >> > > >> >> > >> > > >> >> > >> > > >> >> On Tue, Jul 24, 2018, 00:03 Mike Drob <md...@apache.org> > wrote: > >> > > >> >> > >> > > >> >> I would be ok with all of the connectors in a single repo. > Doing a > >> > > repo > >> > > >> >>> per > >> > > >> >>> connector seems like a large amount of overhead work. > >> > > >> >>> > >> > > >> >>> On Mon, Jul 23, 2018, 9:12 PM Clay B. <c...@clayb.net> wrote: > >> > > >> >>> > >> > > >> >>> [Non-binding] > >> > > >> >>>> > >> > > >> >>>> I am all for the Kafka Connect(er) as indeed it makes HBase > "more > >> > > >> >>>> relevant" and generates buzz to help me sell HBase adoption > in my > >> > > >> >>>> endeavors. > >> > > >> >>>> > >> > > >> >>>> Also, I would like to see a connectors repo a lot as I would > >> > > expect it > >> > > >> >>>> > >> > > >> >>> can > >> > > >> >>> > >> > > >> >>>> make the HBase source and releases more obvious in what is > >> > > changing. > >> > > >> Not > >> > > >> >>>> to distract from Kafka, but Spark has in the past been a > hang-up > >> > > and > >> > > >> >>>> > >> > > >> >>> seems > >> > > >> >>> > >> > > >> >>>> a good fit in such a repo too; as such, I would prefer > Apache > >> > over > >> > > >> >>>> > >> > > >> >>> GitHub. > >> > > >> >>> > >> > > >> >>>> > >> > > >> >>>> -Clay > >> > > >> >>>> > >> > > >> >>>> On Mon, 23 Jul 2018, Andrew Purtell wrote: > >> > > >> >>>> > >> > > >> >>>> Would we make a new repo called hbase-connectors and move > REST, > >> > > >> >>>>>> > >> > > >> >>>>> thrift, > >> > > >> >>> > >> > > >> >>>> and this new patch there? > >> > > >> >>>>> > >> > > >> >>>>> I like this idea. We are already releasing > hbase-thirdparty like > >> > > >> this. > >> > > >> >>>>> > >> > > >> >>>>> > >> > > >> >>>>> On Mon, Jul 23, 2018 at 5:47 PM Stack <st...@duboce.net> > wrote: > >> > > >> >>>>> > >> > > >> >>>>> (Thanks for the good discussion) > >> > > >> >>>>>> > >> > > >> >>>>>> Where we think 'outside of HBase' would be? > >> > > >> >>>>>> > >> > > >> >>>>>> Github seems too 'remote' from project and from Apache? > Would > >> > we > >> > > >> make > >> > > >> >>>>>> > >> > > >> >>>>> a > >> > > >> >>> > >> > > >> >>>> new > >> > > >> >>>> > >> > > >> >>>>> repo called hbase-connectors and move REST, thrift, and > this new > >> > > >> patch > >> > > >> >>>>>> there? > >> > > >> >>>>>> > >> > > >> >>>>>> Thanks, > >> > > >> >>>>>> S > >> > > >> >>>>>> > >> > > >> >>>>>> On Mon, Jul 23, 2018 at 3:50 PM Josh Elser < > els...@apache.org> > >> > > >> wrote: > >> > > >> >>>>>> > >> > > >> >>>>>> I'm -0 for including this into the main hbase tree. I > feel like > >> > > >> we've > >> > > >> >>>>>>> made a bit of progress in cleaning up our core, and this > >> > > strikes me > >> > > >> >>>>>>> > >> > > >> >>>>>> as > >> > > >> >>> > >> > > >> >>>> a > >> > > >> >>>> > >> > > >> >>>>> step in the wrong direction. > >> > > >> >>>>>>> > >> > > >> >>>>>>> At the same time, the integration seems nice enough (for > the > >> > > same > >> > > >> >>>>>>> reasons Andrew points out). Is there a reason this > couldn't > >> > > exist > >> > > >> >>>>>>> outside of HBase (at the ASF or otherwise)? Given a quick > >> > > glance at > >> > > >> >>>>>>> > >> > > >> >>>>>> the > >> > > >> >>> > >> > > >> >>>> patch, it would be quite trivial to keep separate (just > requires > >> > > some > >> > > >> >>>>>>> heavier scripting to get it off the ground that the HBase > >> > > scripts > >> > > >> do > >> > > >> >>>>>>> setup for). I feel like that will decrease our debt > while we > >> > > see if > >> > > >> >>>>>>> people start using it. Our API should be more than stable > >> > > enough to > >> > > >> >>>>>>> prevent any worry about drift happening from core to this > >> > > project. > >> > > >> >>>>>>> > >> > > >> >>>>>>> On 7/23/18 6:35 PM, Stack wrote: > >> > > >> >>>>>>> > >> > > >> >>>>>>>> We have a very nice contrib sitting up in HBASE-15320 > which > >> > > via a > >> > > >> >>>>>>>> > >> > > >> >>>>>>> proxy > >> > > >> >>>> > >> > > >> >>>>> -- > >> > > >> >>>>>>> > >> > > >> >>>>>>>> so minimal dependencies -- adds source and sink for > Kafka > >> > > Connect. > >> > > >> >>>>>>>> > >> > > >> >>>>>>> It > >> > > >> >>> > >> > > >> >>>> is > >> > > >> >>>>>> > >> > > >> >>>>>>> nicely contained inside two new hbase-kafka-* modules. > >> > > >> >>>>>>>> > >> > > >> >>>>>>>> We good w/ taking on this new feature? > >> > > >> >>>>>>>> > >> > > >> >>>>>>>> It looks good to me. Check it out up on HBASE-15320. I > was > >> > > going > >> > > >> to > >> > > >> >>>>>>>> > >> > > >> >>>>>>> commit > >> > > >> >>>>>>> > >> > > >> >>>>>>>> to tip of branch-2 so it'd show up in hbase-2.2.x > unless you > >> > > all > >> > > >> >>>>>>>> > >> > > >> >>>>>>> want > >> > > >> >>> > >> > > >> >>>> some > >> > > >> >>>>>>> > >> > > >> >>>>>>>> backporting action going on. > >> > > >> >>>>>>>> > >> > > >> >>>>>>>> S > >> > > >> >>>>>>>> > >> > > >> >>>>>>>> > >> > > >> >>>>>>> > >> > > >> >>>>>> > >> > > >> >>>>> > >> > > >> >>>>> -- > >> > > >> >>>>> Best regards, > >> > > >> >>>>> Andrew > >> > > >> >>>>> > >> > > >> >>>>> Words like orphans lost among the crosstalk, meaning torn > from > >> > > >> truth's > >> > > >> >>>>> decrepit hands > >> > > >> >>>>> - A23, Crosstalk > >> > > >> >>>>> > >> > > >> >>>>> > >> > > >> >>>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >> > >> > > >> > >> > > > >> > >
