Hi all! Im new here; I have been using the flink connector for hbase 1.2, but recently opt to upgrading to hbase 2.1(basically because was bundled in CDH6)
it would be nice to add support for hbase 2.x! I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and keeping that in mind last week I sent a PR with a solution supporting 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules sending the PR). i would be happy to help if needed! Felipe. > El 07-08-2020, a la(s) 10:53, Jark Wu <imj...@gmail.com> escribió: > > I'm +1 to add HBase 2.x > > However, I have some concerns about moving HBase 1.x to Bahir: > 1) As discussed above, there are still lots of people using HBase 1.x. > 2) Bahir doesn't have the infrastructure to run the existing HBase E2E tests. > 3) We also paid lots of effort to provide an uber connector jar for HBase > (not yet released), it is helpful to improve the out-of-box experience. > > My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It > doesn't add too much work to maintain a new version. > Keeping the old version can also help us to develop the new one. I would > suggest to keep HBase 1.x in the repository for at least one more release. > Another idea is that maybe it's a good time to have a > "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to it. > It would also be a good place to accept the contribution of pulsar connector > and other connectors. > > Best, > Jark > > > On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rmetz...@apache.org> wrote: >> Hi, >> >> Thank you for picking this up so quickly. I have no objections regarding >> all the proposed items. >> @Gyula: Once the bahir contribution is properly reviewed, ping me if you >> need somebody to merge it. >> >> >> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <balassi.mar...@gmail.com> >> wrote: >> >> > Hi Robert and Gyula, >> > >> > Thanks for reviving this thread. We have the implementation (currently for >> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has >> > recently written a readme for said version, he would be interested in >> > contributing the upgraded connector back. The latest HBase version is >> > 2.3.0, if we are touching the codebase anyway I would propose to have that. >> > >> > If everyone is comfortable with it I would assign [1] to Miklos with >> > double checking the all functionality that Felipe has proposed is included. >> > [1] https://issues.apache.org/jira/browse/FLINK-18795 >> > [2] https://hbase.apache.org/downloads.html >> > >> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gyula.f...@gmail.com> wrote: >> > >> >> Hi Robert, >> >> >> >> I completely agree with you on the Bahir based approach. >> >> >> >> I am happy to help with the contribution on the bahir side, with thorough >> >> review and testing. >> >> >> >> Cheers, >> >> Gyula >> >> >> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rmetz...@apache.org> wrote: >> >> >> >>> It seems that this thead is not on dev@ anymore. Adding it back ... >> >>> >> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rmetz...@apache.org> >> >>> wrote: >> >>> >> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2] >> >>>> for adding HBase 2 support. >> >>>> >> >>>> it seems that there is demand for a HBase 2 connector, and consensus to >> >>>> do it. >> >>>> >> >>>> The remaining question in this thread seems to be the "how". I would >> >>>> propose to go the other way around as Gyula suggested: We move the >> >>>> legacy >> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink. >> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we >> >>>> also run Hbase end to end tests. This will help us to stabilize the new >> >>>> connector and ensure a good quality. >> >>>> It also, the perception of what goes into Flink, and what into Bahir is >> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and >> >>>> legacy, experimental or unstable connectors into Bahir. >> >>>> >> >>>> >> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take, >> >>>> review and contribution to Bahir) >> >>>> >> >>>> >> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795 >> >>>> [2] https://github.com/apache/flink/pull/13047 >> >>>> >> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gyula.f...@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> If we were to go the bahir route, I don't see the point in migrating >> >>>>> the 1.4.x version there since that's already available in Flink. To me >> >>>>> that >> >>>>> is almost the same as dropping explicit support for 1.4 and telling >> >>>>> users >> >>>>> to use older connector versions if they wish to keep using it. >> >>>>> >> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate >> >>>>> that, we can do that inside Flink and only push the 2.4.x version to >> >>>>> bahir. >> >>>>> >> >>>>> What do you think? >> >>>>> >> >>>>> Gyula >> >>>>> >> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> >> >>>>> wrote: >> >>>>> >> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them >> >>>>>> out to Bahir and list them in flink-packages to avoid adding even more >> >>>>>> modules to Flink core? >> >>>>>> >> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <open...@gmail.com> wrote: >> >>>>>> >> >>>>>>> Hi >> >>>>>>> >> >>>>>>> According to my observation in the hbase community, there are still >> >>>>>>> lots of hbase users running their production cluster with version >> >>>>>>> 1.x (1.4x >> >>>>>>> or 1.5.x). so I'd like to suggest that >> >>>>>>> supporting both hbase1.x & hbase2.x connector. >> >>>>>>> >> >>>>>>> Thanks. >> >>>>>>> >> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <brighty...@gmail.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are >> >>>>>>>> doing for Kafka. >> >>>>>>>> >> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <car...@gmail.com> wrote: >> >>>>>>>> >> >>>>>>>>> One supplement: >> >>>>>>>>> >> >>>>>>>>> I noticed that there are discussions in HBase ML this March about >> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow >> >>>>>>>>> up in >> >>>>>>>>> HBase community about why we didn't take real action. However, >> >>>>>>>>> this doesn't >> >>>>>>>>> change my previous statement / stand due to the number of 1.x >> >>>>>>>>> usages in >> >>>>>>>>> production. >> >>>>>>>>> >> >>>>>>>>> Best Regards, >> >>>>>>>>> Yu >> >>>>>>>>> >> >>>>>>>>> [1] >> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3c30180be2-bd93-d414-a158-16c9c8d01...@apache.org%3E >> >>>>>>>>> >> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <car...@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is >> >>>>>>>>>> indeed an old version. >> >>>>>>>>>> >> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in >> >>>>>>>>>> production. We could also see that the HBase community is still >> >>>>>>>>>> maintaining >> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1] >> >>>>>>>>>> >> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3] >> >>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, >> >>>>>>>>>> etc.) >> >>>>>>>>>> between major versions. So if we only maintain 2.x connector, it >> >>>>>>>>>> would not >> >>>>>>>>>> be able to work with 1.x HBase clusters. >> >>>>>>>>>> >> >>>>>>>>>> I totally understand the additional efforts of maintaining two >> >>>>>>>>>> modules, but since we're also reserving multiple versions for >> >>>>>>>>>> kafka >> >>>>>>>>>> connector, meantime considering the current HBase in-production >> >>>>>>>>>> status, I'd >> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported. >> >>>>>>>>>> >> >>>>>>>>>> Best Regards, >> >>>>>>>>>> Yu >> >>>>>>>>>> >> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html >> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning >> >>>>>>>>>> [3] https://semver.org/ >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xbjt...@gmail.com> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we >> >>>>>>>>>>> deprecates finished(maybe one version is enough). >> >>>>>>>>>>> >> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and >> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure >> >>>>>>>>>>> SQL user, the >> >>>>>>>>>>> dependencies is a little more complex. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <liuyon...@gmail.com> 写道： >> >>>>>>>>>>> >> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies >> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Best, >> >>>>>>>>>>> Leonard Xu >> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687 >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Best Regards >> >>>>>>>> Michael Li >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Arvid Heise | Senior Java Developer >> >>>>>> >> >>>>>> <https://www.ververica.com/> >> >>>>>> >> >>>>>> Follow us @VervericaData >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> >>>>>> Conference >> >>>>>> >> >>>>>> Stream Processing | Event Driven | Real Time >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g> >> >>>>>> >> >>>>>> -- >> >>>>>> Ververica GmbH >> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, >> >>>>>> Ji (Toni) Cheng >> >>>>>> >> >>>>>
