Re: [DISCUSS] Updating Flink HBase Connectors
Hi Ferenc, The repository is there https://github.com/apache/flink-connector-hbase You know where to reach out to if you have any questions. Best regards, Martijn On Mon, Nov 21, 2022 at 12:42 PM Ferenc Csaky wrote: > Hi! > > Sure, thank you both for the input. If the repo is ready, I'll start the > work. > > Best, > F > > > --- Original Message --- > On Thursday, November 17th, 2022 at 09:39, Gabor Somogyi < > gabor.g.somo...@gmail.com> wrote: > > > > > > > > Hi All, > > > > +1 to go. Since we are refurbishing the HBase area maybe we can move the > > token provider into HBase base project. > > This would fit into the high level effort to extract everything into > > external connectors. If you do this and facing any issues just ping me :) > > > > BR, > > G > > > > > > On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser martijnvis...@apache.org > > > > wrote: > > > > > Hi Ferenc, > > > > > > I think you're good to go, since no comments were there. Do let us > know if > > > you need any help :) > > > > > > Thanks, > > > > > > Martijn > > > > > > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > wrote: > > > > > > > Hi, > > > > > > > > just pinging this thread in case someone missed it and has any > opinion > > > > about the discussed actions. > > > > > > > > Best, > > > > F > > > > > > > > --- Original Message --- > > > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky > > > > ferenc.cs...@pm.me.INVALID wrote: > > > > > > > > > Hi Martijn, > > > > > > > > > > Thank you for your comment. About HBase 2.x, correct, that is my > > > > > thought > > > > > process, but it has to be tested and verified. > > > > > > > > > > +1 from my side about merging these updates with the connector > > > > > externalization. > > > > > > > > > > Best, > > > > > F > > > > > > > > > > --- Original Message --- > > > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > > > > > martijnvis...@apache.org wrote: > > > > > > > > > > > Hi Ferenc, > > > > > > > > > > > > Thanks for opening the discussion on this topic! > > > > > > > > > > > > +1 for dropping HBase 1.x. > > > > > > > > > > > > Regarding HBase 2.x, if I understand correctly it should be > possible > > > > > > to > > > > > > connect to any 2.x cluster if you're using the 2.x client. > Wouldn't > > > > > > it > > > > > > make > > > > > > more sense to always support the latest available version, so > > > > > > basically 2.5 > > > > > > at the moment? We could always include a test to check that > > > > > > implementation > > > > > > against an older HBase version. > > > > > > > > > > > > I also have a follow-up question already: if there's an > agreement on > > > > > > this > > > > > > topic, does it make sense to directly build a new HBase > connector in > > > > > > its > > > > > > own external connector repo, since I believe the current > connector > > > > > > uses the > > > > > > old source/sink interfaces. We could then directly drop the ones > in > > > > > > the > > > > > > Flink repo and replace it with new implementations? > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Now that the connector externalization effort ig going on, I > think > > > > > > > it is > > > > > > > definitely work to revisit the currently supported HBase > versions > > > > > > > for the > > > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 > > > > > > > connector > > > > > > > versions, although both of those versions are kind of outdated. > > > > > > > > > > > > > > From the HBase point of view the following can be considered > [1]: > > > > > > > > > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to > > > > > > > drop > > > > > > > it. > > > > > > > - HBase 2.2 is EoL, but still used actively, we are also > supporting > > > > > > > it ins > > > > > > > some of our still active releases as Cloudera. > > > > > > > - HBase 2.4 is the main thing now and probably will be > supported > > > > > > > for > > > > > > > a > > > > > > > while (by us, definitely). > > > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so > it > > > > > > > is > > > > > > > possible it won't live long. > > > > > > > - HBase 3 is in alpha, but shooting for that probably would be > > > > > > > early > > > > > > > in > > > > > > > the near future. > > > > > > > > > > > > > > In addition, if we are only using the standard HBase 2.x client > > > > > > > APIs, then > > > > > > > it should be possible to be compile it with any Hbase 2.x > version. > > > > > > > Also, > > > > > > > any HBase 2.x cluster should be backwards compatible with all > > > > > > > earlier HBase > > > > > > > 2.x client libraries. I did not check this part thorougly but I > > > > > > > think this > > > > > > > should be true, so ideally it would be enough to
Re: [DISCUSS] Updating Flink HBase Connectors
Hi! Sure, thank you both for the input. If the repo is ready, I'll start the work. Best, F --- Original Message --- On Thursday, November 17th, 2022 at 09:39, Gabor Somogyi wrote: > > > Hi All, > > +1 to go. Since we are refurbishing the HBase area maybe we can move the > token provider into HBase base project. > This would fit into the high level effort to extract everything into > external connectors. If you do this and facing any issues just ping me :) > > BR, > G > > > On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser martijnvis...@apache.org > > wrote: > > > Hi Ferenc, > > > > I think you're good to go, since no comments were there. Do let us know if > > you need any help :) > > > > Thanks, > > > > Martijn > > > > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > > > Hi, > > > > > > just pinging this thread in case someone missed it and has any opinion > > > about the discussed actions. > > > > > > Best, > > > F > > > > > > --- Original Message --- > > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky > > > ferenc.cs...@pm.me.INVALID wrote: > > > > > > > Hi Martijn, > > > > > > > > Thank you for your comment. About HBase 2.x, correct, that is my > > > > thought > > > > process, but it has to be tested and verified. > > > > > > > > +1 from my side about merging these updates with the connector > > > > externalization. > > > > > > > > Best, > > > > F > > > > > > > > --- Original Message --- > > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > > > > martijnvis...@apache.org wrote: > > > > > > > > > Hi Ferenc, > > > > > > > > > > Thanks for opening the discussion on this topic! > > > > > > > > > > +1 for dropping HBase 1.x. > > > > > > > > > > Regarding HBase 2.x, if I understand correctly it should be possible > > > > > to > > > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't > > > > > it > > > > > make > > > > > more sense to always support the latest available version, so > > > > > basically 2.5 > > > > > at the moment? We could always include a test to check that > > > > > implementation > > > > > against an older HBase version. > > > > > > > > > > I also have a follow-up question already: if there's an agreement on > > > > > this > > > > > topic, does it make sense to directly build a new HBase connector in > > > > > its > > > > > own external connector repo, since I believe the current connector > > > > > uses the > > > > > old source/sink interfaces. We could then directly drop the ones in > > > > > the > > > > > Flink repo and replace it with new implementations? > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Now that the connector externalization effort ig going on, I think > > > > > > it is > > > > > > definitely work to revisit the currently supported HBase versions > > > > > > for the > > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 > > > > > > connector > > > > > > versions, although both of those versions are kind of outdated. > > > > > > > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to > > > > > > drop > > > > > > it. > > > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting > > > > > > it ins > > > > > > some of our still active releases as Cloudera. > > > > > > - HBase 2.4 is the main thing now and probably will be supported > > > > > > for > > > > > > a > > > > > > while (by us, definitely). > > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it > > > > > > is > > > > > > possible it won't live long. > > > > > > - HBase 3 is in alpha, but shooting for that probably would be > > > > > > early > > > > > > in > > > > > > the near future. > > > > > > > > > > > > In addition, if we are only using the standard HBase 2.x client > > > > > > APIs, then > > > > > > it should be possible to be compile it with any Hbase 2.x version. > > > > > > Also, > > > > > > any HBase 2.x cluster should be backwards compatible with all > > > > > > earlier HBase > > > > > > 2.x client libraries. I did not check this part thorougly but I > > > > > > think this > > > > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > > > > connector. [2] > > > > > > > > > > > > Looking forward to your opinions about this topic. > > > > > > > > > > > > Best, > > > > > > F > > > > > > > > > > > > [1] https://hbase.apache.org/downloads.html > > > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 > > > > > > (Client > > > > > > API compatibility) > > > > > > > > > > -- > > > > > Martijn > > > > > https://twitter.com/MartijnVisser82 > > > > > https://github.com/MartijnVisser
Re: [DISCUSS] Updating Flink HBase Connectors
Hi All, +1 to go. Since we are refurbishing the HBase area maybe we can move the token provider into HBase base project. This would fit into the high level effort to extract everything into external connectors. If you do this and facing any issues just ping me :) BR, G On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser wrote: > Hi Ferenc, > > I think you're good to go, since no comments were there. Do let us know if > you need any help :) > > Thanks, > > Martijn > > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky > wrote: > > > Hi, > > > > just pinging this thread in case someone missed it and has any opinion > > about the discussed actions. > > > > Best, > > F > > > > > > > > > > --- Original Message --- > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky > > wrote: > > > > > > > > > > > > > Hi Martijn, > > > > > > Thank you for your comment. About HBase 2.x, correct, that is my > thought > > process, but it has to be tested and verified. > > > > > > +1 from my side about merging these updates with the connector > > externalization. > > > > > > Best, > > > F > > > > > > > > > --- Original Message --- > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > > martijnvis...@apache.org wrote: > > > > > > > > > > > > > Hi Ferenc, > > > > > > > > Thanks for opening the discussion on this topic! > > > > > > > > +1 for dropping HBase 1.x. > > > > > > > > Regarding HBase 2.x, if I understand correctly it should be possible > to > > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't > it > > make > > > > more sense to always support the latest available version, so > > basically 2.5 > > > > at the moment? We could always include a test to check that > > implementation > > > > against an older HBase version. > > > > > > > > I also have a follow-up question already: if there's an agreement on > > this > > > > topic, does it make sense to directly build a new HBase connector in > > its > > > > own external connector repo, since I believe the current connector > > uses the > > > > old source/sink interfaces. We could then directly drop the ones in > the > > > > Flink repo and replace it with new implementations? > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > > > > > > > Hi everyone, > > > > > > > > > > Now that the connector externalization effort ig going on, I think > > it is > > > > > definitely work to revisit the currently supported HBase versions > > for the > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 > > connector > > > > > versions, although both of those versions are kind of outdated. > > > > > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to > drop > > it. > > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting > > it ins > > > > > some of our still active releases as Cloudera. > > > > > - HBase 2.4 is the main thing now and probably will be supported > for > > a > > > > > while (by us, definitely). > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it > is > > > > > possible it won't live long. > > > > > - HBase 3 is in alpha, but shooting for that probably would be > early > > in > > > > > the near future. > > > > > > > > > > In addition, if we are only using the standard HBase 2.x client > > APIs, then > > > > > it should be possible to be compile it with any Hbase 2.x version. > > Also, > > > > > any HBase 2.x cluster should be backwards compatible with all > > earlier HBase > > > > > 2.x client libraries. I did not check this part thorougly but I > > think this > > > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > > > connector. [2] > > > > > > > > > > Looking forward to your opinions about this topic. > > > > > > > > > > Best, > > > > > F > > > > > > > > > > [1] https://hbase.apache.org/downloads.html > > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 > > (Client > > > > > API compatibility) > > > > > > > > -- > > > > Martijn > > > > https://twitter.com/MartijnVisser82 > > > > https://github.com/MartijnVisser > > >
Re: [DISCUSS] Updating Flink HBase Connectors
Hi Ferenc, I think you're good to go, since no comments were there. Do let us know if you need any help :) Thanks, Martijn On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky wrote: > Hi, > > just pinging this thread in case someone missed it and has any opinion > about the discussed actions. > > Best, > F > > > > > --- Original Message --- > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky > wrote: > > > > > > > > Hi Martijn, > > > > Thank you for your comment. About HBase 2.x, correct, that is my thought > process, but it has to be tested and verified. > > > > +1 from my side about merging these updates with the connector > externalization. > > > > Best, > > F > > > > > > --- Original Message --- > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > martijnvis...@apache.org wrote: > > > > > > > > > Hi Ferenc, > > > > > > Thanks for opening the discussion on this topic! > > > > > > +1 for dropping HBase 1.x. > > > > > > Regarding HBase 2.x, if I understand correctly it should be possible to > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't it > make > > > more sense to always support the latest available version, so > basically 2.5 > > > at the moment? We could always include a test to check that > implementation > > > against an older HBase version. > > > > > > I also have a follow-up question already: if there's an agreement on > this > > > topic, does it make sense to directly build a new HBase connector in > its > > > own external connector repo, since I believe the current connector > uses the > > > old source/sink interfaces. We could then directly drop the ones in the > > > Flink repo and replace it with new implementations? > > > > > > Best regards, > > > > > > Martijn > > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > > > > Hi everyone, > > > > > > > > Now that the connector externalization effort ig going on, I think > it is > > > > definitely work to revisit the currently supported HBase versions > for the > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 > connector > > > > versions, although both of those versions are kind of outdated. > > > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to drop > it. > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting > it ins > > > > some of our still active releases as Cloudera. > > > > - HBase 2.4 is the main thing now and probably will be supported for > a > > > > while (by us, definitely). > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is > > > > possible it won't live long. > > > > - HBase 3 is in alpha, but shooting for that probably would be early > in > > > > the near future. > > > > > > > > In addition, if we are only using the standard HBase 2.x client > APIs, then > > > > it should be possible to be compile it with any Hbase 2.x version. > Also, > > > > any HBase 2.x cluster should be backwards compatible with all > earlier HBase > > > > 2.x client libraries. I did not check this part thorougly but I > think this > > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > > connector. [2] > > > > > > > > Looking forward to your opinions about this topic. > > > > > > > > Best, > > > > F > > > > > > > > [1] https://hbase.apache.org/downloads.html > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 > (Client > > > > API compatibility) > > > > > > -- > > > Martijn > > > https://twitter.com/MartijnVisser82 > > > https://github.com/MartijnVisser >
Re: [DISCUSS] Updating Flink HBase Connectors
Hi, just pinging this thread in case someone missed it and has any opinion about the discussed actions. Best, F --- Original Message --- On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky wrote: > > > Hi Martijn, > > Thank you for your comment. About HBase 2.x, correct, that is my thought > process, but it has to be tested and verified. > > +1 from my side about merging these updates with the connector > externalization. > > Best, > F > > > --- Original Message --- > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > martijnvis...@apache.org wrote: > > > > > Hi Ferenc, > > > > Thanks for opening the discussion on this topic! > > > > +1 for dropping HBase 1.x. > > > > Regarding HBase 2.x, if I understand correctly it should be possible to > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't it make > > more sense to always support the latest available version, so basically 2.5 > > at the moment? We could always include a test to check that implementation > > against an older HBase version. > > > > I also have a follow-up question already: if there's an agreement on this > > topic, does it make sense to directly build a new HBase connector in its > > own external connector repo, since I believe the current connector uses the > > old source/sink interfaces. We could then directly drop the ones in the > > Flink repo and replace it with new implementations? > > > > Best regards, > > > > Martijn > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > Hi everyone, > > > > > > Now that the connector externalization effort ig going on, I think it is > > > definitely work to revisit the currently supported HBase versions for the > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 connector > > > versions, although both of those versions are kind of outdated. > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to drop it. > > > - HBase 2.2 is EoL, but still used actively, we are also supporting it ins > > > some of our still active releases as Cloudera. > > > - HBase 2.4 is the main thing now and probably will be supported for a > > > while (by us, definitely). > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is > > > possible it won't live long. > > > - HBase 3 is in alpha, but shooting for that probably would be early in > > > the near future. > > > > > > In addition, if we are only using the standard HBase 2.x client APIs, then > > > it should be possible to be compile it with any Hbase 2.x version. Also, > > > any HBase 2.x cluster should be backwards compatible with all earlier > > > HBase > > > 2.x client libraries. I did not check this part thorougly but I think this > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > connector. [2] > > > > > > Looking forward to your opinions about this topic. > > > > > > Best, > > > F > > > > > > [1] https://hbase.apache.org/downloads.html > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 (Client > > > API compatibility) > > > > -- > > Martijn > > https://twitter.com/MartijnVisser82 > > https://github.com/MartijnVisser
Re: [DISCUSS] Updating Flink HBase Connectors
Hi Martijn, Thank you for your comment. About HBase 2.x, correct, that is my thought process, but it has to be tested and verified. +1 from my side about merging these updates with the connector externalization. Best, F --- Original Message --- On Tuesday, October 11th, 2022 at 16:30, Martijn Visser wrote: > > > Hi Ferenc, > > Thanks for opening the discussion on this topic! > > +1 for dropping HBase 1.x. > > Regarding HBase 2.x, if I understand correctly it should be possible to > connect to any 2.x cluster if you're using the 2.x client. Wouldn't it make > more sense to always support the latest available version, so basically 2.5 > at the moment? We could always include a test to check that implementation > against an older HBase version. > > I also have a follow-up question already: if there's an agreement on this > topic, does it make sense to directly build a new HBase connector in its > own external connector repo, since I believe the current connector uses the > old source/sink interfaces. We could then directly drop the ones in the > Flink repo and replace it with new implementations? > > Best regards, > > Martijn > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > Hi everyone, > > > > Now that the connector externalization effort ig going on, I think it is > > definitely work to revisit the currently supported HBase versions for the > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 connector > > versions, although both of those versions are kind of outdated. > > > > From the HBase point of view the following can be considered [1]: > > > > - HBase 1.x is dead, so on the way forward it should be safe to drop it. > > - HBase 2.2 is EoL, but still used actively, we are also supporting it ins > > some of our still active releases as Cloudera. > > - HBase 2.4 is the main thing now and probably will be supported for a > > while (by us, definitely). > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is > > possible it won't live long. > > - HBase 3 is in alpha, but shooting for that probably would be early in > > the near future. > > > > In addition, if we are only using the standard HBase 2.x client APIs, then > > it should be possible to be compile it with any Hbase 2.x version. Also, > > any HBase 2.x cluster should be backwards compatible with all earlier HBase > > 2.x client libraries. I did not check this part thorougly but I think this > > should be true, so ideally it would be enough to have an HBase 2.4 > > connector. [2] > > > > Looking forward to your opinions about this topic. > > > > Best, > > F > > > > [1] https://hbase.apache.org/downloads.html > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 (Client > > API compatibility) > > > -- > Martijn > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser
Re: [DISCUSS] Updating Flink HBase Connectors
Hi Ferenc, Thanks for opening the discussion on this topic! +1 for dropping HBase 1.x. Regarding HBase 2.x, if I understand correctly it should be possible to connect to any 2.x cluster if you're using the 2.x client. Wouldn't it make more sense to always support the latest available version, so basically 2.5 at the moment? We could always include a test to check that implementation against an older HBase version. I also have a follow-up question already: if there's an agreement on this topic, does it make sense to directly build a new HBase connector in its own external connector repo, since I believe the current connector uses the old source/sink interfaces. We could then directly drop the ones in the Flink repo and replace it with new implementations? Best regards, Martijn Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > Hi everyone, > > Now that the connector externalization effort ig going on, I think it is > definitely work to revisit the currently supported HBase versions for the > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 connector > versions, although both of those versions are kind of outdated. > > From the HBase point of view the following can be considered [1]: > > - HBase 1.x is dead, so on the way forward it should be safe to drop it. > - HBase 2.2 is EoL, but still used actively, we are also supporting it ins > some of our still active releases as Cloudera. > - HBase 2.4 is the main thing now and probably will be supported for a > while (by us, definitely). > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is > possible it won't live long. > - HBase 3 is in alpha, but shooting for that probably would be early in > the near future. > > In addition, if we are only using the standard HBase 2.x client APIs, then > it should be possible to be compile it with any Hbase 2.x version. Also, > any HBase 2.x cluster should be backwards compatible with all earlier HBase > 2.x client libraries. I did not check this part thorougly but I think this > should be true, so ideally it would be enough to have an HBase 2.4 > connector. [2] > > Looking forward to your opinions about this topic. > > Best, > F > > [1] https://hbase.apache.org/downloads.html > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 (Client > API compatibility) -- Martijn https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser
