That tracker is not a restriction tracker which I need for my Bigtable reader SDF. When I started working on this tracker I noticed that it was implemented in Java and I figured it would be best to make functionally similar implementation in Python. LexicographicKeyRangeTracker is not that different except it can also handle strings as keys. I did not need the tracker to do this so I left it out to keep it more simple and closer to Java implementation.
I’m open to changes in implementation but I would like to keep it simple and not too far away from Java implementation. On 2022/02/15 16:42:35 Robert Bradshaw wrote: > On Tue, Feb 15, 2022 at 2:03 AM Sami Niemi > <sa...@solita.fi<mailto:sa...@solita.fi>> wrote: > > > > Hi Ismaël, > > > > > > > > What I’ve currently been working on locally is almost 100% based on that > > Java implementation. > > Did the existing LexicographicKeyRangeTracker not meet your needs? > > > I suppose I need to create Jira issue and make the contribution. > > > > > > > > On 2022/02/15 09:19:33 Ismaël Mejía wrote: > > > > > Oh, forgot to add also the link to the tests that cover most of those > > > > > unexpected cases: > > > > > [2] > > > > > https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java > > > > > > > > > > > > > > > On Tue, Feb 15, 2022 at 10:17 AM Ismaël Mejía > > > <ie...@gmail.com<mailto:ie...@gmail.com>> wrote: > > > > > > > > > > > Great idea, please take a look at the Java ByteKeyRestrictionTracker > > > > > > implementation for consistency [1] > > > > > > I remember we had to deal with lots of corner cases so probably worth a > > > > > > look. > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java > > > > > > > > > > > > > > > > > > On Mon, Feb 14, 2022 at 6:39 PM Robert Bradshaw > > > > <ro...@google.com<mailto:ro...@google.com>> > > > > > > wrote: > > > > > > > > > > > >> +1 to being forward looking and making restriction trackers. > > > > > >> Hopefully the restriction tracker and existing range tracker could > > > >> share > > > > > >> 90% of their code. > > > > > >> > > > > > >> On Mon, Feb 14, 2022 at 9:36 AM Sami Niemi > > > >> <sa...@solita.fi<mailto:sa...@solita.fi>> wrote: > > > > > >> > > > > > >>> Hello Robert, > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> Beam has documented only OffsetRangeTracker [1] for new SDF API. Since > > > > > >>> Beam is moving away from Source API, I thought it would be nice to > > > >>> develop > > > > > >>> IO connectors by using new SDFs. For this I need to create restriction > > > > > >>> tracker that follows new SDF API. > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> So I propose adding ByteKeyRange as new restriction class and > > > > > >>> ByteKeyRestrictionTracker as new restriction tracker class. In my > > > > > >>> implementation I’ve also used ByteKey class which are given to > > > >>> restriction. > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> 1. > > > > > >>> > > > >>> https://github.com/apache/beam/blob/7eb7fd017a43353204eb8037603409dda7e0414a/sdks/python/apache_beam/io/restriction_trackers.py#L76 > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> On 2022/02/11 18:27:23 Robert Bradshaw wrote: > > > > > >>> > > > > > >>> > Hi Sam! Glad to hear you're willing to contribute. > > > > > >>> > > > > > >>> > > > > > > >>> > > > > > >>> > Though the name is a bit different, I'm wondering if this is already > > > > > >>> > > > > > >>> > present as LexicographicKeyRangeTracker. > > > > > >>> > > > > > >>> > > > > > > >>> https://github.com/apache/beam/blob/release-2.35.0/sdks/python/apache_beam/io/range_trackers.py#L349 > > > > > >>> > > > > > >>> > > > > > > >>> > > > > > >>> > On Fri, Feb 11, 2022 at 9:54 AM Ahmet Altay > > > >>> > <al...@google.com<mailto:al...@google.com>> wrote: > > > > > >>> > > > > > >>> > > > > > > > >>> > > > > > >>> > > Hi Sami. Thank you for your interest. > > > > > >>> > > > > > >>> > > > > > > > >>> > > > > > >>> > > Adding people who might be able to comment: @Chamikara Jayalath > > > > > >>> @Lukasz Cwik > > > > > >>> > > > > > >>> > > > > > > > >>> > > > > > >>> > > On Thu, Feb 10, 2022 at 8:38 AM Sami Niemi > > > >>> > > <sa...@solita.fi<mailto:sa...@solita.fi>> wrote: > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> Hello, > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> I noticed that Python SDK only has implementation for > > > > > >>> OffsetRangeTracker and OffsetRange while Java also has ByteKeyRange > > > >>> and > > > > > >>> -Tracker. > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> I have currently created simple implementations of following > > > >>> > >> Python > > > > > >>> classes: > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> ByteKey > > > > > >>> > > > > > >>> > >> ByteKeyRange > > > > > >>> > > > > > >>> > >> ByteKeyRestrictionTracker > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> I would like to make contribution and make these available in > > > > > >>> Python SDK in addition to OffsetRange and -Tracker. I would like to > > > >>> hear > > > > > >>> any thoughts about this and should I make a contribution. > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> Thank you, > > > > > >>> > > > > > >>> > >> > > > > > >>> > > > > > >>> > >> Sami Niemi > > > > > >>> > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> *SAMI NIEMI* > > > > > >>> Data Engineer > > > > > >>> +358 50 412 2115 <+358%2050%204122115> > > > > > >>> sami.ni...@solita.fi<mailto:sami.ni...@solita.fi> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> *SOLITA* > > > > > >>> Eteläesplanadi 8 > > > > > >>> 00130 Helsinki > > > > > >>> solita.fi <https://www.solita.fi><https://www.solita.fi%3e> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > >
