Great idea, please take a look at the Java ByteKeyRestrictionTracker implementation for consistency [1] I remember we had to deal with lots of corner cases so probably worth a look.
[1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java On Mon, Feb 14, 2022 at 6:39 PM Robert Bradshaw <rober...@google.com> wrote: > +1 to being forward looking and making restriction trackers. Hopefully the > restriction tracker and existing range tracker could share 90% of their > code. > > On Mon, Feb 14, 2022 at 9:36 AM Sami Niemi <sami.ni...@solita.fi> wrote: > >> Hello Robert, >> >> >> >> Beam has documented only OffsetRangeTracker [1] for new SDF API. Since >> Beam is moving away from Source API, I thought it would be nice to develop >> IO connectors by using new SDFs. For this I need to create restriction >> tracker that follows new SDF API. >> >> >> >> So I propose adding ByteKeyRange as new restriction class and >> ByteKeyRestrictionTracker as new restriction tracker class. In my >> implementation I’ve also used ByteKey class which are given to restriction. >> >> >> >> 1. >> >> https://github.com/apache/beam/blob/7eb7fd017a43353204eb8037603409dda7e0414a/sdks/python/apache_beam/io/restriction_trackers.py#L76 >> >> >> >> On 2022/02/11 18:27:23 Robert Bradshaw wrote: >> >> > Hi Sam! Glad to hear you're willing to contribute. >> >> > >> >> > Though the name is a bit different, I'm wondering if this is already >> >> > present as LexicographicKeyRangeTracker. >> >> > >> https://github.com/apache/beam/blob/release-2.35.0/sdks/python/apache_beam/io/range_trackers.py#L349 >> >> > >> >> > On Fri, Feb 11, 2022 at 9:54 AM Ahmet Altay <al...@google.com> wrote: >> >> > > >> >> > > Hi Sami. Thank you for your interest. >> >> > > >> >> > > Adding people who might be able to comment: @Chamikara Jayalath >> @Lukasz Cwik >> >> > > >> >> > > On Thu, Feb 10, 2022 at 8:38 AM Sami Niemi <sa...@solita.fi> wrote: >> >> > >> >> >> > >> Hello, >> >> > >> >> >> > >> >> >> > >> >> >> > >> I noticed that Python SDK only has implementation for >> OffsetRangeTracker and OffsetRange while Java also has ByteKeyRange and >> -Tracker. >> >> > >> >> >> > >> >> >> > >> >> >> > >> I have currently created simple implementations of following Python >> classes: >> >> > >> >> >> > >> ByteKey >> >> > >> ByteKeyRange >> >> > >> ByteKeyRestrictionTracker >> >> > >> >> >> > >> >> >> > >> >> >> > >> I would like to make contribution and make these available in Python >> SDK in addition to OffsetRange and -Tracker. I would like to hear any >> thoughts about this and should I make a contribution. >> >> > >> >> >> > >> >> >> > >> >> >> > >> Thank you, >> >> > >> >> >> > >> Sami Niemi >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> *SAMI NIEMI* >> Data Engineer >> +358 50 412 2115 <+358%2050%204122115> >> sami.ni...@solita.fi >> >> >> >> *SOLITA* >> Eteläesplanadi 8 >> 00130 Helsinki >> solita.fi <https://www.solita.fi> >> >> >> >
