Oh, forgot to add also the link to the tests that cover most of those unexpected cases: [2] https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java
On Tue, Feb 15, 2022 at 10:17 AM Ismaël Mejía <[email protected]> wrote: > Great idea, please take a look at the Java ByteKeyRestrictionTracker > implementation for consistency [1] > I remember we had to deal with lots of corner cases so probably worth a > look. > > [1] > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java > > > On Mon, Feb 14, 2022 at 6:39 PM Robert Bradshaw <[email protected]> > wrote: > >> +1 to being forward looking and making restriction trackers. >> Hopefully the restriction tracker and existing range tracker could share >> 90% of their code. >> >> On Mon, Feb 14, 2022 at 9:36 AM Sami Niemi <[email protected]> wrote: >> >>> Hello Robert, >>> >>> >>> >>> Beam has documented only OffsetRangeTracker [1] for new SDF API. Since >>> Beam is moving away from Source API, I thought it would be nice to develop >>> IO connectors by using new SDFs. For this I need to create restriction >>> tracker that follows new SDF API. >>> >>> >>> >>> So I propose adding ByteKeyRange as new restriction class and >>> ByteKeyRestrictionTracker as new restriction tracker class. In my >>> implementation I’ve also used ByteKey class which are given to restriction. >>> >>> >>> >>> 1. >>> >>> https://github.com/apache/beam/blob/7eb7fd017a43353204eb8037603409dda7e0414a/sdks/python/apache_beam/io/restriction_trackers.py#L76 >>> >>> >>> >>> On 2022/02/11 18:27:23 Robert Bradshaw wrote: >>> >>> > Hi Sam! Glad to hear you're willing to contribute. >>> >>> > >>> >>> > Though the name is a bit different, I'm wondering if this is already >>> >>> > present as LexicographicKeyRangeTracker. >>> >>> > >>> https://github.com/apache/beam/blob/release-2.35.0/sdks/python/apache_beam/io/range_trackers.py#L349 >>> >>> > >>> >>> > On Fri, Feb 11, 2022 at 9:54 AM Ahmet Altay <[email protected]> wrote: >>> >>> > > >>> >>> > > Hi Sami. Thank you for your interest. >>> >>> > > >>> >>> > > Adding people who might be able to comment: @Chamikara Jayalath >>> @Lukasz Cwik >>> >>> > > >>> >>> > > On Thu, Feb 10, 2022 at 8:38 AM Sami Niemi <[email protected]> wrote: >>> >>> > >> >>> >>> > >> Hello, >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> I noticed that Python SDK only has implementation for >>> OffsetRangeTracker and OffsetRange while Java also has ByteKeyRange and >>> -Tracker. >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> I have currently created simple implementations of following Python >>> classes: >>> >>> > >> >>> >>> > >> ByteKey >>> >>> > >> ByteKeyRange >>> >>> > >> ByteKeyRestrictionTracker >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> I would like to make contribution and make these available in >>> Python SDK in addition to OffsetRange and -Tracker. I would like to hear >>> any thoughts about this and should I make a contribution. >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> Thank you, >>> >>> > >> >>> >>> > >> Sami Niemi >>> >>> > >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *SAMI NIEMI* >>> Data Engineer >>> +358 50 412 2115 <+358%2050%204122115> >>> [email protected] >>> >>> >>> >>> *SOLITA* >>> Eteläesplanadi 8 >>> 00130 Helsinki >>> solita.fi <https://www.solita.fi> >>> >>> >>> >>
