A filesystem is a lower level abstraction that a PTransform can use thus there is no need to consider SDF when creating the S3 filesytem. If we were redesigning the interface to all filesystems, then SDF should be considered.
On Mon, Apr 8, 2019 at 10:54 AM Lara Schmidt <[email protected]> wrote: > I'd push towards waiting until SDF is working end to end to begin > converting things. Unless it's something like Text.ReadAll batch API that > gets benefits without a SDF implementation. I don't have a lot of context > on what file APIs python already supports. > > On Mon, Apr 8, 2019 at 10:06 AM Pablo Estrada <[email protected]> wrote: > >> Currently, Pasan is working on a design for adding a couple >> implementations to the Filesystem interface in Python, and it's not >> necessary to consider SDF here. IMHO. >> >> On the other hand, Python's fileio[1] could probably use SDF-based >> improvements to split when many files are being matched. >> Best >> -P. >> >> On Mon, Apr 8, 2019 at 10:00 AM Alex Amato <[email protected]> wrote: >> >>> +Lukasz Cwik <[email protected]>, +Boyuan Zhang <[email protected]>, +Lara >>> Schmidt <[email protected]> >>> >>> Should splittable DoFn be considered in this design? In order to split >>> and scale the source step properly? >>> >>> On Mon, Apr 8, 2019 at 9:11 AM Ahmet Altay <[email protected]> wrote: >>> >>>> +dev <[email protected]> +Pablo Estrada <[email protected]> +Chamikara >>>> Jayalath <[email protected]> +Udi Meiri <[email protected]> >>>> >>>> Thank you Pasan. I quickly looked at the proposal and it looks good. >>>> Added a few folks who could offer additional feedback. >>>> >>>> On Mon, Apr 8, 2019 at 12:13 AM Pasan Kamburugamuwa < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have updated the project proposal according to the given feedback. >>>>> So can you guys check my proposal again and give me your feedback about >>>>> corrections I have done. >>>>> >>>>> Here is the link to the updated project proposal >>>>> >>>>> https://docs.google.com/document/d/1i_PoIrbmhNgwKCS1TYWC28A9RsyZQFsQCJic3aCXO-8/edit?usp=sharing >>>>> >>>>> Thank you >>>>> Pasan Kamburugamuwa >>>>> >>>>
