I think a way to have transform-specific options could be useful, regardless of this use case.
On Mon, Mar 12, 2018 at 12:42 PM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > Le 12 mars 2018 18:56, "Chamikara Jayalath" <chamik...@google.com> a > écrit : > > Agree. We need file-system abstractions in all languages since (1) users > may need to directly access file-systems from DoFns (2) common file-based > sources/sinks will probably will be available in multiple languages even > with portability API and cross language IO (these are usually the first > sources/sinks that get implemented in an SDK and server as reference > implementations for other sources/sinks). > > > Can you detail 1? Shouldnt otherwise the abstraction is not that great. > > Side note on this one: vfs uses static utilitiez to setup custom configs > for specific impl, beam can do the same and hides the unwrapping. > > However Im a bit sad we always end up of the fs whereas the issue is > generic: how to bind some specific config for a transform? > > Ex: > - parallelism by source in the direct runner and not globally > - different pool config for jdbc.read1 and jdbc.write1 cause concurrency > is different > - different jms provider for input1 and output2 > - etc... > > As soon as: > - the pipeline is "generic" (it takes a jms source/output or a file path) > Or > - an io uses a not exposed through the main api but configurable impl > > Then you need a config per transform of the pipeline. Filesystem is not > different from the dozens of the cases so it would be great to fix it once > for all no? > > Thinking a bit more, and without using hints, we can just use the name if > set in apply: > > p.apply("myprefix", ...) would accept -Dmyprefix::myOption=foo for the > scope of the fn and if not set it would try to use myOption (inheritance > from the global context). This way we are simple, generic, and our config > becomes scalable and stays storable nicely. > > > > - Cham > > > On Mon, Mar 12, 2018 at 10:48 AM Lukasz Cwik <lc...@google.com> wrote: > >> There is still a lot of work before we get to supporting cross language >> transforms and hence get access to filesystems written in different >> languages but how the options are passed through from one to the other will >> need to be well understood and it would be best if the way a user defines >> these filesystems is the same in all languages because it would be annoying >> to provide the same configuration (in slightly different ways) for Java, >> Python, Go, ... >> >> On Fri, Mar 9, 2018 at 2:01 PM, Romain Manni-Bucau <rmannibu...@gmail.com >> > wrote: >> >>> >>> >>> Le 9 mars 2018 21:35, "Lukasz Cwik" <lc...@google.com> a écrit : >>> >>> The blocker is to get someone to follow through on the original design >>> or to get a new design (with feedback) and have it implemented. >>> >>> >>> If the pipelineoptionsfactory related pr are merged i can do a >>> pr/proposal bases on this thread draft this month. >>> >>> >>> Note that this impacts more than just Java as it also exists in Python >>> and Go as well. >>> >>> >>> Clearly outside my knowledge but since it is mainly java backed it >>> should be almost transparent no? If not should it be part of the portable >>> api on top of runners? >>> >>> >>> On Fri, Mar 9, 2018 at 12:18 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> Hmm, it doesnt solve the issue that beam doesnt enable to configure >>>> transform from its "config" (let say the cli). >>>> >>>> So if i have a generic pipeline taking a file as input and another as >>>> output then i must register 2 filesystems in all cases? If the pipeline is >>>> dynamic i must make it dynamic too? >>>> >>>> Sounds pretty bad for end users and not generic - all transform hit >>>> this issue since beam cant assume the impl. Using a prefix (namespace which >>>> can be implicit or not) is simple, straight forward and enables all cases >>>> to be handled smoothly for end users. >>>> >>>> What is the blocker to fix this design issue? I kind of fail to see why >>>> we end up on a few particular cases with workarounds right now :s. >>>> >>>> Le 9 mars 2018 19:00, "Jacob Marble" <jacobmar...@gmail.com> a écrit : >>>> >>>>> I think when I wrote the S3 code, I couldn't see how to set storage >>>>> class per-bucket, so put it in a flag. It's easy to imagine a use case >>>>> where storage class differs per filespec, not only per bucket. >>>>> >>>>> Jacob >>>>> >>>>> On Fri, Mar 9, 2018 at 9:51 AM, Jacob Marble <jacobmar...@gmail.com> >>>>> wrote: >>>>> >>>>>> Yes, I agree with all of this. >>>>>> >>>>>> Jacob >>>>>> >>>>>> On Thu, Mar 8, 2018 at 9:52 PM, Robert Bradshaw <rober...@google.com> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:38 PM Eugene Kirpichov < >>>>>>> kirpic...@google.com> wrote: >>>>>>> >>>>>>>> I think it may have been an API design mistake to put the S3 region >>>>>>>> into PipelineOptions. >>>>>>>> >>>>>>> >>>>>>> +1, IMHO it's generally a mistake to put any transform configuration >>>>>>> into PipelineOptions for exactly this reason. >>>>>>> >>>>>>> >>>>>>>> PipelineOptions are global per pipeline, whereas it's totally >>>>>>>> reasonable to access S3 files in different regions even from the code >>>>>>>> of a >>>>>>>> single DoFn running on a single element. The same applies to >>>>>>>> "setS3StorageClass". >>>>>>>> >>>>>>>> Jacob: what do you think? Why is it necessary to specify the S3 >>>>>>>> region at all - can AWS infer it automatically? Per >>>>>>>> https://github.com/aws/aws-sdk-java/issues/1107 it seems that this >>>>>>>> is possible via a setting on the client, so that the specified region >>>>>>>> is >>>>>>>> used as the default but if the bucket is in a different region things >>>>>>>> still >>>>>>>> work. >>>>>>>> >>>>>>>> As for the storage class: so far nobody complained ;) but it should >>>>>>>> probably be specified via >>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/CreateOptions.java >>>>>>>> instead >>>>>>>> of a pipeline option. >>>>>>>> >>>>>>>> On Thu, Mar 8, 2018 at 9:16 PM Romain Manni-Bucau < >>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>> >>>>>>>>> The "hint" would probably to use hints :) - indees this joke >>>>>>>>> refers to the hint thread. >>>>>>>>> >>>>>>>>> Long story short with hints you should be able to say "use that >>>>>>>>> specialize config here". >>>>>>>>> >>>>>>>>> Now, personally, I'd like to see a way to specialize config per >>>>>>>>> transform. With an hint an easy way is to use a prefix: --s3-region >>>>>>>>> would >>>>>>>>> become --prefix_transform1-s3-region. But to impl it i have >>>>>>>>> https://github.com/apache/beam/pull/4683 which needs to be merged >>>>>>>>> before ;). >>>>>>>>> >>>>>>>>> Le 8 mars 2018 23:03, "Ismaël Mejía" <ieme...@gmail.com> a écrit : >>>>>>>>> >>>>>>>>>> I was trying to create a really simple pipeline that read from a >>>>>>>>>> bucket in a filesystem (s3) and writes to a different bucket in >>>>>>>>>> the >>>>>>>>>> same filesystem. >>>>>>>>>> >>>>>>>>>> S3Options options = >>>>>>>>>> >>>>>>>>>> PipelineOptionsFactory.fromArgs(args).create().as(S3Options.class); >>>>>>>>>> Pipeline pipeline = Pipeline.create(options); >>>>>>>>>> pipeline >>>>>>>>>> .apply("ReadLines", TextIO.read().from("s3://src-bucket/*")) >>>>>>>>>> // .apply("AllOtherMagic", ...) >>>>>>>>>> .apply("WriteCounts", >>>>>>>>>> TextIO.write().to("s3://dst-bucket/")); >>>>>>>>>> p.run().waitUntilFinish(); >>>>>>>>>> >>>>>>>>>> I discovered that my original bucket was in a different region so >>>>>>>>>> I >>>>>>>>>> needed to pass a different S3Options object to the Write >>>>>>>>>> ‘options.setAwsRegion(“dst-region”)’, but I could not find a way >>>>>>>>>> to do >>>>>>>>>> it. Can somebody give me a hint on how to do this? >>>>>>>>>> >>>>>>>>>> I was wondering that since File-based IOs use the configuration >>>>>>>>>> implied by the Filesystem if this was possible. With non-file >>>>>>>>>> based >>>>>>>>>> IOs all the configuration details are explicit in each specific >>>>>>>>>> transform, but this is not the case for these file-based >>>>>>>>>> transforms. >>>>>>>>>> >>>>>>>>>> Note. I know this question probably belongs more to user@ but >>>>>>>>>> since I >>>>>>>>>> couldn’t find an easy way to do it I was wondering if this is an >>>>>>>>>> issue >>>>>>>>>> we should consider at dev@ from an API point of view. >>>>>>>>>> >>>>>>>>> >>>>>> >>>>> >>> >>> >> >