+1 to new component not splitted Other use case is using libraries not available in your language e.g. using some python transform that relies in a python only API in the middle of a Java pipeline.
On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath <chamik...@google.com> wrote: > I proposed three components since the audience might be different. Also we > can use the same component to track issues related to all cross-language > wrappers available in a given SDK. If this is too much a single component > is fine as well. > > Ashwin, as others pointed out, the cross-language transforms framework is > primarily for giving SDKs access to transforms that are not > available natively. But there are other potential use-cases as well (for > example, using two different Python environments within the same > pipeline). > Exact performance will depend on the runner implementation as well as the > additional cost involved due to serializing/deserializing data across > environment boundaries. But we haven't done enough analysis/benchmarking to > provide more details on this. > > Thanks, > Cham > > On Thu, May 28, 2020 at 1:55 PM Kyle Weaver <kcwea...@google.com> wrote: > >> > What are some of the benefits / drawbacks of using cross-language >> transforms? Would a native Python transform perform better than a >> cross-language transform written in Java that is then used in a Python >> pipeline? >> >> As Rui says, the main advantage is code reuse. See >> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >> information. >> >> On Thu, May 28, 2020 at 4:53 PM Rui Wang <ruw...@google.com> wrote: >> >>> +1 on dedicated components for cross-language transform. It might be >>> easy to manage to have one component (one tag for all SDK) rather than >>> multiple ones. >>> >>> >>> Re Ashwin, >>> >>> Cham knows more than me. AFAIK, cross-language transforms will maximize >>> code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of >>> course, a SDK can develop its own IOs, but it's lots of work. >>> >>> >>> -Rui >>> >>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami <aramaswa...@gmail.com> >>> wrote: >>> >>>> What are some of the benefits / drawbacks of using cross-language >>>> transforms? Would a native Python transform perform better than a >>>> cross-language transform written in Java that is then used in a Python >>>> pipeline? >>>> >>>> Ashwin Ramaswami >>>> Student >>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>> Website <https://epicfaace.github.io/> | GitHub >>>> <https://github.com/epicfaace> >>>> >>>> >>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver <kcwea...@google.com> >>>> wrote: >>>> >>>>> SGTM. Though I'm not sure it's necessary to split by language. It >>>>> might be easier to use a single cross-language tag, rather than having to >>>>> tag lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>>> >>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I think it's good if we can have new Jira components to easily track >>>>>> various issues related to cross-language transforms. >>>>>> >>>>>> What do you think about adding the following Jira components ? >>>>>> >>>>>> sdks-python-xlang >>>>>> sdks-java-xlang >>>>>> sdks-go-xlang >>>>>> >>>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>>> cross-language transforms for SDK Foo. For example, >>>>>> * Issues related cross-language transforms wrappers written in SDK Foo >>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>> offered as cross-language transforms to other SDKs >>>>>> * Issues related to cross-language transform expansion service >>>>>> implemented for SDK Foo >>>>>> >>>>>> Thanks, >>>>>> Cham >>>>>> >>>>>
