Re: JIRA components
The component would be for Python type related code. Mostly the stuff under sdks/python/apache_beam/typehints. Thanks, I'll use labels. On Fri, Sep 18, 2020 at 10:00 AM Kyle Weaver wrote: > Why not use the existing sdk-py-core component and make Python types a > label? > > On Thu, Sep 17, 2020 at 7:17 PM Kenneth Knowles wrote: > >> What does it mean? >> >> Kenn >> >> On Thu, Sep 17, 2020 at 6:18 PM Udi Meiri wrote: >> >>> Hi, I was going to create a Python types component in our JIRA. >>> (sdk-py-types) >>> Any objections? >>> >> smime.p7s Description: S/MIME Cryptographic Signature
Re: JIRA components
Why not use the existing sdk-py-core component and make Python types a label? On Thu, Sep 17, 2020 at 7:17 PM Kenneth Knowles wrote: > What does it mean? > > Kenn > > On Thu, Sep 17, 2020 at 6:18 PM Udi Meiri wrote: > >> Hi, I was going to create a Python types component in our JIRA. >> (sdk-py-types) >> Any objections? >> >
Re: JIRA components
What does it mean? Kenn On Thu, Sep 17, 2020 at 6:18 PM Udi Meiri wrote: > Hi, I was going to create a Python types component in our JIRA. > (sdk-py-types) > Any objections? >
JIRA components
Hi, I was going to create a Python types component in our JIRA. (sdk-py-types) Any objections? smime.p7s Description: S/MIME Cryptographic Signature
Re: Jira components for cross-language transforms
t;> aromanenko@gmail.com> wrote: >>>>>>> >>>>>>>> +1 to new non split component. >>>>>>>> >>>>>>>> On 29 May 2020, at 07:19, Heejong Lee wrote: >>>>>>>> >>>>>>>> If we use one meta component tag for all xlang related issues, I >>>>>>>> would prefer just "xlang". Then we could attach the "xlang" tag to not >>>>>>>> only >>>>>>>> language specific sdk tags but also other runner tags e.g. ['xlang', >>>>>>>> 'io-java-kafka'], ['xlang'', 'runner-dataflow']. >>>>>>>> >>>>>>>> On Thu, May 28, 2020 at 7:49 PM Robert Burke >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 to new component not split. The language concerns can be >>>>>>>>> represented and filtered with the existing sdk tags. I know I'm >>>>>>>>> interested >>>>>>>>> in all sdk-go issues, and would prefer not to have to union tags when >>>>>>>>> searching for Go related issues. >>>>>>>>> >>>>>>>>> On Thu, 28 May 2020 at 15:48, Ismaël Mejía >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 to new component not splitted >>>>>>>>>> >>>>>>>>>> Other use case is using libraries not available in your language >>>>>>>>>> e.g. using some python transform that relies in a python only API in >>>>>>>>>> the >>>>>>>>>> middle of a Java pipeline. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath < >>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> I proposed three components since the audience might be >>>>>>>>>>> different. Also we can use the same component to track issues >>>>>>>>>>> related to >>>>>>>>>>> all cross-language wrappers available in a given SDK. If this is >>>>>>>>>>> too much a >>>>>>>>>>> single component is fine as well. >>>>>>>>>>> >>>>>>>>>>> Ashwin, as others pointed out, the cross-language transforms >>>>>>>>>>> framework is primarily for giving SDKs access to transforms that >>>>>>>>>>> are not >>>>>>>>>>> available natively. But there are other potential use-cases as well >>>>>>>>>>> (for >>>>>>>>>>> example, using two different Python environments within the same >>>>>>>>>>> pipeline). >>>>>>>>>>> Exact performance will depend on the runner implementation as >>>>>>>>>>> well as the additional cost involved due to >>>>>>>>>>> serializing/deserializing data >>>>>>>>>>> across environment boundaries. But we haven't done enough >>>>>>>>>>> analysis/benchmarking to provide more details on this. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Cham >>>>>>>>>>> >>>>>>>>>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> > What are some of the benefits / drawbacks of using >>>>>>>>>>>> cross-language transforms? Would a native Python transform perform >>>>>>>>>>>> better >>>>>>>>>>>> than a cross-language transform written in Java that is then used >>>>>>>>>>>> in a >>>>>>>>>>>> Python pipeline? >>>>>>>>>>>> >>>>>>>>>>>> As Rui says, the main advantage is code reuse. See >>>>>>>>>>>>
Re: Jira components for cross-language transforms
; in all sdk-go issues, and would prefer not to have to union tags when >>>>>>>> searching for Go related issues. >>>>>>>> >>>>>>>> On Thu, 28 May 2020 at 15:48, Ismaël Mejía >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 to new component not splitted >>>>>>>>> >>>>>>>>> Other use case is using libraries not available in your language >>>>>>>>> e.g. using some python transform that relies in a python only API in >>>>>>>>> the >>>>>>>>> middle of a Java pipeline. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath < >>>>>>>>> chamik...@google.com> wrote: >>>>>>>>> >>>>>>>>>> I proposed three components since the audience might be >>>>>>>>>> different. Also we can use the same component to track issues >>>>>>>>>> related to >>>>>>>>>> all cross-language wrappers available in a given SDK. If this is too >>>>>>>>>> much a >>>>>>>>>> single component is fine as well. >>>>>>>>>> >>>>>>>>>> Ashwin, as others pointed out, the cross-language transforms >>>>>>>>>> framework is primarily for giving SDKs access to transforms that are >>>>>>>>>> not >>>>>>>>>> available natively. But there are other potential use-cases as well >>>>>>>>>> (for >>>>>>>>>> example, using two different Python environments within the same >>>>>>>>>> pipeline). >>>>>>>>>> Exact performance will depend on the runner implementation as >>>>>>>>>> well as the additional cost involved due to >>>>>>>>>> serializing/deserializing data >>>>>>>>>> across environment boundaries. But we haven't done enough >>>>>>>>>> analysis/benchmarking to provide more details on this. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Cham >>>>>>>>>> >>>>>>>>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> > What are some of the benefits / drawbacks of using >>>>>>>>>>> cross-language transforms? Would a native Python transform perform >>>>>>>>>>> better >>>>>>>>>>> than a cross-language transform written in Java that is then used >>>>>>>>>>> in a >>>>>>>>>>> Python pipeline? >>>>>>>>>>> >>>>>>>>>>> As Rui says, the main advantage is code reuse. See >>>>>>>>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>>>>>>>> information. >>>>>>>>>>> >>>>>>>>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 on dedicated components for cross-language transform. It >>>>>>>>>>>> might be easy to manage to have one component (one tag for all >>>>>>>>>>>> SDK) rather >>>>>>>>>>>> than multiple ones. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Re Ashwin, >>>>>>>>>>>> >>>>>>>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms >>>>>>>>>>>> for Go >>>>>>>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of >>>>>>>>>>>> work. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -Rui >>>>>>>>>>>> >>>>>>>>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>>>>>>>> aramaswa...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> What are some of the benefits / drawbacks of using >>>>>>>>>>>>> cross-language transforms? Would a native Python transform >>>>>>>>>>>>> perform better >>>>>>>>>>>>> than a cross-language transform written in Java that is then used >>>>>>>>>>>>> in a >>>>>>>>>>>>> Python pipeline? >>>>>>>>>>>>> >>>>>>>>>>>>> Ashwin Ramaswami >>>>>>>>>>>>> Student >>>>>>>>>>>>> *Find me on my:* LinkedIn >>>>>>>>>>>>> <https://www.linkedin.com/in/ashwin-r> | Website >>>>>>>>>>>>> <https://epicfaace.github.io/> | GitHub >>>>>>>>>>>>> <https://github.com/epicfaace> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver < >>>>>>>>>>>>> kcwea...@google.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> SGTM. Though I'm not sure it's necessary to split by >>>>>>>>>>>>>> language. It might be easier to use a single cross-language tag, >>>>>>>>>>>>>> rather >>>>>>>>>>>>>> than having to tag lots of issues as both sdks-python-xlang and >>>>>>>>>>>>>> sdks-java-xlang. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think it's good if we can have new Jira components to >>>>>>>>>>>>>>> easily track various issues related to cross-language >>>>>>>>>>>>>>> transforms. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do you think about adding the following Jira components >>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sdks-python-xlang >>>>>>>>>>>>>>> sdks-java-xlang >>>>>>>>>>>>>>> sdks-go-xlang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jira component sdks-foo-xlang is for tracking issues related >>>>>>>>>>>>>>> to cross-language transforms for SDK Foo. For example, >>>>>>>>>>>>>>> * Issues related cross-language transforms wrappers written >>>>>>>>>>>>>>> in SDK Foo >>>>>>>>>>>>>>> * Issues related to transforms implemented in SDK Foo that >>>>>>>>>>>>>>> are offered as cross-language transforms to other SDKs >>>>>>>>>>>>>>> * Issues related to cross-language transform expansion >>>>>>>>>>>>>>> service implemented for SDK Foo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Cham >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>
Re: Jira components for cross-language transforms
appers available in a given SDK. If this is too much >>>>>>>>> a >>>>>>>>> single component is fine as well. >>>>>>>>> >>>>>>>>> Ashwin, as others pointed out, the cross-language transforms >>>>>>>>> framework is primarily for giving SDKs access to transforms that are >>>>>>>>> not >>>>>>>>> available natively. But there are other potential use-cases as well >>>>>>>>> (for >>>>>>>>> example, using two different Python environments within the same >>>>>>>>> pipeline). >>>>>>>>> Exact performance will depend on the runner implementation as well >>>>>>>>> as the additional cost involved due to serializing/deserializing data >>>>>>>>> across environment boundaries. But we haven't done enough >>>>>>>>> analysis/benchmarking to provide more details on this. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Cham >>>>>>>>> >>>>>>>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> > What are some of the benefits / drawbacks of using >>>>>>>>>> cross-language transforms? Would a native Python transform perform >>>>>>>>>> better >>>>>>>>>> than a cross-language transform written in Java that is then used in >>>>>>>>>> a >>>>>>>>>> Python pipeline? >>>>>>>>>> >>>>>>>>>> As Rui says, the main advantage is code reuse. See >>>>>>>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>>>>>>> information. >>>>>>>>>> >>>>>>>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +1 on dedicated components for cross-language transform. It >>>>>>>>>>> might be easy to manage to have one component (one tag for all SDK) >>>>>>>>>>> rather >>>>>>>>>>> than multiple ones. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Re Ashwin, >>>>>>>>>>> >>>>>>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for >>>>>>>>>>> Go >>>>>>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of >>>>>>>>>>> work. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -Rui >>>>>>>>>>> >>>>>>>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>>>>>>> aramaswa...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> What are some of the benefits / drawbacks of using >>>>>>>>>>>> cross-language transforms? Would a native Python transform perform >>>>>>>>>>>> better >>>>>>>>>>>> than a cross-language transform written in Java that is then used >>>>>>>>>>>> in a >>>>>>>>>>>> Python pipeline? >>>>>>>>>>>> >>>>>>>>>>>> Ashwin Ramaswami >>>>>>>>>>>> Student >>>>>>>>>>>> *Find me on my:* LinkedIn >>>>>>>>>>>> <https://www.linkedin.com/in/ashwin-r> | Website >>>>>>>>>>>> <https://epicfaace.github.io/> | GitHub >>>>>>>>>>>> <https://github.com/epicfaace> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver < >>>>>>>>>>>> kcwea...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> SGTM. Though I'm not sure it's necessary to split by language. >>>>>>>>>>>>> It might be easier to use a single cross-language tag, rather >>>>>>>>>>>>> than having >>>>>>>>>>>>> to tag lots of issues as both sdks-python-xlang and >>>>>>>>>>>>> sdks-java-xlang. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think it's good if we can have new Jira components to >>>>>>>>>>>>>> easily track various issues related to cross-language transforms. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you think about adding the following Jira components ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> sdks-python-xlang >>>>>>>>>>>>>> sdks-java-xlang >>>>>>>>>>>>>> sdks-go-xlang >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jira component sdks-foo-xlang is for tracking issues related >>>>>>>>>>>>>> to cross-language transforms for SDK Foo. For example, >>>>>>>>>>>>>> * Issues related cross-language transforms wrappers written >>>>>>>>>>>>>> in SDK Foo >>>>>>>>>>>>>> * Issues related to transforms implemented in SDK Foo that >>>>>>>>>>>>>> are offered as cross-language transforms to other SDKs >>>>>>>>>>>>>> * Issues related to cross-language transform expansion >>>>>>>>>>>>>> service implemented for SDK Foo >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Cham >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>
Re: Jira components for cross-language transforms
t;>> than a cross-language transform written in Java that is then used in a >>>>>>>>> Python pipeline? >>>>>>>>> >>>>>>>>> As Rui says, the main advantage is code reuse. See >>>>>>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>>>>>> information. >>>>>>>>> >>>>>>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 on dedicated components for cross-language transform. It might >>>>>>>>>> be easy to manage to have one component (one tag for all SDK) rather >>>>>>>>>> than >>>>>>>>>> multiple ones. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Re Ashwin, >>>>>>>>>> >>>>>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for >>>>>>>>>> Go >>>>>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of >>>>>>>>>> work. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -Rui >>>>>>>>>> >>>>>>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>>>>>> aramaswa...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> What are some of the benefits / drawbacks of using >>>>>>>>>>> cross-language transforms? Would a native Python transform perform >>>>>>>>>>> better >>>>>>>>>>> than a cross-language transform written in Java that is then used >>>>>>>>>>> in a >>>>>>>>>>> Python pipeline? >>>>>>>>>>> >>>>>>>>>>> Ashwin Ramaswami >>>>>>>>>>> Student >>>>>>>>>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>>>>>>>>> Website <https://epicfaace.github.io/> | GitHub >>>>>>>>>>> <https://github.com/epicfaace> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> SGTM. Though I'm not sure it's necessary to split by language. >>>>>>>>>>>> It might be easier to use a single cross-language tag, rather than >>>>>>>>>>>> having >>>>>>>>>>>> to tag lots of issues as both sdks-python-xlang and >>>>>>>>>>>> sdks-java-xlang. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> I think it's good if we can have new Jira components to easily >>>>>>>>>>>>> track various issues related to cross-language transforms. >>>>>>>>>>>>> >>>>>>>>>>>>> What do you think about adding the following Jira components ? >>>>>>>>>>>>> >>>>>>>>>>>>> sdks-python-xlang >>>>>>>>>>>>> sdks-java-xlang >>>>>>>>>>>>> sdks-go-xlang >>>>>>>>>>>>> >>>>>>>>>>>>> Jira component sdks-foo-xlang is for tracking issues related >>>>>>>>>>>>> to cross-language transforms for SDK Foo. For example, >>>>>>>>>>>>> * Issues related cross-language transforms wrappers written in >>>>>>>>>>>>> SDK Foo >>>>>>>>>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>>>>>>>>> offered as cross-language transforms to other SDKs >>>>>>>>>>>>> * Issues related to cross-language transform expansion service >>>>>>>>>>>>> implemented for SDK Foo >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Cham >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>
Re: Jira components for cross-language transforms
ed components for cross-language transform. It might >>>>>>>>> be easy to manage to have one component (one tag for all SDK) rather >>>>>>>>> than >>>>>>>>> multiple ones. >>>>>>>>> >>>>>>>>> >>>>>>>>> Re Ashwin, >>>>>>>>> >>>>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for Go >>>>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of work. >>>>>>>>> >>>>>>>>> >>>>>>>>> -Rui >>>>>>>>> >>>>>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>>>>> aramaswa...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> What are some of the benefits / drawbacks of using cross-language >>>>>>>>>> transforms? Would a native Python transform perform better than a >>>>>>>>>> cross-language transform written in Java that is then used in a >>>>>>>>>> Python >>>>>>>>>> pipeline? >>>>>>>>>> >>>>>>>>>> Ashwin Ramaswami >>>>>>>>>> Student >>>>>>>>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>>>>>>>> Website <https://epicfaace.github.io/> | GitHub >>>>>>>>>> <https://github.com/epicfaace> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> SGTM. Though I'm not sure it's necessary to split by language. >>>>>>>>>>> It might be easier to use a single cross-language tag, rather than >>>>>>>>>>> having >>>>>>>>>>> to tag lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>>>>>>>>> >>>>>>>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> I think it's good if we can have new Jira components to easily >>>>>>>>>>>> track various issues related to cross-language transforms. >>>>>>>>>>>> >>>>>>>>>>>> What do you think about adding the following Jira components ? >>>>>>>>>>>> >>>>>>>>>>>> sdks-python-xlang >>>>>>>>>>>> sdks-java-xlang >>>>>>>>>>>> sdks-go-xlang >>>>>>>>>>>> >>>>>>>>>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>>>>>>>>> cross-language transforms for SDK Foo. For example, >>>>>>>>>>>> * Issues related cross-language transforms wrappers written in >>>>>>>>>>>> SDK Foo >>>>>>>>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>>>>>>>> offered as cross-language transforms to other SDKs >>>>>>>>>>>> * Issues related to cross-language transform expansion service >>>>>>>>>>>> implemented for SDK Foo >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Cham >>>>>>>>>>>> >>>>>>>>>>> >>>>
Re: Jira components for cross-language transforms
Good point. "cross-language" sgtm. On Fri, May 29, 2020, 8:57 AM Kyle Weaver wrote: > Nit: can we name it `cross-language` instead of `xlang`? Component names > auto-complete, so there's no reason to abbreviate. > > On Fri, May 29, 2020 at 11:54 AM Chamikara Jayalath > wrote: > >> Thanks for the comments. >> >> Can someone please create a single Jira component named "xlang" ? Looks >> like I don't have access to create components. >> >> Thanks, >> Cham >> >> On Fri, May 29, 2020 at 7:24 AM Alexey Romanenko < >> aromanenko@gmail.com> wrote: >> >>> +1 to new non split component. >>> >>> On 29 May 2020, at 07:19, Heejong Lee wrote: >>> >>> If we use one meta component tag for all xlang related issues, I would >>> prefer just "xlang". Then we could attach the "xlang" tag to not only >>> language specific sdk tags but also other runner tags e.g. ['xlang', >>> 'io-java-kafka'], ['xlang'', 'runner-dataflow']. >>> >>> On Thu, May 28, 2020 at 7:49 PM Robert Burke wrote: >>> >>>> +1 to new component not split. The language concerns can be represented >>>> and filtered with the existing sdk tags. I know I'm interested in all >>>> sdk-go issues, and would prefer not to have to union tags when searching >>>> for Go related issues. >>>> >>>> On Thu, 28 May 2020 at 15:48, Ismaël Mejía wrote: >>>> >>>>> +1 to new component not splitted >>>>> >>>>> Other use case is using libraries not available in your language e.g. >>>>> using some python transform that relies in a python only API in the middle >>>>> of a Java pipeline. >>>>> >>>>> >>>>> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>> I proposed three components since the audience might be different. >>>>>> Also we can use the same component to track issues related to all >>>>>> cross-language wrappers available in a given SDK. If this is too much a >>>>>> single component is fine as well. >>>>>> >>>>>> Ashwin, as others pointed out, the cross-language transforms >>>>>> framework is primarily for giving SDKs access to transforms that are not >>>>>> available natively. But there are other potential use-cases as well (for >>>>>> example, using two different Python environments within the same >>>>>> pipeline). >>>>>> Exact performance will depend on the runner implementation as well as >>>>>> the additional cost involved due to serializing/deserializing data across >>>>>> environment boundaries. But we haven't done enough analysis/benchmarking >>>>>> to >>>>>> provide more details on this. >>>>>> >>>>>> Thanks, >>>>>> Cham >>>>>> >>>>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>>>> wrote: >>>>>> >>>>>>> > What are some of the benefits / drawbacks of using cross-language >>>>>>> transforms? Would a native Python transform perform better than a >>>>>>> cross-language transform written in Java that is then used in a Python >>>>>>> pipeline? >>>>>>> >>>>>>> As Rui says, the main advantage is code reuse. See >>>>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>>>> information. >>>>>>> >>>>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >>>>>>> >>>>>>>> +1 on dedicated components for cross-language transform. It might >>>>>>>> be easy to manage to have one component (one tag for all SDK) rather >>>>>>>> than >>>>>>>> multiple ones. >>>>>>>> >>>>>>>> >>>>>>>> Re Ashwin, >>>>>>>> >>>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for Go >>>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of work. >>>
Re: Jira components for cross-language transforms
Nit: can we name it `cross-language` instead of `xlang`? Component names auto-complete, so there's no reason to abbreviate. On Fri, May 29, 2020 at 11:54 AM Chamikara Jayalath wrote: > Thanks for the comments. > > Can someone please create a single Jira component named "xlang" ? Looks > like I don't have access to create components. > > Thanks, > Cham > > On Fri, May 29, 2020 at 7:24 AM Alexey Romanenko > wrote: > >> +1 to new non split component. >> >> On 29 May 2020, at 07:19, Heejong Lee wrote: >> >> If we use one meta component tag for all xlang related issues, I would >> prefer just "xlang". Then we could attach the "xlang" tag to not only >> language specific sdk tags but also other runner tags e.g. ['xlang', >> 'io-java-kafka'], ['xlang'', 'runner-dataflow']. >> >> On Thu, May 28, 2020 at 7:49 PM Robert Burke wrote: >> >>> +1 to new component not split. The language concerns can be represented >>> and filtered with the existing sdk tags. I know I'm interested in all >>> sdk-go issues, and would prefer not to have to union tags when searching >>> for Go related issues. >>> >>> On Thu, 28 May 2020 at 15:48, Ismaël Mejía wrote: >>> >>>> +1 to new component not splitted >>>> >>>> Other use case is using libraries not available in your language e.g. >>>> using some python transform that relies in a python only API in the middle >>>> of a Java pipeline. >>>> >>>> >>>> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath < >>>> chamik...@google.com> wrote: >>>> >>>>> I proposed three components since the audience might be different. >>>>> Also we can use the same component to track issues related to all >>>>> cross-language wrappers available in a given SDK. If this is too much a >>>>> single component is fine as well. >>>>> >>>>> Ashwin, as others pointed out, the cross-language transforms framework >>>>> is primarily for giving SDKs access to transforms that are not >>>>> available natively. But there are other potential use-cases as well (for >>>>> example, using two different Python environments within the same >>>>> pipeline). >>>>> Exact performance will depend on the runner implementation as well as >>>>> the additional cost involved due to serializing/deserializing data across >>>>> environment boundaries. But we haven't done enough analysis/benchmarking >>>>> to >>>>> provide more details on this. >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>>> wrote: >>>>> >>>>>> > What are some of the benefits / drawbacks of using cross-language >>>>>> transforms? Would a native Python transform perform better than a >>>>>> cross-language transform written in Java that is then used in a Python >>>>>> pipeline? >>>>>> >>>>>> As Rui says, the main advantage is code reuse. See >>>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>>> information. >>>>>> >>>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >>>>>> >>>>>>> +1 on dedicated components for cross-language transform. It might be >>>>>>> easy to manage to have one component (one tag for all SDK) rather than >>>>>>> multiple ones. >>>>>>> >>>>>>> >>>>>>> Re Ashwin, >>>>>>> >>>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for Go >>>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of work. >>>>>>> >>>>>>> >>>>>>> -Rui >>>>>>> >>>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>>> aramaswa...@gmail.com> wrote: >>>>>>> >>>>>>>> What are some of the benefits / drawbacks of using cross-language >>>>>>>> transforms? Would a native Python transform perform better than a >>>>>>>> cross-language transform writt
Re: Jira components for cross-language transforms
Thanks for the comments. Can someone please create a single Jira component named "xlang" ? Looks like I don't have access to create components. Thanks, Cham On Fri, May 29, 2020 at 7:24 AM Alexey Romanenko wrote: > +1 to new non split component. > > On 29 May 2020, at 07:19, Heejong Lee wrote: > > If we use one meta component tag for all xlang related issues, I would > prefer just "xlang". Then we could attach the "xlang" tag to not only > language specific sdk tags but also other runner tags e.g. ['xlang', > 'io-java-kafka'], ['xlang'', 'runner-dataflow']. > > On Thu, May 28, 2020 at 7:49 PM Robert Burke wrote: > >> +1 to new component not split. The language concerns can be represented >> and filtered with the existing sdk tags. I know I'm interested in all >> sdk-go issues, and would prefer not to have to union tags when searching >> for Go related issues. >> >> On Thu, 28 May 2020 at 15:48, Ismaël Mejía wrote: >> >>> +1 to new component not splitted >>> >>> Other use case is using libraries not available in your language e.g. >>> using some python transform that relies in a python only API in the middle >>> of a Java pipeline. >>> >>> >>> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath < >>> chamik...@google.com> wrote: >>> >>>> I proposed three components since the audience might be different. Also >>>> we can use the same component to track issues related to all cross-language >>>> wrappers available in a given SDK. If this is too much a single component >>>> is fine as well. >>>> >>>> Ashwin, as others pointed out, the cross-language transforms framework >>>> is primarily for giving SDKs access to transforms that are not >>>> available natively. But there are other potential use-cases as well (for >>>> example, using two different Python environments within the same >>>> pipeline). >>>> Exact performance will depend on the runner implementation as well as >>>> the additional cost involved due to serializing/deserializing data across >>>> environment boundaries. But we haven't done enough analysis/benchmarking to >>>> provide more details on this. >>>> >>>> Thanks, >>>> Cham >>>> >>>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver >>>> wrote: >>>> >>>>> > What are some of the benefits / drawbacks of using cross-language >>>>> transforms? Would a native Python transform perform better than a >>>>> cross-language transform written in Java that is then used in a Python >>>>> pipeline? >>>>> >>>>> As Rui says, the main advantage is code reuse. See >>>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>>> information. >>>>> >>>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >>>>> >>>>>> +1 on dedicated components for cross-language transform. It might be >>>>>> easy to manage to have one component (one tag for all SDK) rather than >>>>>> multiple ones. >>>>>> >>>>>> >>>>>> Re Ashwin, >>>>>> >>>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for Go >>>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of work. >>>>>> >>>>>> >>>>>> -Rui >>>>>> >>>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>>> aramaswa...@gmail.com> wrote: >>>>>> >>>>>>> What are some of the benefits / drawbacks of using cross-language >>>>>>> transforms? Would a native Python transform perform better than a >>>>>>> cross-language transform written in Java that is then used in a Python >>>>>>> pipeline? >>>>>>> >>>>>>> Ashwin Ramaswami >>>>>>> Student >>>>>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>>>>> Website <https://epicfaace.github.io/> | GitHub >>>>>>> <https://github.com/epicfaace> >>>>>>> >>>>>>> >>>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >&
Re: Jira components for cross-language transforms
+1 to new non split component. > On 29 May 2020, at 07:19, Heejong Lee wrote: > > If we use one meta component tag for all xlang related issues, I would prefer > just "xlang". Then we could attach the "xlang" tag to not only language > specific sdk tags but also other runner tags e.g. ['xlang', 'io-java-kafka'], > ['xlang'', 'runner-dataflow']. > > On Thu, May 28, 2020 at 7:49 PM Robert Burke <mailto:rob...@frantil.com>> wrote: > +1 to new component not split. The language concerns can be represented and > filtered with the existing sdk tags. I know I'm interested in all sdk-go > issues, and would prefer not to have to union tags when searching for Go > related issues. > > On Thu, 28 May 2020 at 15:48, Ismaël Mejía <mailto:ieme...@gmail.com>> wrote: > +1 to new component not splitted > > Other use case is using libraries not available in your language e.g. using > some python transform that relies in a python only API in the middle of a > Java pipeline. > > > On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath <mailto:chamik...@google.com>> wrote: > I proposed three components since the audience might be different. Also we > can use the same component to track issues related to all cross-language > wrappers available in a given SDK. If this is too much a single component is > fine as well. > > Ashwin, as others pointed out, the cross-language transforms framework is > primarily for giving SDKs access to transforms that are not available > natively. But there are other potential use-cases as well (for example, using > two different Python environments within the same pipeline). > Exact performance will depend on the runner implementation as well as the > additional cost involved due to serializing/deserializing data across > environment boundaries. But we haven't done enough analysis/benchmarking to > provide more details on this. > > Thanks, > Cham > > On Thu, May 28, 2020 at 1:55 PM Kyle Weaver <mailto:kcwea...@google.com>> wrote: > > What are some of the benefits / drawbacks of using cross-language > > transforms? Would a native Python transform perform better than a > > cross-language transform written in Java that is then used in a Python > > pipeline? > > As Rui says, the main advantage is code reuse. See > https://beam.apache.org/roadmap/connectors-multi-sdk/ > <https://beam.apache.org/roadmap/connectors-multi-sdk/> for more information. > > On Thu, May 28, 2020 at 4:53 PM Rui Wang <mailto:ruw...@google.com>> wrote: > +1 on dedicated components for cross-language transform. It might be easy to > manage to have one component (one tag for all SDK) rather than multiple ones. > > > Re Ashwin, > > Cham knows more than me. AFAIK, cross-language transforms will maximize code > reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of course, a > SDK can develop its own IOs, but it's lots of work. > > > -Rui > > On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami <mailto:aramaswa...@gmail.com>> wrote: > What are some of the benefits / drawbacks of using cross-language transforms? > Would a native Python transform perform better than a cross-language > transform written in Java that is then used in a Python pipeline? > > Ashwin Ramaswami > Student > Find me on my: LinkedIn <https://www.linkedin.com/in/ashwin-r> | Website > <https://epicfaace.github.io/> | GitHub <https://github.com/epicfaace> > > On Thu, May 28, 2020 at 4:44 PM Kyle Weaver <mailto:kcwea...@google.com>> wrote: > SGTM. Though I'm not sure it's necessary to split by language. It might be > easier to use a single cross-language tag, rather than having to tag lots of > issues as both sdks-python-xlang and sdks-java-xlang. > > On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath <mailto:chamik...@google.com>> wrote: > Hi All, > > I think it's good if we can have new Jira components to easily track various > issues related to cross-language transforms. > > What do you think about adding the following Jira components ? > > sdks-python-xlang > sdks-java-xlang > sdks-go-xlang > > Jira component sdks-foo-xlang is for tracking issues related to > cross-language transforms for SDK Foo. For example, > * Issues related cross-language transforms wrappers written in SDK Foo > * Issues related to transforms implemented in SDK Foo that are offered as > cross-language transforms to other SDKs > * Issues related to cross-language transform expansion service implemented > for SDK Foo > > Thanks, > Cham
Re: Jira components for cross-language transforms
If we use one meta component tag for all xlang related issues, I would prefer just "xlang". Then we could attach the "xlang" tag to not only language specific sdk tags but also other runner tags e.g. ['xlang', 'io-java-kafka'], ['xlang'', 'runner-dataflow']. On Thu, May 28, 2020 at 7:49 PM Robert Burke wrote: > +1 to new component not split. The language concerns can be represented > and filtered with the existing sdk tags. I know I'm interested in all > sdk-go issues, and would prefer not to have to union tags when searching > for Go related issues. > > On Thu, 28 May 2020 at 15:48, Ismaël Mejía wrote: > >> +1 to new component not splitted >> >> Other use case is using libraries not available in your language e.g. >> using some python transform that relies in a python only API in the middle >> of a Java pipeline. >> >> >> On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath >> wrote: >> >>> I proposed three components since the audience might be different. Also >>> we can use the same component to track issues related to all cross-language >>> wrappers available in a given SDK. If this is too much a single component >>> is fine as well. >>> >>> Ashwin, as others pointed out, the cross-language transforms framework >>> is primarily for giving SDKs access to transforms that are not >>> available natively. But there are other potential use-cases as well (for >>> example, using two different Python environments within the same >>> pipeline). >>> Exact performance will depend on the runner implementation as well as >>> the additional cost involved due to serializing/deserializing data across >>> environment boundaries. But we haven't done enough analysis/benchmarking to >>> provide more details on this. >>> >>> Thanks, >>> Cham >>> >>> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver wrote: >>> >>>> > What are some of the benefits / drawbacks of using cross-language >>>> transforms? Would a native Python transform perform better than a >>>> cross-language transform written in Java that is then used in a Python >>>> pipeline? >>>> >>>> As Rui says, the main advantage is code reuse. See >>>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>>> information. >>>> >>>> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >>>> >>>>> +1 on dedicated components for cross-language transform. It might be >>>>> easy to manage to have one component (one tag for all SDK) rather than >>>>> multiple ones. >>>>> >>>>> >>>>> Re Ashwin, >>>>> >>>>> Cham knows more than me. AFAIK, cross-language transforms will >>>>> maximize code reuse for newly developed SDK (e.g. IO transforms for Go >>>>> SDK). Of course, a SDK can develop its own IOs, but it's lots of work. >>>>> >>>>> >>>>> -Rui >>>>> >>>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami < >>>>> aramaswa...@gmail.com> wrote: >>>>> >>>>>> What are some of the benefits / drawbacks of using cross-language >>>>>> transforms? Would a native Python transform perform better than a >>>>>> cross-language transform written in Java that is then used in a Python >>>>>> pipeline? >>>>>> >>>>>> Ashwin Ramaswami >>>>>> Student >>>>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>>>> Website <https://epicfaace.github.io/> | GitHub >>>>>> <https://github.com/epicfaace> >>>>>> >>>>>> >>>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >>>>>> wrote: >>>>>> >>>>>>> SGTM. Though I'm not sure it's necessary to split by language. It >>>>>>> might be easier to use a single cross-language tag, rather than having >>>>>>> to >>>>>>> tag lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>>>>> >>>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>>> chamik...@google.com> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I think it's good if we can have new Jira components to easily >>>>>>>> track various issues related to cross-language transforms. >>>>>>>> >>>>>>>> What do you think about adding the following Jira components ? >>>>>>>> >>>>>>>> sdks-python-xlang >>>>>>>> sdks-java-xlang >>>>>>>> sdks-go-xlang >>>>>>>> >>>>>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>>>>> cross-language transforms for SDK Foo. For example, >>>>>>>> * Issues related cross-language transforms wrappers written in SDK >>>>>>>> Foo >>>>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>>>> offered as cross-language transforms to other SDKs >>>>>>>> * Issues related to cross-language transform expansion service >>>>>>>> implemented for SDK Foo >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Cham >>>>>>>> >>>>>>>
Re: Jira components for cross-language transforms
+1 to new component not split. The language concerns can be represented and filtered with the existing sdk tags. I know I'm interested in all sdk-go issues, and would prefer not to have to union tags when searching for Go related issues. On Thu, 28 May 2020 at 15:48, Ismaël Mejía wrote: > +1 to new component not splitted > > Other use case is using libraries not available in your language e.g. > using some python transform that relies in a python only API in the middle > of a Java pipeline. > > > On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath > wrote: > >> I proposed three components since the audience might be different. Also >> we can use the same component to track issues related to all cross-language >> wrappers available in a given SDK. If this is too much a single component >> is fine as well. >> >> Ashwin, as others pointed out, the cross-language transforms framework is >> primarily for giving SDKs access to transforms that are not >> available natively. But there are other potential use-cases as well (for >> example, using two different Python environments within the same >> pipeline). >> Exact performance will depend on the runner implementation as well as the >> additional cost involved due to serializing/deserializing data across >> environment boundaries. But we haven't done enough analysis/benchmarking to >> provide more details on this. >> >> Thanks, >> Cham >> >> On Thu, May 28, 2020 at 1:55 PM Kyle Weaver wrote: >> >>> > What are some of the benefits / drawbacks of using cross-language >>> transforms? Would a native Python transform perform better than a >>> cross-language transform written in Java that is then used in a Python >>> pipeline? >>> >>> As Rui says, the main advantage is code reuse. See >>> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >>> information. >>> >>> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >>> >>>> +1 on dedicated components for cross-language transform. It might be >>>> easy to manage to have one component (one tag for all SDK) rather than >>>> multiple ones. >>>> >>>> >>>> Re Ashwin, >>>> >>>> Cham knows more than me. AFAIK, cross-language transforms will maximize >>>> code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of >>>> course, a SDK can develop its own IOs, but it's lots of work. >>>> >>>> >>>> -Rui >>>> >>>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami >>>> wrote: >>>> >>>>> What are some of the benefits / drawbacks of using cross-language >>>>> transforms? Would a native Python transform perform better than a >>>>> cross-language transform written in Java that is then used in a Python >>>>> pipeline? >>>>> >>>>> Ashwin Ramaswami >>>>> Student >>>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>>> Website <https://epicfaace.github.io/> | GitHub >>>>> <https://github.com/epicfaace> >>>>> >>>>> >>>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >>>>> wrote: >>>>> >>>>>> SGTM. Though I'm not sure it's necessary to split by language. It >>>>>> might be easier to use a single cross-language tag, rather than having to >>>>>> tag lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>>>> >>>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>>> chamik...@google.com> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I think it's good if we can have new Jira components to easily track >>>>>>> various issues related to cross-language transforms. >>>>>>> >>>>>>> What do you think about adding the following Jira components ? >>>>>>> >>>>>>> sdks-python-xlang >>>>>>> sdks-java-xlang >>>>>>> sdks-go-xlang >>>>>>> >>>>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>>>> cross-language transforms for SDK Foo. For example, >>>>>>> * Issues related cross-language transforms wrappers written in SDK >>>>>>> Foo >>>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>>> offered as cross-language transforms to other SDKs >>>>>>> * Issues related to cross-language transform expansion service >>>>>>> implemented for SDK Foo >>>>>>> >>>>>>> Thanks, >>>>>>> Cham >>>>>>> >>>>>>
Re: Jira components for cross-language transforms
+1 to new component not splitted Other use case is using libraries not available in your language e.g. using some python transform that relies in a python only API in the middle of a Java pipeline. On Thu, May 28, 2020 at 11:12 PM Chamikara Jayalath wrote: > I proposed three components since the audience might be different. Also we > can use the same component to track issues related to all cross-language > wrappers available in a given SDK. If this is too much a single component > is fine as well. > > Ashwin, as others pointed out, the cross-language transforms framework is > primarily for giving SDKs access to transforms that are not > available natively. But there are other potential use-cases as well (for > example, using two different Python environments within the same > pipeline). > Exact performance will depend on the runner implementation as well as the > additional cost involved due to serializing/deserializing data across > environment boundaries. But we haven't done enough analysis/benchmarking to > provide more details on this. > > Thanks, > Cham > > On Thu, May 28, 2020 at 1:55 PM Kyle Weaver wrote: > >> > What are some of the benefits / drawbacks of using cross-language >> transforms? Would a native Python transform perform better than a >> cross-language transform written in Java that is then used in a Python >> pipeline? >> >> As Rui says, the main advantage is code reuse. See >> https://beam.apache.org/roadmap/connectors-multi-sdk/ for more >> information. >> >> On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: >> >>> +1 on dedicated components for cross-language transform. It might be >>> easy to manage to have one component (one tag for all SDK) rather than >>> multiple ones. >>> >>> >>> Re Ashwin, >>> >>> Cham knows more than me. AFAIK, cross-language transforms will maximize >>> code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of >>> course, a SDK can develop its own IOs, but it's lots of work. >>> >>> >>> -Rui >>> >>> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami >>> wrote: >>> >>>> What are some of the benefits / drawbacks of using cross-language >>>> transforms? Would a native Python transform perform better than a >>>> cross-language transform written in Java that is then used in a Python >>>> pipeline? >>>> >>>> Ashwin Ramaswami >>>> Student >>>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>>> Website <https://epicfaace.github.io/> | GitHub >>>> <https://github.com/epicfaace> >>>> >>>> >>>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver >>>> wrote: >>>> >>>>> SGTM. Though I'm not sure it's necessary to split by language. It >>>>> might be easier to use a single cross-language tag, rather than having to >>>>> tag lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>>> >>>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I think it's good if we can have new Jira components to easily track >>>>>> various issues related to cross-language transforms. >>>>>> >>>>>> What do you think about adding the following Jira components ? >>>>>> >>>>>> sdks-python-xlang >>>>>> sdks-java-xlang >>>>>> sdks-go-xlang >>>>>> >>>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>>> cross-language transforms for SDK Foo. For example, >>>>>> * Issues related cross-language transforms wrappers written in SDK Foo >>>>>> * Issues related to transforms implemented in SDK Foo that are >>>>>> offered as cross-language transforms to other SDKs >>>>>> * Issues related to cross-language transform expansion service >>>>>> implemented for SDK Foo >>>>>> >>>>>> Thanks, >>>>>> Cham >>>>>> >>>>>
Re: Jira components for cross-language transforms
I proposed three components since the audience might be different. Also we can use the same component to track issues related to all cross-language wrappers available in a given SDK. If this is too much a single component is fine as well. Ashwin, as others pointed out, the cross-language transforms framework is primarily for giving SDKs access to transforms that are not available natively. But there are other potential use-cases as well (for example, using two different Python environments within the same pipeline). Exact performance will depend on the runner implementation as well as the additional cost involved due to serializing/deserializing data across environment boundaries. But we haven't done enough analysis/benchmarking to provide more details on this. Thanks, Cham On Thu, May 28, 2020 at 1:55 PM Kyle Weaver wrote: > > What are some of the benefits / drawbacks of using cross-language > transforms? Would a native Python transform perform better than a > cross-language transform written in Java that is then used in a Python > pipeline? > > As Rui says, the main advantage is code reuse. See > https://beam.apache.org/roadmap/connectors-multi-sdk/ for more > information. > > On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: > >> +1 on dedicated components for cross-language transform. It might be easy >> to manage to have one component (one tag for all SDK) rather than >> multiple ones. >> >> >> Re Ashwin, >> >> Cham knows more than me. AFAIK, cross-language transforms will maximize >> code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of >> course, a SDK can develop its own IOs, but it's lots of work. >> >> >> -Rui >> >> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami >> wrote: >> >>> What are some of the benefits / drawbacks of using cross-language >>> transforms? Would a native Python transform perform better than a >>> cross-language transform written in Java that is then used in a Python >>> pipeline? >>> >>> Ashwin Ramaswami >>> Student >>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>> Website <https://epicfaace.github.io/> | GitHub >>> <https://github.com/epicfaace> >>> >>> >>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver wrote: >>> >>>> SGTM. Though I'm not sure it's necessary to split by language. It might >>>> be easier to use a single cross-language tag, rather than having to tag >>>> lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>> >>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>> chamik...@google.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> I think it's good if we can have new Jira components to easily track >>>>> various issues related to cross-language transforms. >>>>> >>>>> What do you think about adding the following Jira components ? >>>>> >>>>> sdks-python-xlang >>>>> sdks-java-xlang >>>>> sdks-go-xlang >>>>> >>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>> cross-language transforms for SDK Foo. For example, >>>>> * Issues related cross-language transforms wrappers written in SDK Foo >>>>> * Issues related to transforms implemented in SDK Foo that are >>>>> offered as cross-language transforms to other SDKs >>>>> * Issues related to cross-language transform expansion service >>>>> implemented for SDK Foo >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>
Re: Jira components for cross-language transforms
+1 to a new component. I would not split things by language. On Thu, May 28, 2020 at 1:55 PM Kyle Weaver wrote: > > What are some of the benefits / drawbacks of using cross-language > transforms? Would a native Python transform perform better than a > cross-language transform written in Java that is then used in a Python > pipeline? > > As Rui says, the main advantage is code reuse. See > https://beam.apache.org/roadmap/connectors-multi-sdk/ for more > information. > > On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: > >> +1 on dedicated components for cross-language transform. It might be easy >> to manage to have one component (one tag for all SDK) rather than >> multiple ones. >> >> >> Re Ashwin, >> >> Cham knows more than me. AFAIK, cross-language transforms will maximize >> code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of >> course, a SDK can develop its own IOs, but it's lots of work. >> >> >> -Rui >> >> On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami >> wrote: >> >>> What are some of the benefits / drawbacks of using cross-language >>> transforms? Would a native Python transform perform better than a >>> cross-language transform written in Java that is then used in a Python >>> pipeline? >>> >>> Ashwin Ramaswami >>> Student >>> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >>> Website <https://epicfaace.github.io/> | GitHub >>> <https://github.com/epicfaace> >>> >>> >>> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver wrote: >>> >>>> SGTM. Though I'm not sure it's necessary to split by language. It might >>>> be easier to use a single cross-language tag, rather than having to tag >>>> lots of issues as both sdks-python-xlang and sdks-java-xlang. >>>> >>>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath < >>>> chamik...@google.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> I think it's good if we can have new Jira components to easily track >>>>> various issues related to cross-language transforms. >>>>> >>>>> What do you think about adding the following Jira components ? >>>>> >>>>> sdks-python-xlang >>>>> sdks-java-xlang >>>>> sdks-go-xlang >>>>> >>>>> Jira component sdks-foo-xlang is for tracking issues related to >>>>> cross-language transforms for SDK Foo. For example, >>>>> * Issues related cross-language transforms wrappers written in SDK Foo >>>>> * Issues related to transforms implemented in SDK Foo that are >>>>> offered as cross-language transforms to other SDKs >>>>> * Issues related to cross-language transform expansion service >>>>> implemented for SDK Foo >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>
Re: Jira components for cross-language transforms
> What are some of the benefits / drawbacks of using cross-language transforms? Would a native Python transform perform better than a cross-language transform written in Java that is then used in a Python pipeline? As Rui says, the main advantage is code reuse. See https://beam.apache.org/roadmap/connectors-multi-sdk/ for more information. On Thu, May 28, 2020 at 4:53 PM Rui Wang wrote: > +1 on dedicated components for cross-language transform. It might be easy > to manage to have one component (one tag for all SDK) rather than > multiple ones. > > > Re Ashwin, > > Cham knows more than me. AFAIK, cross-language transforms will maximize > code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of > course, a SDK can develop its own IOs, but it's lots of work. > > > -Rui > > On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami > wrote: > >> What are some of the benefits / drawbacks of using cross-language >> transforms? Would a native Python transform perform better than a >> cross-language transform written in Java that is then used in a Python >> pipeline? >> >> Ashwin Ramaswami >> Student >> *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | >> Website <https://epicfaace.github.io/> | GitHub >> <https://github.com/epicfaace> >> >> >> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver wrote: >> >>> SGTM. Though I'm not sure it's necessary to split by language. It might >>> be easier to use a single cross-language tag, rather than having to tag >>> lots of issues as both sdks-python-xlang and sdks-java-xlang. >>> >>> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath >>> wrote: >>> >>>> Hi All, >>>> >>>> I think it's good if we can have new Jira components to easily track >>>> various issues related to cross-language transforms. >>>> >>>> What do you think about adding the following Jira components ? >>>> >>>> sdks-python-xlang >>>> sdks-java-xlang >>>> sdks-go-xlang >>>> >>>> Jira component sdks-foo-xlang is for tracking issues related to >>>> cross-language transforms for SDK Foo. For example, >>>> * Issues related cross-language transforms wrappers written in SDK Foo >>>> * Issues related to transforms implemented in SDK Foo that are >>>> offered as cross-language transforms to other SDKs >>>> * Issues related to cross-language transform expansion service >>>> implemented for SDK Foo >>>> >>>> Thanks, >>>> Cham >>>> >>>
Re: Jira components for cross-language transforms
+1 on dedicated components for cross-language transform. It might be easy to manage to have one component (one tag for all SDK) rather than multiple ones. Re Ashwin, Cham knows more than me. AFAIK, cross-language transforms will maximize code reuse for newly developed SDK (e.g. IO transforms for Go SDK). Of course, a SDK can develop its own IOs, but it's lots of work. -Rui On Thu, May 28, 2020 at 1:50 PM Ashwin Ramaswami wrote: > What are some of the benefits / drawbacks of using cross-language > transforms? Would a native Python transform perform better than a > cross-language transform written in Java that is then used in a Python > pipeline? > > Ashwin Ramaswami > Student > *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | Website > <https://epicfaace.github.io/> | GitHub <https://github.com/epicfaace> > > > On Thu, May 28, 2020 at 4:44 PM Kyle Weaver wrote: > >> SGTM. Though I'm not sure it's necessary to split by language. It might >> be easier to use a single cross-language tag, rather than having to tag >> lots of issues as both sdks-python-xlang and sdks-java-xlang. >> >> On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath >> wrote: >> >>> Hi All, >>> >>> I think it's good if we can have new Jira components to easily track >>> various issues related to cross-language transforms. >>> >>> What do you think about adding the following Jira components ? >>> >>> sdks-python-xlang >>> sdks-java-xlang >>> sdks-go-xlang >>> >>> Jira component sdks-foo-xlang is for tracking issues related to >>> cross-language transforms for SDK Foo. For example, >>> * Issues related cross-language transforms wrappers written in SDK Foo >>> * Issues related to transforms implemented in SDK Foo that are >>> offered as cross-language transforms to other SDKs >>> * Issues related to cross-language transform expansion service >>> implemented for SDK Foo >>> >>> Thanks, >>> Cham >>> >>
Re: Jira components for cross-language transforms
What are some of the benefits / drawbacks of using cross-language transforms? Would a native Python transform perform better than a cross-language transform written in Java that is then used in a Python pipeline? Ashwin Ramaswami Student *Find me on my:* LinkedIn <https://www.linkedin.com/in/ashwin-r> | Website <https://epicfaace.github.io/> | GitHub <https://github.com/epicfaace> On Thu, May 28, 2020 at 4:44 PM Kyle Weaver wrote: > SGTM. Though I'm not sure it's necessary to split by language. It might be > easier to use a single cross-language tag, rather than having to tag lots > of issues as both sdks-python-xlang and sdks-java-xlang. > > On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath > wrote: > >> Hi All, >> >> I think it's good if we can have new Jira components to easily track >> various issues related to cross-language transforms. >> >> What do you think about adding the following Jira components ? >> >> sdks-python-xlang >> sdks-java-xlang >> sdks-go-xlang >> >> Jira component sdks-foo-xlang is for tracking issues related to >> cross-language transforms for SDK Foo. For example, >> * Issues related cross-language transforms wrappers written in SDK Foo >> * Issues related to transforms implemented in SDK Foo that are offered as >> cross-language transforms to other SDKs >> * Issues related to cross-language transform expansion service >> implemented for SDK Foo >> >> Thanks, >> Cham >> >
Re: Jira components for cross-language transforms
SGTM. Though I'm not sure it's necessary to split by language. It might be easier to use a single cross-language tag, rather than having to tag lots of issues as both sdks-python-xlang and sdks-java-xlang. On Thu, May 28, 2020 at 4:29 PM Chamikara Jayalath wrote: > Hi All, > > I think it's good if we can have new Jira components to easily track > various issues related to cross-language transforms. > > What do you think about adding the following Jira components ? > > sdks-python-xlang > sdks-java-xlang > sdks-go-xlang > > Jira component sdks-foo-xlang is for tracking issues related to > cross-language transforms for SDK Foo. For example, > * Issues related cross-language transforms wrappers written in SDK Foo > * Issues related to transforms implemented in SDK Foo that are offered as > cross-language transforms to other SDKs > * Issues related to cross-language transform expansion service implemented > for SDK Foo > > Thanks, > Cham >
Jira components for cross-language transforms
Hi All, I think it's good if we can have new Jira components to easily track various issues related to cross-language transforms. What do you think about adding the following Jira components ? sdks-python-xlang sdks-java-xlang sdks-go-xlang Jira component sdks-foo-xlang is for tracking issues related to cross-language transforms for SDK Foo. For example, * Issues related cross-language transforms wrappers written in SDK Foo * Issues related to transforms implemented in SDK Foo that are offered as cross-language transforms to other SDKs * Issues related to cross-language transform expansion service implemented for SDK Foo Thanks, Cham
