It may or may not work. Some runners may have additional optimizations/rewiring and runners are not aware whether two arbitrary coders implemented in different languages are equivalent or not unless they use the same URN defined in standard coders. It's safe to stick to standard coders. We already have RowCoder for encoding "arbitrary" objects across SDK boundaries.
Thanks, Cham On Mon, Jun 15, 2020 at 10:34 AM Boyuan Zhang <[email protected]> wrote: > Thanks Cham. Standard coder is a good point. Does it mean non-standard > coder doesn't work when crossing language boundaries even if it is > implemented in both Java and Python sdk? > > On Mon, Jun 15, 2020 at 10:08 AM Chamikara Jayalath <[email protected]> > wrote: > >> Thanks. +1 for using RowCoder. We should try to use standard coders [1] >> in the x-lang SDK boundaries. >> If we use other coders (for example, ProtoCoder) it may or may not work >> depending on how various runners implement support for x-lang. >> >> This might require slightly updating existing transforms or adding >> additional conversion transforms to cross-language builders. >> >> [1] >> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L669 >> >> Thanks, >> Cham >> >> On Mon, Jun 15, 2020 at 10:00 AM Piotr Szuberski < >> [email protected]> wrote: >> >>> Right now I'm working on JdbcIO and I'm using Row and Schema protobuffs. >>> I'm figuring out how to use them properly. Thanks for the article - for >>> sure it will be helpful! >>> >>> On 2020/06/12 20:32:16, Brian Hulette <[email protected]> wrote: >>> > Thanks! I see there are jiras for SpannerIO and JdbcIO as part of >>> that. Are >>> > you planning on using row coder for them? >>> > If so I want to make sure you're aware of >>> > https://s.apache.org/beam-schema-io (sent to the dev list last week >>> > [1]). +Scott >>> > Lukas <[email protected]> will be working on building out the ideas >>> there >>> > this summer. His work could be useful for making these IOs >>> cross-language >>> > (and you would get a mapping to SQL out of it without much more >>> effort). >>> > >>> > Brian >>> > >>> > [1] >>> > >>> https://lists.apache.org/thread.html/rc1695025d41c5dc38cdf7bc32bea0e7421379b1c543c2d82f69aa179%40%3Cdev.beam.apache.org%3E >>> > >>> > On Tue, Jun 2, 2020 at 9:30 AM Piotr Szuberski < >>> [email protected]> >>> > wrote: >>> > >>> > > Sure, I'll do that >>> > > >>> > > On 2020/05/28 17:54:49, Chamikara Jayalath <[email protected]> >>> wrote: >>> > > > Great. Thanks for working on this. Can you please add these tasks >>> and >>> > > JIRAs >>> > > > to the cross-language transforms roadmap under "Connector/transform >>> > > > support". >>> > > > https://beam.apache.org/roadmap/connectors-multi-sdk/ >>> > > > >>> > > > Happy to help if you run into any issues during this task. >>> > > > >>> > > > <https://beam.apache.org/roadmap/connectors-multi-sdk/>Thanks, >>> > > > Cham >>> > > > >>> > > > On Thu, May 28, 2020 at 9:59 AM Piotr Szuberski < >>> > > [email protected]> >>> > > > wrote: >>> > > > >>> > > > > I added to Jira task of creating cross-language wrappers for >>> Java IOs. >>> > > It >>> > > > > will soon be in progress. >>> > > > > >>> > > > >>> > > >>> > >>> >>
