Hi Ke, A cross-language pipeline looks a lot like a pipeline written natively in one of the Beam SDKs, the difference is that some of the transforms in the pipeline may be "external transforms" that actually have implementations in a different language. There are a few examples in the beam repo that use Java transforms from Python pipelines: - kafkataxi [1]: Uses Java's KafkaIO from Python - wordcount_xlang_sql [2] and sql_taxi [3]: Use Java's SqlTransform from Python
To create your own cross-language pipeline, you'll need to decide which SDK you want to use primarily, and then create an expansion service to expose the transforms you want to use from the other SDK (if one doesn't exist already). [1] https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi [2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang_sql.py [3] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py On Wed, Nov 11, 2020 at 11:07 AM Ke Wu <ke.wu...@gmail.com> wrote: > Hello, > > Is there an example demonstrating how a cross language pipeline look like? > e.g. a pipeline where it is composes of Java and Python code/transforms. > > Best, > Ke
