Hey Yu, as the error you posted suggests, the Go direct runner which you're
using in your local development environment doesn't support external
transforms using an expansion service. If you're going to do a x-lang
transform using an expansion service you should use a different runner like
Dataflow, Flink, Spark, or one of the other runners listed here -
https://beam.apache.org/documentation/runners/capability-matrix/
Thanks,
Danny
On Sun, Aug 28, 2022 at 7:50 AM Yu Watanabe wrote:
> Hello.
>
> I would like to ask a question about expansion service. I'm currently
> testing my expansion service in my local development environment.
> I have read notes about kafka in advance,
>
>
> https://github.com/apache/beam/blob/master/sdks/go/examples/kafka/taxi.go#L93
>
> I have prepared sdk containers .
>
> [ywatanabe@laptop-archlinux Development]$ docker image ls | grep apache
> apache/beam_java8_sdk 2.42.0.dev
> f7e9d38b01fe 11 days ago 643MB
> apache/beam_go_sdk latest
> 8a87ea45255b 11 days ago 149MB
>
> However, when I run the code in my local environment, I get an error.
>
> [ywatanabe@laptop-archlinux go]$ go run
> ./examples/elasticsearch/sample.go \
> --runner direct \
> --sdk_harness_container_image_override
> ".*java.*,apache/beam_java8_sdk:2.42.0.dev"
> Hello world.
> 2022/08/28 20:39:01 Executing pipeline with the direct runner.
> 2022/08/28 20:39:01 Pipeline:
> 2022/08/28 20:39:01 Nodes: {1: []uint8/bytes GLO}
> {2: string/string GLO}
> {3: []uint8/bytes GLO}
> {4: []uint8/bytes GLO}
> Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
> 2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T ->
> {2: string/string GLO}]
> 3: External [In(Main): string <- {2: string/string GLO}] -> [Out:
> []uint8 -> {3: []uint8/bytes GLO} Out: []uint8 -> {4: []uint8/bytes
> GLO}]
> Pipeline failed: translation failed
> caused by:
> external transforms like 3: External [In(Main): string <- {2:
> string/string GLO}] -> [Out: []uint8 -> {3: []uint8/bytes GLO} Out:
> []uint8 -> {4: []uint8/bytes GLO}] are not supported in the Go direct
> runner, please execute your pipel[ywatanabe@laptop-archlinux go]$
>
> Am I missing something ?
>
> My main and io code can be found below.
>
> https://gist.github.com/yuwtennis/dec3bf3bfc0c4fa54d9d3565c98d008e
>
> Thanks,
> Yu
>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
