At present, there's no way to write an unbounded datasource with the Go SDK, which would require DoFn Self Checkpointing ( https://issues.apache.org/jira/browse/BEAM-11104) and Watermark Estimation (https://issues.apache.org/jira/browse/BEAM-11105).
Daniel is working on wrapping the Java kafka connector in the next few months using the system Robert linked. Older kafka + Go specific JIRAs exist at https://issues.apache.org/jira/browse/BEAM-4250 and https://issues.apache.org/jira/browse/BEAM-6260, but they should probably be retired, in favour of one that mentions Xlang specifically. I believe Daniel will file them once he breaks down the task. See https://cwiki.apache.org/confluence/display/BEAM/Supporting+Streaming+in+the+Go+SDK for more details on what it will take for streaming with the Go SDK. On Mon, Mar 29, 2021 at 9:28 AM Robert Bradshaw <[email protected]> wrote: > On Wed, Mar 24, 2021 at 4:24 AM Đức Trần Tiến <[email protected]> > wrote: > >> >> And the last question: Could I write that pipeline in Java and invoke >> that pipeline from Go? :D >> > > That is exactly the story we're trying to pursue for getting the large set > of Java connectors available to Go: > > > https://cloud.google.com/blog/products/data-analytics/multi-language-sdks-for-building-cloud-pipelines > > > Cc'ing some folks that can comment on the status. > >
