mxtymoshyk commented on issue #26039: URL: https://github.com/apache/beam/issues/26039#issuecomment-6807737951
@mingmwang Java used to have this. `PipelineTranslation.fromProto(RunnerApi.Pipeline)` rebuilt an SDK `Pipeline` from the proto, and it was removed on purpose in 2018 (f2b4596149224b2229c683a68f17d8127e3fd144, BEAM-3971 / BEAM-4284). The commit message says runners should work on the proto directly, because there is no expectation that every SDK construct can be rebuilt from it. The reason still applies. A Java `DoFn` or coder can be deserialized if its class is on the classpath, but transforms written in Python or Go, or added through cross-language expansion, can't become Java objects. Composites like `Combine` or `WriteFiles` would come back as placeholders without their `expand()` logic. The Job API already handles sending a pipeline over gRPC. The client calls `Prepare` and `Run` on a `JobService` (`beam_job_api.proto`) with the `RunnerApi.Pipeline`, and the runner translates the proto itself. In Java, `PortableRunner` and `PrismRunner` are clients, and `runners/java-job-service` has `InMemoryJobService` as a starting point for a server. If you need individual pieces back as Java objects, `RehydratedComponents`, `CoderTranslation.fromProto` and `ParDoTranslation.getDoFn` do that one component at a time. I'd suggest closing this as won't fix. If you have a use case the Job API doesn't cover, please describe it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
