Hi Kenn, Thanks a lot for the information.
Sure, below are more details about the problem I encountered. I am developing a runner for IBM Streams, and am exploring possible ways to conduct integration tests. As Streams is not an open-source project, we cannot add the full set of libraries to Maven Central repo. Nor can we guarantee to provide a server (with Streams installed) as a long-term Jenkins slave. So, it seems more flexible to let the runner submit the graph to a Streams cloud service, and provide the account info through "-DrunnableOnServicePipelineOptions" (please correct me if it does not work in this way). The problem is that the runner cannot convert the Pipeline into a Streams graph format without a local Streams install. So, I am thinking about sending the serialized Pipeline to the Cloud service for execution. Maybe I should create some intermediate format between the Pipeline and Streams graph format. Or, is there any other way to carry out the integration test without a Streams install? Thanks, Shen On Wed, Dec 21, 2016 at 12:08 PM, Kenneth Knowles <k...@google.com.invalid> wrote: > Hi Shen, > > I want to tell you (1) how things work today and (2) how we want them to be > eventually. > > (1) So far, each runner translates the Pipeline to their own graph format > before serialization, so we have not yet encountered this issue. > > (2) We intend to make a standard mostly-readable JSON format for a > Pipeline. It is based the Avro schema sketched in the design doc at > https://s.apache.org/beam-runner-api and there is also a draft JSON schema > at https://github.com/apache/incubator-beam/pull/662. > > You may wish to follow https://issues.apache.org/jira/browse/BEAM-115, > though that is a very general ticket. > > Can you share any more details? > > Kenn > > On Wed, Dec 21, 2016 at 8:47 AM, Shen Li <cs.she...@gmail.com> wrote: > > > Hi, > > > > What are the recommended ways to serialize/deserialize a Pipeline > object? I > > need to submit a pipeline object to cloud for execution and fetch the > > result. > > > > Thanks, > > > > Shen > > >
