You should create placeholders inside of your Twister2/OpenMPI implementation that represent these functions and then instantiate actual instances of them on the workers if you want to write your own pipeline representation and format for OpenMPI/Twister2.
Or consider converting the pipeline to its proto representation and building a portable pipeline runner. This way you could run Go and Python pipelines as well. The best example of this is the current Flink integration[1] 1: https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Dev's > > Currently, the Pipeline class in Beam is not Serializable. This is not a > problem for the current runners since the pipeline is translated and > submitted through a centralized Driver like model. However, if the runner > has a decentralized model similar to OpenMPI (MPI), which is also the case > with Twister2, which I am developing a runner currently, it would have been > better if the pipeline itself was Serializable. > > Currently, I am trying to transform the Pipeline into a Twister2 graph and > then send over to the workers, however since there are some functions such > as "SystemReduceFn" that are not serializable this also is somewhat > troublesome. > > Was the decision to make Pipelines not Serializable made due to some > specific reason or because all the current use cases did not present any > valid requirement to make them Serializable? > > Best Regards, > Pulasthi > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 <(224)%20386-9035> >
