Hi Pulasthi, This might be possible but I don't know if anybody has done this. API of Beam sources are no different from other Beam PTransforms and we highly recommend hiding away various implementations of source framework related abstractions in a composite transform [1]. So what you are looking for is a way to use a Beam transform in a separate system.
Thanks, Cham [1] https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Dev's > > We have a big data processing framework named Twister2, and wanted to know > if there is any way we could leverage the I/O Transforms that are built > into Apache Beam externally. That is rather than using it in a Beam > pipeline just use them as data sources in our project. Just wanted to check > with the dev's if such an approach has been done by anyone before, so we > could get some pointers on how this could be done. Any pointers in the > right direction would be highly appreciated. > > Best Regards, > Pulasthi > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 <(224)%20386-9035> >
