Hi @Saikat Maitra I have my implementation on an Ignite fork on my personal GitHub account [1] It is based on the Kafka implementation which is of a polling nature. GCP does not provide a service like Kafka. Instead it provides us with Pub/Sub which is just like Kafka. Therefore using the Kafka streamer as a guideline made more sense to me.
Apart from the implementation I have also added a unit test. It definitely needs some polishing and a double check on my side to make the review smooth. For the tests a local Pub/Sub [2] server has to be setup however I am not sure which is the best way to do so with regards to your CI/CD. [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hello Emmanouil, > > Can you please share more info about the implementation? > > We have similar implementation for data streamers with Kafka and RocketMQ > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > Please review and share if you have any questions. > > Regards, > Saikat > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > wrote: > > > Hi all, > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] > > Yes it is indeed cloud specific. It is the messaging service provided by > > GCP. > > I have made the integration with it, but I haven't published it yet (I am > > waiting for the discussion/ticket and proceed with the contribution). > > > > Kind regards, > > Emmanouil > > > > [1] https://cloud.google.com/pubsub/docs/overview > > > > *Emmanouil Gkatziouras* > > https://egkatzioura.com/ | > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > https://github.com/gkatzioura > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <dma...@apache.org> wrote: > > > > > Hello Emmanouil, > > > > > > Are you referring to any specific implementation tailored for a cloud? > If > > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github > source > > > code? > > > > > > Here is a list of existing integrations with streaming and pub/sub > > > frameworks: > > > https://apacheignite-mix.readme.io/docs/overview > > > > > > with the source code in Ignite master: > > > https://github.com/apache/ignite/tree/master/modules > > > > > > Check out those resources to see what are the most suitable Ignite APIs > > for > > > such integrations. > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman <rsht...@yahoo.com> > , > > as > > > streaming technologies contributors/maintainers, could you please join > > this > > > thread? > > > > > > - > > > Denis > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < > > > gkatzio...@gmail.com> > > > wrote: > > > > > > > Hello everyone, > > > > > > > > I started using Ignite lately. Part of my work involves Pub/Sub, > thus I > > > > created a pub/sub streamer. > > > > Pub/Sub is pretty close to Kafka. > > > > Here's how the streamer works. Every node creates an instance of a > > > Pub/Sub > > > > client. > > > > The clients subscribe to the same topic using the same subscription > > name. > > > > A message is being sent and one of the nodes will receive the > message, > > > > process it and then forward it to Ignite using the extractor > provided. > > > > > > > > Any thoughts? > > > > > > > > *Emmanouil Gkatziouras* > > > > https://egkatzioura.com/ | > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > https://github.com/gkatzioura > > > > > > > > > >
