Saikat, thanks for checking with Bahir! However, let’s run this idea through the rest of the community before getting down to the implementation. As an alternate option, we might adopt their approach for Ignite extensions but keep those within Ignite community in separate repositories. Would you mind starting a separate discussion with Ignite dev?
Denis On Tuesday, October 22, 2019, Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hello Emmanouil, > > As discussed earlier, I discussed with Apache Bahir community and they are > interested to have Apache Ignite extensions as part of Apache Bahir > project. > > Can you please share Pub/Sub streamer implementation details with Apache > Bahir community and request for feedback. > > https://bahir.apache.org/ > > I have also requested for contributor access in Jira for Apache Bahir > project so that I can create issues and assign to myself. > > I can help with code reviews as well. > > Here is the email thread for reference > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > Regards, > Saikat > > > > > > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.mai...@gmail.com> > wrote: > > > Hello, > > > > As discussed I have created following discussion threads on our migration > > proposals for Apache Ignite extensions: > > > > > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS- > Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator- > project-td29829.html > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > > > I will share update as I hear more information. > > > > Regards, > > Saikat > > > > > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.mai...@gmail.com> > > wrote: > > > >> Hello Denis, > >> > >> Sure, sounds good. I will create a separate discussion thread to get > >> community feedback on whether we > >> should join the Bahir or kick-off "Ignite Extensions" project. > >> > >> Regards, > >> Saikat > >> > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dma...@apache.org> wrote: > >> > >>> Folks, > >>> > >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks > for > >>> doing the research. Why I believe the * idea* fits us well: > >>> > >>> - All integrations can be stored in separate Github repositories and > >>> have their dev lifecycles. We've not obliged to couple all the > >>> integrations > >>> in a single repo. For instance, Spark can be located in a dedicated > >>> repo > >>> while streaming integrations might be in a single one. It's up to > us. > >>> - All the repositories and integrations belong to ASF. We're not > >>> dumping > >>> them on Github but rather keep supporting and developing in > >>> accordance with > >>> ASF vision and practices. > >>> - There is a way to reward contributors via committership and PMC > >>> membership. You won't get this if a project is just one of the > >>> millions of > >>> Github projects. > >>> > >>> Saikat, as Ignite PMC member, could you please kick-off a separate > >>> dev-list > >>> discussion to involve more community members and referring to this > >>> thread? > >>> I think the primary question the community needs to answer whether we > >>> should join the Bahir or kick-off "Ignite Extensions" project? > >>> > >>> - > >>> Denis > >>> > >>> > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < > zaleslaw....@gmail.com> > >>> wrote: > >>> > >>> > Maybe we could move all our Streaming Integrations there, but what is > >>> about > >>> > maintaining and committer permissions to the new repositories? > >>> > I see the list of the committers and PMC members there > >>> > https://bahir.apache.org/community-members/ > >>> > > >>> > Could somebody from Ignite community be added to this list as a > >>> > PMC/committer to maintain Ignite integrations? > >>> > If yes, I'd happy to join this project and collaborate with these > guys > >>> > together > >>> > If no, and we should start from the zero level with external PRs - > >>> hmmm, > >>> > it's better to have our own external repository with ApacheBahirr > >>> ideology. > >>> > > >>> > Also, I agree, that all connectors could be moved there (in > >>> ApacheBahirr > >>> > like repository), but not all modules from the page > >>> > > >>> > > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > >>> > because > >>> > they use different parts of core modules and could be developed with > >>> > different velocity. > >>> > > >>> > In reality, before creation of new repositories we need wide > discussion > >>> > based on the document mentioned above. > >>> > > >>> > P.S. Of course, we could experiment in the next release 2.8 with one > >>> or two > >>> > integrations like twitter or ZeroMQ > >>> > Also, I'd like Denis Magda idea for the separate repositories in > Apache > >>> > Github to maintain them by anybody. > >>> > > >>> > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.mai...@gmail.com > >: > >>> > > >>> > > Hi Denis, Emmanouil > >>> > > > >>> > > Thank you for your email. I think creating a separate integration > >>> project > >>> > > in github and also proposing it as an apache incubator project will > >>> be a > >>> > > good move. The other separate integration modules > >>> > > > >>> > > > >>> > > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > >>> > > can > >>> > > be moved to this new apache incubator project. > >>> > > > >>> > > There are similar integration available for Flink and Spark in > Apache > >>> > Bahir > >>> > > https://bahir.apache.org/ > >>> > > > >>> > > Do you think we should reach out to Apache Bahir community with a > >>> > proposal > >>> > > or we should plan to create a separate Apache Incubator project? > >>> > > > >>> > > Regards, > >>> > > Saikat > >>> > > > >>> > > > >>> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > >>> > > gkatzio...@gmail.com> > >>> > > wrote: > >>> > > > >>> > > > Hi all! > >>> > > > > >>> > > > Based on the discussions most probably this is going to be on > >>> another > >>> > > > GitHub repo. Therefore I will prepare a standalone project with > the > >>> > > Pub/Sub > >>> > > > module and once the repository is created a pull request shall be > >>> > issued. > >>> > > > If there is anything else I can do in order to facilitate this > >>> please > >>> > let > >>> > > > me know. > >>> > > > > >>> > > > If a ticket is created for this issue, I shall be able to create > a > >>> pull > >>> > > > request and thus a build will take place on ignite's CI. > >>> > > > If everyone is aligned with adding a Pub/Sub implementation may I > >>> > create > >>> > > a > >>> > > > ticket on JIRA? > >>> > > > > >>> > > > My next question has to do on how Ignite configures the build > jobs > >>> in > >>> > > Team > >>> > > > City. The unit tests for my implementation need a local Pub/Sub > >>> server > >>> > up > >>> > > > and running (they provide one for testing purposes). Will this > take > >>> > > effect > >>> > > > in a form of a build script inside my implementation or is it > >>> something > >>> > > > that should be configured on Team City? > >>> > > > > >>> > > > Kind regards, > >>> > > > Emmanouil > >>> > > > > >>> > > > > >>> > > > *Emmanouil Gkatziouras* > >>> > > > https://egkatzioura.com/ | > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > >>> > > > https://github.com/gkatzioura > >>> > > > > >>> > > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> > >>> wrote: > >>> > > > > >>> > > > > Emmanouil, Saikat, > >>> > > > > > >>> > > > > After contemplating on this, let me propose another way with > this > >>> > > > > particular integration that is aligned with our modularization > >>> > > endeavor. > >>> > > > > The modularization [1] defines the Ignite core, Ignite modules, > >>> and > >>> > > > > independent integrations. The first two are to be located in > >>> Ignite > >>> > > > > repositories and to be tested/updated by the community for > every > >>> > > release. > >>> > > > > The modules are a subset of existing integrations that are > >>> adopted > >>> > > widely > >>> > > > > and require community attention. > >>> > > > > > >>> > > > > As for the independent integrations [1], those are supposed to > be > >>> > > located > >>> > > > > in a separate GitHub repo. I would advise us to start > >>> implementing > >>> > this > >>> > > > > concept with the Pub/Sub integration. Even though it will be > >>> located > >>> > > in a > >>> > > > > separate Github repo, it will be listed among officially > >>> available > >>> > > > > integrations on the Ignite website featuring Emmanouil as a > >>> > > contributor. > >>> > > > > > >>> > > > > What do you think? > >>> > > > > > >>> > > > > > >>> > > > > [1] > >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > - > >>> > > > > Denis > >>> > > > > > >>> > > > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < > >>> > saikat.mai...@gmail.com > >>> > > > > >>> > > > > wrote: > >>> > > > > > >>> > > > > > Hi Emmanouil, > >>> > > > > > > >>> > > > > > The changes looks good to me. I wanted to check if you are > also > >>> > able > >>> > > to > >>> > > > > > validate that the once data is added to Ignite cluster you > are > >>> also > >>> > > > able > >>> > > > > to > >>> > > > > > access it using another Ignite client or using rest > endpoints. > >>> > > > > > > >>> > > > > > We use Teamcity for CI process, the details are as follows > >>> > > > > > > >>> > > > > > Trigger validation of those test suites that have been > >>> affected by > >>> > > your > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < > >>> > > > > http://204.14.53.153/ > >>> > > > > > > > >>> > > > > > > >>> > > > > > - Locate a test suite you have to check, press button > named > >>> > "..." > >>> > > > that > >>> > > > > > is located on the left of "Run" button. "Run custom build" > >>> > dialog > >>> > > > will > >>> > > > > > appear; > >>> > > > > > - Go to "Changes" tab and choose > >>> > "pull/<pull-request-number>/head" > >>> > > > in > >>> > > > > > "Build branch" dropdown list; > >>> > > > > > - Press "Run build" button and monitor tests results. > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also please take a look? > >>> > > > > > > >>> > > > > > Regards, > >>> > > > > > Saikat > >>> > > > > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > >>> > > > > > gkatzio...@gmail.com> > >>> > > > > > wrote: > >>> > > > > > > >>> > > > > > > Hi @Saikat Maitra > >>> > > > > > > > >>> > > > > > > I have my implementation on an Ignite fork on my personal > >>> GitHub > >>> > > > > account > >>> > > > > > > [1] > >>> > > > > > > It is based on the Kafka implementation which is of a > polling > >>> > > nature. > >>> > > > > > > GCP does not provide a service like Kafka. Instead it > >>> provides us > >>> > > > with > >>> > > > > > > Pub/Sub which is just like Kafka. > >>> > > > > > > Therefore using the Kafka streamer as a guideline made more > >>> sense > >>> > > to > >>> > > > > me. > >>> > > > > > > > >>> > > > > > > Apart from the implementation I have also added a unit > test. > >>> > > > > > > It definitely needs some polishing and a double check on my > >>> side > >>> > to > >>> > > > > make > >>> > > > > > > the review smooth. > >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup > >>> however > >>> > I > >>> > > am > >>> > > > > not > >>> > > > > > > sure which is the best way to do so with regards to your > >>> CI/CD. > >>> > > > > > > > >>> > > > > > > [1] > >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > >>> > > > > > > [2] > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > >>> > > > > > > > >>> > > > > > > *Emmanouil Gkatziouras* > >>> > > > > > > https://egkatzioura.com/ | > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > >>> > > > > > > https://github.com/gkatzioura > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > >>> > > saikat.mai...@gmail.com > >>> > > > > > >>> > > > > > > wrote: > >>> > > > > > > > >>> > > > > > > > Hello Emmanouil, > >>> > > > > > > > > >>> > > > > > > > Can you please share more info about the implementation? > >>> > > > > > > > > >>> > > > > > > > We have similar implementation for data streamers with > >>> Kafka > >>> > and > >>> > > > > > RocketMQ > >>> > > > > > > > > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > >>> > > > > > > > > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq- > streamer > >>> > > > > > > > > >>> > > > > > > > Please review and share if you have any questions. > >>> > > > > > > > > >>> > > > > > > > Regards, > >>> > > > > > > > Saikat > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > >>> > > > > > > > gkatzio...@gmail.com> > >>> > > > > > > > wrote: > >>> > > > > > > > > >>> > > > > > > > > Hi all, > >>> > > > > > > > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] > >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging > >>> service > >>> > > > > provided > >>> > > > > > > by > >>> > > > > > > > > GCP. > >>> > > > > > > > > I have made the integration with it, but I haven't > >>> published > >>> > it > >>> > > > yet > >>> > > > > > (I > >>> > > > > > > am > >>> > > > > > > > > waiting for the discussion/ticket and proceed with the > >>> > > > > contribution). > >>> > > > > > > > > > >>> > > > > > > > > Kind regards, > >>> > > > > > > > > Emmanouil > >>> > > > > > > > > > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > >>> > > > > > > > > > >>> > > > > > > > > *Emmanouil Gkatziouras* > >>> > > > > > > > > https://egkatzioura.com/ | > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > >>> > > > > > > > > https://github.com/gkatzioura > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < > >>> dma...@apache.org > >>> > > > >>> > > > > wrote: > >>> > > > > > > > > > >>> > > > > > > > > > Hello Emmanouil, > >>> > > > > > > > > > > >>> > > > > > > > > > Are you referring to any specific implementation > >>> tailored > >>> > > for a > >>> > > > > > > cloud? > >>> > > > > > > > If > >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is > there > >>> any > >>> > > > Github > >>> > > > > > > > source > >>> > > > > > > > > > code? > >>> > > > > > > > > > > >>> > > > > > > > > > Here is a list of existing integrations with > streaming > >>> and > >>> > > > > pub/sub > >>> > > > > > > > > > frameworks: > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview > >>> > > > > > > > > > > >>> > > > > > > > > > with the source code in Ignite master: > >>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules > >>> > > > > > > > > > > >>> > > > > > > > > > Check out those resources to see what are the most > >>> suitable > >>> > > > > Ignite > >>> > > > > > > APIs > >>> > > > > > > > > for > >>> > > > > > > > > > such integrations. > >>> > > > > > > > > > > >>> > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman < > >>> > > > > > rsht...@yahoo.com > >>> > > > > > > > > >>> > > > > > > > , > >>> > > > > > > > > as > >>> > > > > > > > > > streaming technologies contributors/maintainers, > could > >>> you > >>> > > > please > >>> > > > > > > join > >>> > > > > > > > > this > >>> > > > > > > > > > thread? > >>> > > > > > > > > > > >>> > > > > > > > > > - > >>> > > > > > > > > > Denis > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil > Gkatziouras < > >>> > > > > > > > > > gkatzio...@gmail.com> > >>> > > > > > > > > > wrote: > >>> > > > > > > > > > > >>> > > > > > > > > > > Hello everyone, > >>> > > > > > > > > > > > >>> > > > > > > > > > > I started using Ignite lately. Part of my work > >>> involves > >>> > > > > Pub/Sub, > >>> > > > > > > > thus I > >>> > > > > > > > > > > created a pub/sub streamer. > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. > >>> > > > > > > > > > > Here's how the streamer works. Every node creates > an > >>> > > instance > >>> > > > > of > >>> > > > > > a > >>> > > > > > > > > > Pub/Sub > >>> > > > > > > > > > > client. > >>> > > > > > > > > > > The clients subscribe to the same topic using the > >>> same > >>> > > > > > subscription > >>> > > > > > > > > name. > >>> > > > > > > > > > > A message is being sent and one of the nodes will > >>> receive > >>> > > the > >>> > > > > > > > message, > >>> > > > > > > > > > > process it and then forward it to Ignite using the > >>> > > extractor > >>> > > > > > > > provided. > >>> > > > > > > > > > > > >>> > > > > > > > > > > Any thoughts? > >>> > > > > > > > > > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* > >>> > > > > > > > > > > https://egkatzioura.com/ | > >>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > >>> > > > > > > > > > > https://github.com/gkatzioura > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> > -- - Denis
