Hello Emmanouil, As discussed earlier, I discussed with Apache Bahir community and they are interested to have Apache Ignite extensions as part of Apache Bahir project.
Can you please share Pub/Sub streamer implementation details with Apache Bahir community and request for feedback. https://bahir.apache.org/ I have also requested for contributor access in Jira for Apache Bahir project so that I can create issues and assign to myself. I can help with code reviews as well. Here is the email thread for reference https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html Regards, Saikat On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hello, > > As discussed I have created following discussion threads on our migration > proposals for Apache Ignite extensions: > > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > I will share update as I hear more information. > > Regards, > Saikat > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.mai...@gmail.com> > wrote: > >> Hello Denis, >> >> Sure, sounds good. I will create a separate discussion thread to get >> community feedback on whether we >> should join the Bahir or kick-off "Ignite Extensions" project. >> >> Regards, >> Saikat >> >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dma...@apache.org> wrote: >> >>> Folks, >>> >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks for >>> doing the research. Why I believe the * idea* fits us well: >>> >>> - All integrations can be stored in separate Github repositories and >>> have their dev lifecycles. We've not obliged to couple all the >>> integrations >>> in a single repo. For instance, Spark can be located in a dedicated >>> repo >>> while streaming integrations might be in a single one. It's up to us. >>> - All the repositories and integrations belong to ASF. We're not >>> dumping >>> them on Github but rather keep supporting and developing in >>> accordance with >>> ASF vision and practices. >>> - There is a way to reward contributors via committership and PMC >>> membership. You won't get this if a project is just one of the >>> millions of >>> Github projects. >>> >>> Saikat, as Ignite PMC member, could you please kick-off a separate >>> dev-list >>> discussion to involve more community members and referring to this >>> thread? >>> I think the primary question the community needs to answer whether we >>> should join the Bahir or kick-off "Ignite Extensions" project? >>> >>> - >>> Denis >>> >>> >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <zaleslaw....@gmail.com> >>> wrote: >>> >>> > Maybe we could move all our Streaming Integrations there, but what is >>> about >>> > maintaining and committer permissions to the new repositories? >>> > I see the list of the committers and PMC members there >>> > https://bahir.apache.org/community-members/ >>> > >>> > Could somebody from Ignite community be added to this list as a >>> > PMC/committer to maintain Ignite integrations? >>> > If yes, I'd happy to join this project and collaborate with these guys >>> > together >>> > If no, and we should start from the zero level with external PRs - >>> hmmm, >>> > it's better to have our own external repository with ApacheBahirr >>> ideology. >>> > >>> > Also, I agree, that all connectors could be moved there (in >>> ApacheBahirr >>> > like repository), but not all modules from the page >>> > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > because >>> > they use different parts of core modules and could be developed with >>> > different velocity. >>> > >>> > In reality, before creation of new repositories we need wide discussion >>> > based on the document mentioned above. >>> > >>> > P.S. Of course, we could experiment in the next release 2.8 with one >>> or two >>> > integrations like twitter or ZeroMQ >>> > Also, I'd like Denis Magda idea for the separate repositories in Apache >>> > Github to maintain them by anybody. >>> > >>> > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.mai...@gmail.com>: >>> > >>> > > Hi Denis, Emmanouil >>> > > >>> > > Thank you for your email. I think creating a separate integration >>> project >>> > > in github and also proposing it as an apache incubator project will >>> be a >>> > > good move. The other separate integration modules >>> > > >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > can >>> > > be moved to this new apache incubator project. >>> > > >>> > > There are similar integration available for Flink and Spark in Apache >>> > Bahir >>> > > https://bahir.apache.org/ >>> > > >>> > > Do you think we should reach out to Apache Bahir community with a >>> > proposal >>> > > or we should plan to create a separate Apache Incubator project? >>> > > >>> > > Regards, >>> > > Saikat >>> > > >>> > > >>> > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < >>> > > gkatzio...@gmail.com> >>> > > wrote: >>> > > >>> > > > Hi all! >>> > > > >>> > > > Based on the discussions most probably this is going to be on >>> another >>> > > > GitHub repo. Therefore I will prepare a standalone project with the >>> > > Pub/Sub >>> > > > module and once the repository is created a pull request shall be >>> > issued. >>> > > > If there is anything else I can do in order to facilitate this >>> please >>> > let >>> > > > me know. >>> > > > >>> > > > If a ticket is created for this issue, I shall be able to create a >>> pull >>> > > > request and thus a build will take place on ignite's CI. >>> > > > If everyone is aligned with adding a Pub/Sub implementation may I >>> > create >>> > > a >>> > > > ticket on JIRA? >>> > > > >>> > > > My next question has to do on how Ignite configures the build jobs >>> in >>> > > Team >>> > > > City. The unit tests for my implementation need a local Pub/Sub >>> server >>> > up >>> > > > and running (they provide one for testing purposes). Will this take >>> > > effect >>> > > > in a form of a build script inside my implementation or is it >>> something >>> > > > that should be configured on Team City? >>> > > > >>> > > > Kind regards, >>> > > > Emmanouil >>> > > > >>> > > > >>> > > > *Emmanouil Gkatziouras* >>> > > > https://egkatzioura.com/ | >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > > https://github.com/gkatzioura >>> > > > >>> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> >>> wrote: >>> > > > >>> > > > > Emmanouil, Saikat, >>> > > > > >>> > > > > After contemplating on this, let me propose another way with this >>> > > > > particular integration that is aligned with our modularization >>> > > endeavor. >>> > > > > The modularization [1] defines the Ignite core, Ignite modules, >>> and >>> > > > > independent integrations. The first two are to be located in >>> Ignite >>> > > > > repositories and to be tested/updated by the community for every >>> > > release. >>> > > > > The modules are a subset of existing integrations that are >>> adopted >>> > > widely >>> > > > > and require community attention. >>> > > > > >>> > > > > As for the independent integrations [1], those are supposed to be >>> > > located >>> > > > > in a separate GitHub repo. I would advise us to start >>> implementing >>> > this >>> > > > > concept with the Pub/Sub integration. Even though it will be >>> located >>> > > in a >>> > > > > separate Github repo, it will be listed among officially >>> available >>> > > > > integrations on the Ignite website featuring Emmanouil as a >>> > > contributor. >>> > > > > >>> > > > > What do you think? >>> > > > > >>> > > > > >>> > > > > [1] >>> > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > > > >>> > > > > >>> > > > > >>> > > > > - >>> > > > > Denis >>> > > > > >>> > > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < >>> > saikat.mai...@gmail.com >>> > > > >>> > > > > wrote: >>> > > > > >>> > > > > > Hi Emmanouil, >>> > > > > > >>> > > > > > The changes looks good to me. I wanted to check if you are also >>> > able >>> > > to >>> > > > > > validate that the once data is added to Ignite cluster you are >>> also >>> > > > able >>> > > > > to >>> > > > > > access it using another Ignite client or using rest endpoints. >>> > > > > > >>> > > > > > We use Teamcity for CI process, the details are as follows >>> > > > > > >>> > > > > > Trigger validation of those test suites that have been >>> affected by >>> > > your >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < >>> > > > > http://204.14.53.153/ >>> > > > > > > >>> > > > > > >>> > > > > > - Locate a test suite you have to check, press button named >>> > "..." >>> > > > that >>> > > > > > is located on the left of "Run" button. "Run custom build" >>> > dialog >>> > > > will >>> > > > > > appear; >>> > > > > > - Go to "Changes" tab and choose >>> > "pull/<pull-request-number>/head" >>> > > > in >>> > > > > > "Build branch" dropdown list; >>> > > > > > - Press "Run build" button and monitor tests results. >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also please take a look? >>> > > > > > >>> > > > > > Regards, >>> > > > > > Saikat >>> > > > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < >>> > > > > > gkatzio...@gmail.com> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Hi @Saikat Maitra >>> > > > > > > >>> > > > > > > I have my implementation on an Ignite fork on my personal >>> GitHub >>> > > > > account >>> > > > > > > [1] >>> > > > > > > It is based on the Kafka implementation which is of a polling >>> > > nature. >>> > > > > > > GCP does not provide a service like Kafka. Instead it >>> provides us >>> > > > with >>> > > > > > > Pub/Sub which is just like Kafka. >>> > > > > > > Therefore using the Kafka streamer as a guideline made more >>> sense >>> > > to >>> > > > > me. >>> > > > > > > >>> > > > > > > Apart from the implementation I have also added a unit test. >>> > > > > > > It definitely needs some polishing and a double check on my >>> side >>> > to >>> > > > > make >>> > > > > > > the review smooth. >>> > > > > > > For the tests a local Pub/Sub [2] server has to be setup >>> however >>> > I >>> > > am >>> > > > > not >>> > > > > > > sure which is the best way to do so with regards to your >>> CI/CD. >>> > > > > > > >>> > > > > > > [1] >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >>> > > > > > > [2] >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ >>> > > > > > > >>> > > > > > > *Emmanouil Gkatziouras* >>> > > > > > > https://egkatzioura.com/ | >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > > > > > https://github.com/gkatzioura >>> > > > > > > >>> > > > > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < >>> > > saikat.mai...@gmail.com >>> > > > > >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > > > Hello Emmanouil, >>> > > > > > > > >>> > > > > > > > Can you please share more info about the implementation? >>> > > > > > > > >>> > > > > > > > We have similar implementation for data streamers with >>> Kafka >>> > and >>> > > > > > RocketMQ >>> > > > > > > > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer >>> > > > > > > > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer >>> > > > > > > > >>> > > > > > > > Please review and share if you have any questions. >>> > > > > > > > >>> > > > > > > > Regards, >>> > > > > > > > Saikat >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < >>> > > > > > > > gkatzio...@gmail.com> >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > Hi all, >>> > > > > > > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] >>> > > > > > > > > Yes it is indeed cloud specific. It is the messaging >>> service >>> > > > > provided >>> > > > > > > by >>> > > > > > > > > GCP. >>> > > > > > > > > I have made the integration with it, but I haven't >>> published >>> > it >>> > > > yet >>> > > > > > (I >>> > > > > > > am >>> > > > > > > > > waiting for the discussion/ticket and proceed with the >>> > > > > contribution). >>> > > > > > > > > >>> > > > > > > > > Kind regards, >>> > > > > > > > > Emmanouil >>> > > > > > > > > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview >>> > > > > > > > > >>> > > > > > > > > *Emmanouil Gkatziouras* >>> > > > > > > > > https://egkatzioura.com/ | >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > > > > > > > https://github.com/gkatzioura >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < >>> dma...@apache.org >>> > > >>> > > > > wrote: >>> > > > > > > > > >>> > > > > > > > > > Hello Emmanouil, >>> > > > > > > > > > >>> > > > > > > > > > Are you referring to any specific implementation >>> tailored >>> > > for a >>> > > > > > > cloud? >>> > > > > > > > If >>> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there >>> any >>> > > > Github >>> > > > > > > > source >>> > > > > > > > > > code? >>> > > > > > > > > > >>> > > > > > > > > > Here is a list of existing integrations with streaming >>> and >>> > > > > pub/sub >>> > > > > > > > > > frameworks: >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview >>> > > > > > > > > > >>> > > > > > > > > > with the source code in Ignite master: >>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules >>> > > > > > > > > > >>> > > > > > > > > > Check out those resources to see what are the most >>> suitable >>> > > > > Ignite >>> > > > > > > APIs >>> > > > > > > > > for >>> > > > > > > > > > such integrations. >>> > > > > > > > > > >>> > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman < >>> > > > > > rsht...@yahoo.com >>> > > > > > > > >>> > > > > > > > , >>> > > > > > > > > as >>> > > > > > > > > > streaming technologies contributors/maintainers, could >>> you >>> > > > please >>> > > > > > > join >>> > > > > > > > > this >>> > > > > > > > > > thread? >>> > > > > > > > > > >>> > > > > > > > > > - >>> > > > > > > > > > Denis >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < >>> > > > > > > > > > gkatzio...@gmail.com> >>> > > > > > > > > > wrote: >>> > > > > > > > > > >>> > > > > > > > > > > Hello everyone, >>> > > > > > > > > > > >>> > > > > > > > > > > I started using Ignite lately. Part of my work >>> involves >>> > > > > Pub/Sub, >>> > > > > > > > thus I >>> > > > > > > > > > > created a pub/sub streamer. >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. >>> > > > > > > > > > > Here's how the streamer works. Every node creates an >>> > > instance >>> > > > > of >>> > > > > > a >>> > > > > > > > > > Pub/Sub >>> > > > > > > > > > > client. >>> > > > > > > > > > > The clients subscribe to the same topic using the >>> same >>> > > > > > subscription >>> > > > > > > > > name. >>> > > > > > > > > > > A message is being sent and one of the nodes will >>> receive >>> > > the >>> > > > > > > > message, >>> > > > > > > > > > > process it and then forward it to Ignite using the >>> > > extractor >>> > > > > > > > provided. >>> > > > > > > > > > > >>> > > > > > > > > > > Any thoughts? >>> > > > > > > > > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* >>> > > > > > > > > > > https://egkatzioura.com/ | >>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > > > > > > > > > https://github.com/gkatzioura >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >>
