Sorry I should have been more clear. My question is even more simpler and naive. Say I am writing a HTTP based microservice (Now I assume Nifi has integration with HTTP since it is a well known protocol ). Now, how would I integrate Nifi with my HTTP based server?
On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky < ozhurakou...@hortonworks.com> wrote: > Kant > > There are couple of questions here so, let me try one at the time. > 1. “why Nifi would act as a server?”. Well, NiFi is a runtime environment > where things are *happening*. To be more exact; NiFi is a runtime > environment where things are *triggered*. The big distinction here is > trigger vs happening. In other words you may choose (as most processors do) > to have NiFi act as an execution container and run your code, or you may > chose for the NiFi to be a triggering container that triggers to run your > code elsewhere. Example of the later one is the SpringContextProcessor > which while still runs in the same JVM delegates execution to Spring > application that runs in its own container (which could as well be a > separate JVM or nah other container - ala micro services). So in this case > NiFi still manages the orchestration, mediation, security etc. > > 2. “...do I need to worry about how my service would talk to Nifi or do I > have the freedom of just focusing on my application and using whatever > protocol I want and In the end just plugin to Nifi which would take care?” > That is a loaded question and I must admit not fully understood, but i’ll > give it a try. When integrating with NiFi you use one of the integration > points such as Processor and/or ControllerService. Those are NiFi known > strategies (interfaces) and so your custom Processor would need to > implement such strategy and obviously be compliant with its contract. > However, the other part of your question is about implementing something > “independent” and just plug-in and if t’s possible. My answer is still YES > as long as you design it that way. As an example of such design you may > want to look at AMQP support where there are a pair of processors > (PublishAMQP/ConsumeAMQP), but when you look at the code of these > processors you’ll see that neither has any AMQP dependencies. Instead they > depend on another case (let’s call it Worker) and that class is completely > independent of NiFi. So in summary your protocol-specific Processor is > independent of the protocol and your protocol-specific Worker is > independent of NiFi and the two delegate between one another through common > to both object (in this case bye[]). The Spring processor mentioned above > implements the same pattern. > > And one more point about Microservices. I am willing to go as far as > saying that NiFi is a variation of Microservices container. I am basing it > n the fact that NiFi components (i.e., processors) implement a fundamental > micro services pattern - certain independence from it’s runtime and > persistence of its results - which allows each and every component to be > managed (start/stopped/reconfigured/changed) independently of > upstream/downstream components. > > Ok, that is a load to process, so I’ll stop ;) > > Look forward to more questions > > Cheers > Oleg > > > On Dec 7, 2016, at 4:52 AM, kant kodali <kanth...@gmail.com> wrote: > > I am also confused a little bit since I am new to Nifi. I wonder why Nifi > would act as a server? isn't Nifi a routing layer between systems? because > this brings in another question about Nifi in general. > > When I write my applications/microservices do I need to worry about how my > service would talk to Nifi or do I have the freedom of just focusing on my > application and using whatever protocol I want and In the end just plugin > to Nifi which would take care? other words is Nifi a tight integration with > applications such that I always have to import a Nifi Library within my > application/microservice ? other words do I need to worry about Nifi at > programming/development time of an Application/Microservice or at > deployment time? > > Sorry if these are naive questions. But answers to those will help greatly > and prevent me from asking more questions! > > Thanks much! > kant > > > > > On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com> wrote: > >> Hi Koji, >> >> That is an awesome explanation! I expected processors for HTTP2 at very >> least since it is widely used ( the entire GRPC stack runs on that). I am >> not sure how easy or hard it is to build one? >> >> Thanks! >> >> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com> >> wrote: >> >>> Hi Kant, >>> >>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>> has a set of processors for WebSocket since 1.1.0. >>> It enables NiFi to act as a WebSocket client to communicate with a >>> remote WebSocket server, or makes NiFi a WebSocket server so that >>> remote clients access to it via WebSocket protocol. >>> >>> I've written a blog post about how to use it, I hope it will be useful >>> for your use case: >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>> >>> Thanks, >>> Koji >>> >>> >>> >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> wrote: >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another >>> > convincing reason!) >>> > >>> > Coming back to the problem, We use NSQ a lot (although not my >>> favorite) and >>> > want to be able integrate Nifi with NSQ to other systems such as Kafka, >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>> and >>> > some micro service that uses Websockets >>> > >>> > (which brings in other question if Nifi has HTTP2 and Websocket >>> processors?) >>> > >>> > Also below is NSQ protocol spec. They have Java client library as well. >>> > >>> > http://nsq.io/clients/tcp_protocol_spec.html >>> > >>> > >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>> > <ozhurakou...@hortonworks.com> wrote: >>> >> >>> >> Hi Kant >>> >> >>> >> What you’re trying to accomplish is definitely possible, however more >>> >> information may be needed from you. >>> >> For example, the way I understand your statement about “integration >>> with >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is >>> the >>> >> case such integration is definitely possible with your “custom >>> system” by >>> >> developing custom Processor and/or ControllerService. >>> >> Processors and ControllerServices are the two main integration points >>> >> within NiFi >>> >> You can definitely find may examples by looking at some of the >>> processors >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>> >> >>> >> Let us know if you need more help to guide you through the process. >>> >> >>> >> Cheers >>> >> Oleg >>> >> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: >>> >> > >>> >> > HI All, >>> >> > >>> >> > I understand that Apache Nifi has integration with many systems but >>> what >>> >> > If I have an application that talks a custom protocol ? How do I >>> integrate >>> >> > Apache Nifi with the custom protocol? >>> >> > >>> >> > Thanks, >>> >> > kant >>> >> >>> > >>> >> >> > >
