Re: How to integrate a custom protocol with Apache Nifi
time of an Application/Microservice or at >> > deployment >> > time? >> > >> > Sorry if these are naive questions. But answers to those will help greatly >> > and prevent me from asking more questions! >> > >> > Thanks much! >> > kant >> > >> > >> > >> > >> > On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com >> > <mailto:kanth...@gmail.com>> wrote: >> >> >> >> Hi Koji, >> >> >> >> That is an awesome explanation! I expected processors for HTTP2 at very >> >> least since it is widely used ( the entire GRPC stack runs on that). I am >> >> not sure how easy or hard it is to build one? >> >> >> >> Thanks! >> >> >> >> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com >> >> <mailto:ijokaruma...@gmail.com>> >> >> wrote: >> >>> >> >>> Hi Kant, >> >>> >> >>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >> >>> has a set of processors for WebSocket since 1.1.0. >> >>> It enables NiFi to act as a WebSocket client to communicate with a >> >>> remote WebSocket server, or makes NiFi a WebSocket server so that >> >>> remote clients access to it via WebSocket protocol. >> >>> >> >>> I've written a blog post about how to use it, I hope it will be useful >> >>> for your use case: >> >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >> >>> <http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/> >> >>> >> >>> Thanks, >> >>> Koji >> >>> >> >>> >> >>> >> >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com >> >>> <mailto:kanth...@gmail.com>> wrote: >> >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another >> >>> > convincing reason!) >> >>> > >> >>> > Coming back to the problem, We use NSQ a lot (although not my favorite) >> >>> > and >> >>> > want to be able integrate Nifi with NSQ to other systems such as Kafka, >> >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >> >>> > and >> >>> > some micro service that uses Websockets >> >>> > >> >>> > (which brings in other question if Nifi has HTTP2 and Websocket >> >>> > processors?) >> >>> > >> >>> > Also below is NSQ protocol spec. They have Java client library as well. >> >>> > >> >>> > http://nsq.io/clients/tcp_protocol_spec.html >> >>> > <http://nsq.io/clients/tcp_protocol_spec.html> >> >>> > >> >>> > >> >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >> >>> > <ozhurakou...@hortonworks.com <mailto:ozhurakou...@hortonworks.com>> >> >>> > wrote: >> >>> >> >> >>> >> Hi Kant >> >>> >> >> >>> >> What you’re trying to accomplish is definitely possible, however more >> >>> >> information may be needed from you. >> >>> >> For example, the way I understand your statement about “integration >> >>> >> with >> >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is >> >>> >> the >> >>> >> case such integration is definitely possible with your “custom system” >> >>> >> by >> >>> >> developing custom Processor and/or ControllerService. >> >>> >> Processors and ControllerServices are the two main integration points >> >>> >> within NiFi >> >>> >> You can definitely find may examples by looking at some of the >> >>> >> processors >> >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >>> >> >> >>> >> Let us know if you need more help to guide you through the process. >> >>> >> >> >>> >> Cheers >> >>> >> Oleg >> >>> >> >> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com >> >>> >> > <mailto:kanth...@gmail.com>> wrote: >> >>> >> > >> >>> >> > HI All, >> >>> >> > >> >>> >> > I understand that Apache Nifi has integration with many systems but >> >>> >> > what >> >>> >> > If I have an application that talks a custom protocol ? How do I >> >>> >> > integrate >> >>> >> > Apache Nifi with the custom protocol? >> >>> >> > >> >>> >> > Thanks, >> >>> >> > kant >> >>> >> >> >>> > >> >> >> >> >> > >> > > signature.asc Description: Message signed with OpenPGP using GPGMail
Re: How to integrate a custom protocol with Apache Nifi
kousky > >>> > <ozhurakou...@hortonworks.com <mailto:ozhurakou...@hortonworks.com>> > >>> > wrote: > >>> >> > >>> >> Hi Kant > >>> >> > >>> >> What you’re trying to accomplish is definitely possible, however more > >>> >> information may be needed from you. > >>> >> For example, the way I understand your statement about “integration > >>> >> with > >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is > >>> >> the > >>> >> case such integration is definitely possible with your “custom system” > >>> >> by > >>> >> developing custom Processor and/or ControllerService. > >>> >> Processors and ControllerServices are the two main integration points > >>> >> within NiFi > >>> >> You can definitely find may examples by looking at some of the > >>> >> processors > >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) > >>> >> > >>> >> Let us know if you need more help to guide you through the process. > >>> >> > >>> >> Cheers > >>> >> Oleg > >>> >> > >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com > >>> >> > <mailto:kanth...@gmail.com>> wrote: > >>> >> > > >>> >> > HI All, > >>> >> > > >>> >> > I understand that Apache Nifi has integration with many systems but > >>> >> > what > >>> >> > If I have an application that talks a custom protocol ? How do I > >>> >> > integrate > >>> >> > Apache Nifi with the custom protocol? > >>> >> > > >>> >> > Thanks, > >>> >> > kant > >>> >> > >>> > > >> > >> > > > signature.asc Description: Message signed with OpenPGP using GPGMail
Re: How to integrate a custom protocol with Apache Nifi
; Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi > >>> has a set of processors for WebSocket since 1.1.0. > >>> It enables NiFi to act as a WebSocket client to communicate with a > >>> remote WebSocket server, or makes NiFi a WebSocket server so that > >>> remote clients access to it via WebSocket protocol. > >>> > >>> I've written a blog post about how to use it, I hope it will be useful > >>> for your use case: > >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ > >>> > >>> Thanks, > >>> Koji > >>> > >>> > >>> > >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> > wrote: > >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! > (Another > >>> > convincing reason!) > >>> > > >>> > Coming back to the problem, We use NSQ a lot (although not my > favorite) > >>> > and > >>> > want to be able integrate Nifi with NSQ to other systems such as > Kafka, > >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 > >>> > and > >>> > some micro service that uses Websockets > >>> > > >>> > (which brings in other question if Nifi has HTTP2 and Websocket > >>> > processors?) > >>> > > >>> > Also below is NSQ protocol spec. They have Java client library as > well. > >>> > > >>> > http://nsq.io/clients/tcp_protocol_spec.html > >>> > > >>> > > >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky > >>> > <ozhurakou...@hortonworks.com> wrote: > >>> >> > >>> >> Hi Kant > >>> >> > >>> >> What you’re trying to accomplish is definitely possible, however > more > >>> >> information may be needed from you. > >>> >> For example, the way I understand your statement about “integration > >>> >> with > >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that > is > >>> >> the > >>> >> case such integration is definitely possible with your “custom > system” > >>> >> by > >>> >> developing custom Processor and/or ControllerService. > >>> >> Processors and ControllerServices are the two main integration > points > >>> >> within NiFi > >>> >> You can definitely find may examples by looking at some of the > >>> >> processors > >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) > >>> >> > >>> >> Let us know if you need more help to guide you through the process. > >>> >> > >>> >> Cheers > >>> >> Oleg > >>> >> > >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> > wrote: > >>> >> > > >>> >> > HI All, > >>> >> > > >>> >> > I understand that Apache Nifi has integration with many systems > but > >>> >> > what > >>> >> > If I have an application that talks a custom protocol ? How do I > >>> >> > integrate > >>> >> > Apache Nifi with the custom protocol? > >>> >> > > >>> >> > Thanks, > >>> >> > kant > >>> >> > >>> > > >> > >> > > >
Re: How to integrate a custom protocol with Apache Nifi
ey have Java client library as well. >>> > >>> > http://nsq.io/clients/tcp_protocol_spec.html >>> > >>> > >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>> > <ozhurakou...@hortonworks.com> wrote: >>> >> >>> >> Hi Kant >>> >> >>> >> What you’re trying to accomplish is definitely possible, however more >>> >> information may be needed from you. >>> >> For example, the way I understand your statement about “integration >>> >> with >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is >>> >> the >>> >> case such integration is definitely possible with your “custom system” >>> >> by >>> >> developing custom Processor and/or ControllerService. >>> >> Processors and ControllerServices are the two main integration points >>> >> within NiFi >>> >> You can definitely find may examples by looking at some of the >>> >> processors >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>> >> >>> >> Let us know if you need more help to guide you through the process. >>> >> >>> >> Cheers >>> >> Oleg >>> >> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: >>> >> > >>> >> > HI All, >>> >> > >>> >> > I understand that Apache Nifi has integration with many systems but >>> >> > what >>> >> > If I have an application that talks a custom protocol ? How do I >>> >> > integrate >>> >> > Apache Nifi with the custom protocol? >>> >> > >>> >> > Thanks, >>> >> > kant >>> >> >>> > >> >> >
Re: How to integrate a custom protocol with Apache Nifi
nd the two delegate between one another through common >>> to both object (in this case bye[]). The Spring processor mentioned above >>> implements the same pattern. >>> >>> And one more point about Microservices. I am willing to go as far as >>> saying that NiFi is a variation of Microservices container. I am basing it >>> n the fact that NiFi components (i.e., processors) implement a fundamental >>> micro services pattern - certain independence from it’s runtime and >>> persistence of its results - which allows each and every component to be >>> managed (start/stopped/reconfigured/changed) independently of >>> upstream/downstream components. >>> >>> Ok, that is a load to process, so I’ll stop ;) >>> >>> Look forward to more questions >>> >>> Cheers >>> Oleg >>> >>> >>> On Dec 7, 2016, at 4:52 AM, kant kodali <kanth...@gmail.com> wrote: >>> >>> I am also confused a little bit since I am new to Nifi. I wonder why >>> Nifi would act as a server? isn't Nifi a routing layer between systems? >>> because this brings in another question about Nifi in general. >>> >>> When I write my applications/microservices do I need to worry about how >>> my service would talk to Nifi or do I have the freedom of just focusing on >>> my application and using whatever protocol I want and In the end just >>> plugin to Nifi which would take care? other words is Nifi a tight >>> integration with applications such that I always have to import a Nifi >>> Library within my application/microservice ? other words do I need to worry >>> about Nifi at programming/development time of an Application/Microservice >>> or at deployment time? >>> >>> Sorry if these are naive questions. But answers to those will help >>> greatly and prevent me from asking more questions! >>> >>> Thanks much! >>> kant >>> >>> >>> >>> >>> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com> wrote: >>> >>>> Hi Koji, >>>> >>>> That is an awesome explanation! I expected processors for HTTP2 at very >>>> least since it is widely used ( the entire GRPC stack runs on that). I am >>>> not sure how easy or hard it is to build one? >>>> >>>> Thanks! >>>> >>>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com> >>>> wrote: >>>> >>>>> Hi Kant, >>>>> >>>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>>>> has a set of processors for WebSocket since 1.1.0. >>>>> It enables NiFi to act as a WebSocket client to communicate with a >>>>> remote WebSocket server, or makes NiFi a WebSocket server so that >>>>> remote clients access to it via WebSocket protocol. >>>>> >>>>> I've written a blog post about how to use it, I hope it will be useful >>>>> for your use case: >>>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>>>> >>>>> Thanks, >>>>> Koji >>>>> >>>>> >>>>> >>>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> >>>>> wrote: >>>>> > Thanks a ton guys! Didn't expect Nifi community to be so good! >>>>> (Another >>>>> > convincing reason!) >>>>> > >>>>> > Coming back to the problem, We use NSQ a lot (although not my >>>>> favorite) and >>>>> > want to be able integrate Nifi with NSQ to other systems such as >>>>> Kafka, >>>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>>>> and >>>>> > some micro service that uses Websockets >>>>> > >>>>> > (which brings in other question if Nifi has HTTP2 and Websocket >>>>> processors?) >>>>> > >>>>> > Also below is NSQ protocol spec. They have Java client library as >>>>> well. >>>>> > >>>>> > http://nsq.io/clients/tcp_protocol_spec.html >>>>> > >>>>> > >>>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>>>> > <ozhurakou...@hortonworks.com> wrote: >>>>> >> >>>>> >> Hi Kant >>>>> >> >>>>> >> What you’re trying to accomplish is definitely possible, however >>>>> more >>>>> >> information may be needed from you. >>>>> >> For example, the way I understand your statement about “integration >>>>> with >>>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that >>>>> is the >>>>> >> case such integration is definitely possible with your “custom >>>>> system” by >>>>> >> developing custom Processor and/or ControllerService. >>>>> >> Processors and ControllerServices are the two main integration >>>>> points >>>>> >> within NiFi >>>>> >> You can definitely find may examples by looking at some of the >>>>> processors >>>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>>>> >> >>>>> >> Let us know if you need more help to guide you through the process. >>>>> >> >>>>> >> Cheers >>>>> >> Oleg >>>>> >> >>>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> >>>>> wrote: >>>>> >> > >>>>> >> > HI All, >>>>> >> > >>>>> >> > I understand that Apache Nifi has integration with many systems >>>>> but what >>>>> >> > If I have an application that talks a custom protocol ? How do I >>>>> integrate >>>>> >> > Apache Nifi with the custom protocol? >>>>> >> > >>>>> >> > Thanks, >>>>> >> > kant >>>>> >> >>>>> > >>>>> >>>> >>>> >>> >>> >> >> > >
Re: How to integrate a custom protocol with Apache Nifi
atever protocol I want and In the end just plugin to Nifi which would take care? other words is Nifi a tight integration with applications such that I always have to import a Nifi Library within my application/microservice ? other words do I need to worry about Nifi at programming/development time of an Application/Microservice or at deployment time? Sorry if these are naive questions. But answers to those will help greatly and prevent me from asking more questions! Thanks much! kant On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: Hi Koji, That is an awesome explanation! I expected processors for HTTP2 at very least since it is widely used ( the entire GRPC stack runs on that). I am not sure how easy or hard it is to build one? Thanks! On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com<mailto:ijokaruma...@gmail.com>> wrote: Hi Kant, Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi has a set of processors for WebSocket since 1.1.0. It enables NiFi to act as a WebSocket client to communicate with a remote WebSocket server, or makes NiFi a WebSocket server so that remote clients access to it via WebSocket protocol. I've written a blog post about how to use it, I hope it will be useful for your use case: http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ Thanks, Koji On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another > convincing reason!) > > Coming back to the problem, We use NSQ a lot (although not my favorite) and > want to be able integrate Nifi with NSQ to other systems such as Kafka, > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 and > some micro service that uses Websockets > > (which brings in other question if Nifi has HTTP2 and Websocket processors?) > > Also below is NSQ protocol spec. They have Java client library as well. > > http://nsq.io/clients/tcp_protocol_spec.html > > > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky > <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: >> >> Hi Kant >> >> What you’re trying to accomplish is definitely possible, however more >> information may be needed from you. >> For example, the way I understand your statement about “integration with >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the >> case such integration is definitely possible with your “custom system” by >> developing custom Processor and/or ControllerService. >> Processors and ControllerServices are the two main integration points >> within NiFi >> You can definitely find may examples by looking at some of the processors >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >> Let us know if you need more help to guide you through the process. >> >> Cheers >> Oleg >> >> > On Dec 6, 2016, at 7:46 PM, kant kodali >> > <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: >> > >> > HI All, >> > >> > I understand that Apache Nifi has integration with many systems but what >> > If I have an application that talks a custom protocol ? How do I integrate >> > Apache Nifi with the custom protocol? >> > >> > Thanks, >> > kant >> >
Re: How to integrate a custom protocol with Apache Nifi
on about Nifi in general. >> >> When I write my applications/microservices do I need to worry about how >> my service would talk to Nifi or do I have the freedom of just focusing on >> my application and using whatever protocol I want and In the end just >> plugin to Nifi which would take care? other words is Nifi a tight >> integration with applications such that I always have to import a Nifi >> Library within my application/microservice ? other words do I need to worry >> about Nifi at programming/development time of an Application/Microservice >> or at deployment time? >> >> Sorry if these are naive questions. But answers to those will help >> greatly and prevent me from asking more questions! >> >> Thanks much! >> kant >> >> >> >> >> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com> wrote: >> >>> Hi Koji, >>> >>> That is an awesome explanation! I expected processors for HTTP2 at very >>> least since it is widely used ( the entire GRPC stack runs on that). I am >>> not sure how easy or hard it is to build one? >>> >>> Thanks! >>> >>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com> >>> wrote: >>> >>>> Hi Kant, >>>> >>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>>> has a set of processors for WebSocket since 1.1.0. >>>> It enables NiFi to act as a WebSocket client to communicate with a >>>> remote WebSocket server, or makes NiFi a WebSocket server so that >>>> remote clients access to it via WebSocket protocol. >>>> >>>> I've written a blog post about how to use it, I hope it will be useful >>>> for your use case: >>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>>> >>>> Thanks, >>>> Koji >>>> >>>> >>>> >>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> wrote: >>>> > Thanks a ton guys! Didn't expect Nifi community to be so good! >>>> (Another >>>> > convincing reason!) >>>> > >>>> > Coming back to the problem, We use NSQ a lot (although not my >>>> favorite) and >>>> > want to be able integrate Nifi with NSQ to other systems such as >>>> Kafka, >>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>>> and >>>> > some micro service that uses Websockets >>>> > >>>> > (which brings in other question if Nifi has HTTP2 and Websocket >>>> processors?) >>>> > >>>> > Also below is NSQ protocol spec. They have Java client library as >>>> well. >>>> > >>>> > http://nsq.io/clients/tcp_protocol_spec.html >>>> > >>>> > >>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>>> > <ozhurakou...@hortonworks.com> wrote: >>>> >> >>>> >> Hi Kant >>>> >> >>>> >> What you’re trying to accomplish is definitely possible, however more >>>> >> information may be needed from you. >>>> >> For example, the way I understand your statement about “integration >>>> with >>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that >>>> is the >>>> >> case such integration is definitely possible with your “custom >>>> system” by >>>> >> developing custom Processor and/or ControllerService. >>>> >> Processors and ControllerServices are the two main integration points >>>> >> within NiFi >>>> >> You can definitely find may examples by looking at some of the >>>> processors >>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>>> >> >>>> >> Let us know if you need more help to guide you through the process. >>>> >> >>>> >> Cheers >>>> >> Oleg >>>> >> >>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >> > >>>> >> > HI All, >>>> >> > >>>> >> > I understand that Apache Nifi has integration with many systems >>>> but what >>>> >> > If I have an application that talks a custom protocol ? How do I >>>> integrate >>>> >> > Apache Nifi with the custom protocol? >>>> >> > >>>> >> > Thanks, >>>> >> > kant >>>> >> >>>> > >>>> >>> >>> >> >> > >
Re: How to integrate a custom protocol with Apache Nifi
oji, That is an awesome explanation! I expected processors for HTTP2 at very least since it is widely used ( the entire GRPC stack runs on that). I am not sure how easy or hard it is to build one? Thanks! On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com<mailto:ijokaruma...@gmail.com>> wrote: Hi Kant, Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi has a set of processors for WebSocket since 1.1.0. It enables NiFi to act as a WebSocket client to communicate with a remote WebSocket server, or makes NiFi a WebSocket server so that remote clients access to it via WebSocket protocol. I've written a blog post about how to use it, I hope it will be useful for your use case: http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ Thanks, Koji On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another > convincing reason!) > > Coming back to the problem, We use NSQ a lot (although not my favorite) and > want to be able integrate Nifi with NSQ to other systems such as Kafka, > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 and > some micro service that uses Websockets > > (which brings in other question if Nifi has HTTP2 and Websocket processors?) > > Also below is NSQ protocol spec. They have Java client library as well. > > http://nsq.io/clients/tcp_protocol_spec.html > > > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky > <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: >> >> Hi Kant >> >> What you’re trying to accomplish is definitely possible, however more >> information may be needed from you. >> For example, the way I understand your statement about “integration with >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the >> case such integration is definitely possible with your “custom system” by >> developing custom Processor and/or ControllerService. >> Processors and ControllerServices are the two main integration points >> within NiFi >> You can definitely find may examples by looking at some of the processors >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >> Let us know if you need more help to guide you through the process. >> >> Cheers >> Oleg >> >> > On Dec 6, 2016, at 7:46 PM, kant kodali >> > <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: >> > >> > HI All, >> > >> > I understand that Apache Nifi has integration with many systems but what >> > If I have an application that talks a custom protocol ? How do I integrate >> > Apache Nifi with the custom protocol? >> > >> > Thanks, >> > kant >> >
Re: How to integrate a custom protocol with Apache Nifi
gt; Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>> has a set of processors for WebSocket since 1.1.0. >>> It enables NiFi to act as a WebSocket client to communicate with a >>> remote WebSocket server, or makes NiFi a WebSocket server so that >>> remote clients access to it via WebSocket protocol. >>> >>> I've written a blog post about how to use it, I hope it will be useful >>> for your use case: >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>> >>> Thanks, >>> Koji >>> >>> >>> >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> wrote: >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another >>> > convincing reason!) >>> > >>> > Coming back to the problem, We use NSQ a lot (although not my >>> favorite) and >>> > want to be able integrate Nifi with NSQ to other systems such as Kafka, >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>> and >>> > some micro service that uses Websockets >>> > >>> > (which brings in other question if Nifi has HTTP2 and Websocket >>> processors?) >>> > >>> > Also below is NSQ protocol spec. They have Java client library as well. >>> > >>> > http://nsq.io/clients/tcp_protocol_spec.html >>> > >>> > >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>> > <ozhurakou...@hortonworks.com> wrote: >>> >> >>> >> Hi Kant >>> >> >>> >> What you’re trying to accomplish is definitely possible, however more >>> >> information may be needed from you. >>> >> For example, the way I understand your statement about “integration >>> with >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is >>> the >>> >> case such integration is definitely possible with your “custom >>> system” by >>> >> developing custom Processor and/or ControllerService. >>> >> Processors and ControllerServices are the two main integration points >>> >> within NiFi >>> >> You can definitely find may examples by looking at some of the >>> processors >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>> >> >>> >> Let us know if you need more help to guide you through the process. >>> >> >>> >> Cheers >>> >> Oleg >>> >> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: >>> >> > >>> >> > HI All, >>> >> > >>> >> > I understand that Apache Nifi has integration with many systems but >>> what >>> >> > If I have an application that talks a custom protocol ? How do I >>> integrate >>> >> > Apache Nifi with the custom protocol? >>> >> > >>> >> > Thanks, >>> >> > kant >>> >> >>> > >>> >> >> > >
Re: How to integrate a custom protocol with Apache Nifi
blem, We use NSQ a lot (although not my favorite) and > want to be able integrate Nifi with NSQ to other systems such as Kafka, > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 and > some micro service that uses Websockets > > (which brings in other question if Nifi has HTTP2 and Websocket processors?) > > Also below is NSQ protocol spec. They have Java client library as well. > > http://nsq.io/clients/tcp_protocol_spec.html > > > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky > <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: >> >> Hi Kant >> >> What you’re trying to accomplish is definitely possible, however more >> information may be needed from you. >> For example, the way I understand your statement about “integration with >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the >> case such integration is definitely possible with your “custom system” by >> developing custom Processor and/or ControllerService. >> Processors and ControllerServices are the two main integration points >> within NiFi >> You can definitely find may examples by looking at some of the processors >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >> Let us know if you need more help to guide you through the process. >> >> Cheers >> Oleg >> >> > On Dec 6, 2016, at 7:46 PM, kant kodali >> > <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: >> > >> > HI All, >> > >> > I understand that Apache Nifi has integration with many systems but what >> > If I have an application that talks a custom protocol ? How do I integrate >> > Apache Nifi with the custom protocol? >> > >> > Thanks, >> > kant >> >
Re: How to integrate a custom protocol with Apache Nifi
I am also confused a little bit since I am new to Nifi. I wonder why Nifi would act as a server? isn't Nifi a routing layer between systems? because this brings in another question about Nifi in general. When I write my applications/microservices do I need to worry about how my service would talk to Nifi or do I have the freedom of just focusing on my application and using whatever protocol I want and In the end just plugin to Nifi which would take care? other words is Nifi a tight integration with applications such that I always have to import a Nifi Library within my application/microservice ? other words do I need to worry about Nifi at programming/development time of an Application/Microservice or at deployment time? Sorry if these are naive questions. But answers to those will help greatly and prevent me from asking more questions! Thanks much! kant On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com> wrote: > Hi Koji, > > That is an awesome explanation! I expected processors for HTTP2 at very > least since it is widely used ( the entire GRPC stack runs on that). I am > not sure how easy or hard it is to build one? > > Thanks! > > On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: > >> Hi Kant, >> >> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >> has a set of processors for WebSocket since 1.1.0. >> It enables NiFi to act as a WebSocket client to communicate with a >> remote WebSocket server, or makes NiFi a WebSocket server so that >> remote clients access to it via WebSocket protocol. >> >> I've written a blog post about how to use it, I hope it will be useful >> for your use case: >> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >> >> Thanks, >> Koji >> >> >> >> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com> wrote: >> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another >> > convincing reason!) >> > >> > Coming back to the problem, We use NSQ a lot (although not my favorite) >> and >> > want to be able integrate Nifi with NSQ to other systems such as Kafka, >> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 and >> > some micro service that uses Websockets >> > >> > (which brings in other question if Nifi has HTTP2 and Websocket >> processors?) >> > >> > Also below is NSQ protocol spec. They have Java client library as well. >> > >> > http://nsq.io/clients/tcp_protocol_spec.html >> > >> > >> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >> > <ozhurakou...@hortonworks.com> wrote: >> >> >> >> Hi Kant >> >> >> >> What you’re trying to accomplish is definitely possible, however more >> >> information may be needed from you. >> >> For example, the way I understand your statement about “integration >> with >> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is >> the >> >> case such integration is definitely possible with your “custom system” >> by >> >> developing custom Processor and/or ControllerService. >> >> Processors and ControllerServices are the two main integration points >> >> within NiFi >> >> You can definitely find may examples by looking at some of the >> processors >> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >> >> >> Let us know if you need more help to guide you through the process. >> >> >> >> Cheers >> >> Oleg >> >> >> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: >> >> > >> >> > HI All, >> >> > >> >> > I understand that Apache Nifi has integration with many systems but >> what >> >> > If I have an application that talks a custom protocol ? How do I >> integrate >> >> > Apache Nifi with the custom protocol? >> >> > >> >> > Thanks, >> >> > kant >> >> >> > >> > >
Re: How to integrate a custom protocol with Apache Nifi
Thanks a ton guys! Didn't expect Nifi community to be so good! (Another convincing reason!) Coming back to the problem, We use NSQ a lot (although not my favorite) and want to be able integrate Nifi with NSQ to other systems such as Kafka, Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 and some micro service that uses Websockets (which brings in other question if Nifi has HTTP2 and Websocket processors?) Also below is NSQ protocol spec. They have Java client library as well. http://nsq.io/clients/tcp_protocol_spec.html On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky < ozhurakou...@hortonworks.com> wrote: > Hi Kant > > What you’re trying to accomplish is definitely possible, however more > information may be needed from you. > For example, the way I understand your statement about “integration with > many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the > case such integration is definitely possible with your “custom system” by > developing custom Processor and/or ControllerService. > Processors and ControllerServices are the two main integration points > within NiFi > You can definitely find may examples by looking at some of the processors > (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) > > Let us know if you need more help to guide you through the process. > > Cheers > Oleg > > > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: > > > > HI All, > > > > I understand that Apache Nifi has integration with many systems but what > If I have an application that talks a custom protocol ? How do I integrate > Apache Nifi with the custom protocol? > > > > Thanks, > > kant > >
Re: How to integrate a custom protocol with Apache Nifi
Hi Kant What you’re trying to accomplish is definitely possible, however more information may be needed from you. For example, the way I understand your statement about “integration with many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the case such integration is definitely possible with your “custom system” by developing custom Processor and/or ControllerService. Processors and ControllerServices are the two main integration points within NiFi You can definitely find may examples by looking at some of the processors (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) Let us know if you need more help to guide you through the process. Cheers Oleg > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com> wrote: > > HI All, > > I understand that Apache Nifi has integration with many systems but what If I > have an application that talks a custom protocol ? How do I integrate Apache > Nifi with the custom protocol? > > Thanks, > kant
Re: How to integrate a custom protocol with Apache Nifi
Kant Typically this is as simple as creating your processor to utilize that protocol. Take a look at the developer guide [1] and of course you can always look at the many example processors and components that come in NiFi out of the box [2]. [1] https://nifi.apache.org/docs.html [2] https://git-wip-us.apache.org/repos/asf/nifi.git Thanks Joe On Tue, Dec 6, 2016 at 6:46 PM, kant kodali <kanth...@gmail.com> wrote: > HI All, > > I understand that Apache Nifi has integration with many systems but what If > I have an application that talks a custom protocol ? How do I integrate > Apache Nifi with the custom protocol? > > Thanks, > kant
Re: How to integrate a custom protocol with Apache Nifi
Kant, Look into a custom processor. You have a choice of either implementing e.g. a parser for your data, or, if the protocol is more involved, implement a receiver in the processor as well which would spit out meaningful data messages next. Andrew On Tue, Dec 6, 2016, 7:47 PM kant kodali <kanth...@gmail.com> wrote: > HI All, > > I understand that Apache Nifi has integration with many systems but what > If I have an application that talks a custom protocol ? How do I integrate > Apache Nifi with the custom protocol? > > Thanks, > kant >
How to integrate a custom protocol with Apache Nifi
HI All, I understand that Apache Nifi has integration with many systems but what If I have an application that talks a custom protocol ? How do I integrate Apache Nifi with the custom protocol? Thanks, kant
