If the only communication protocol allowed between the two nodes is CQL3, then yes, you can place NiFi on Node 1 and have the HTTP micro service communicate to the same host. Then NiFi will communicate to Node 2 via CQL. Regardless of which node it is residing on, for this scenario you only need one deployment of NiFi.
Multiple instances of NiFi come into play in the following scenarios:
* clustering [1]
* site-to-site [2] (a NiFi-specific protocol for sending NiFi flowfiles in
their native format between multiple instances of NiFi)
- see also, MiNiFi [3]
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site
[3] https://nifi.apache.org/minifi/index.html
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Dec 7, 2016, at 7:20 PM, kant kodali <kanth...@gmail.com> wrote:
>
> Hi Andy,
>
> I am a bit confused again. In my example my deployment is fixed right
>
> Node1 has my HTTP service and my Node 2 has Cassandra.
> The communication between Node1 and Node2 is through CQL3.
> A client will initiate the request to the my service on Node 1 via HTTP and
> you can assume a client is a browser.
>
> so if you are saying place Nifi on Node 2 and receive incoming HTTP request ?
> it doesn't quite make sense to me because the only communication protocol
> that is allowed between Node1 and Node 2 is CQL3 so I am not sure how one
> can receive HTTP requests in node 2 ?
>
> Thanks!
>
>
>
>
>
>
> On Wed, Dec 7, 2016 at 5:58 PM, Andy LoPresto <alopre...@apache.org
> <mailto:alopre...@apache.org>> wrote:
> Hi Kant,
>
> Before we get too much farther, I think the Overview [1] and Getting Started
> [2] guide and Joe Witt’s OSCON talk [3] might be valuable because they really
> underline the problem(s) NiFi solves and the behaviors it uses to do that.
>
> In the scenario you described, you only need one Nifi node (forgive the ASCII
> diagram).
>
> Node 1 Node 2
> micro service ---HTTP-—> NiFi ListenHTTP processor
> success cxn
> NiFi PutCassandraQL processor ---CQL--> Cassandra
> instance
>
> NiFi will receive the HTTP messages from your micro service and ingest each
> incoming request, generating a flowfile for each request, placing the request
> body into the content claim, and various header data into attributes. Then
> you will create a connection between the ListenHTTP (or HandleHTTPRequest
> processor if it’s more complex) and a PutCassandraQL processor. This
> processor will write the contents of the flowfile to the Cassandra instance
> over CQL3. Now, the PutCassandraQL processor expects the content of the
> incoming flowfile to be a CQL command, so let’s use the example where the
> command is a simple INSERT statement but the values depend on what is being
> generated by the micro service. In that case, you could use a ReplaceText
> processor between ListenHTTP and PutCassandraQL to format the query
> correctly. If the HTTP content was JSON, for example, you could use an
> EvaluateJsonPath processor to extract the values you care about from the
> content to attributes, and ReplaceText will populate the templated expression
> with the proper values using the NiFi Expression Language [4].
>
> For more examples, I’d suggest you look at the NiFi Template Registry [5] or
> watch some of the demo videos [6]. The User Guide [7] is also valuable.
> Hopefully all of these resources and our continuing conversations will help
> you understand and apply NiFi to solve your data flow challenges.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/overview.html
> <https://nifi.apache.org/docs/nifi-docs/html/overview.html>
> [2] https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
> <https://nifi.apache.org/docs/nifi-docs/html/getting-started.html>
> [3] https://youtu.be/sQCgtCoZyFQ <https://youtu.be/sQCgtCoZyFQ>
> [4]
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html>
> [5]
> https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
> <https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates>
> [6] https://nifi.apache.org/videos.html <https://nifi.apache.org/videos.html>
> [7] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html
> <https://nifi.apache.org/docs/nifi-docs/html/user-guide.html>
>
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Dec 7, 2016, at 5:31 PM, kant kodali <kanth...@gmail.com
>> <mailto:kanth...@gmail.com>> wrote:
>>
>> Hi Koji,
>>
>>
>> Thanks again for your response.
>>
>> "Should an application/micro service worry about how it talk with NiFi?
>> My answer would be no. The service should focus and use the best
>> protocol or technology for its purpose."
>>
>> This sounds great. This is what we want! I also understand the other
>> capabilities of Nifi you mentioned!
>>
>> I have the following question. Say I have a HTTP based micro service on node
>> 1 and a single process Cassandra on node 2 and say my micro service talks to
>> cassandra using CQL3 protocol. so how would I integrate NiFi ? Here is what
>> I understood so far from this thread.
>>
>> 1. Run a Nifi JVM on node 1 and node 2.
>> 2. Configure Nifi JVM on node 1 to use HTTP protocol for incoming messages
>> and for output messages configure Nifi JVM to use CQL3 protocol.
>> 3. Configure Nifi JVM on node 2 to use CQL3 protocol messages for both
>> incoming and outgoing messages.
>>
>> Is this correct so far? am I missing something?
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>> On Wed, Dec 7, 2016 at 4:47 PM, Koji Kawamura <ijokaruma...@gmail.com
>> <mailto:ijokaruma...@gmail.com>> wrote:
>> Hi Kant,
>>
>> Glad to know that you liked the explanation, thanks :)
>>
>> By reading discussion in this thread, I assume what you'd like to do
>> with NiFi is, get messages from a system amd do some routing/filtering
>> on NiFi, then send it to another system, using a custom protocol.
>> If so, you can write two custom NiFi processors, typically named as
>> GetXXXX and PutXXXX (XXXX is a name of the protocol).
>>
>> Based on this architecture, NiFi as a routing layer, if you need to
>> store received data into another datastore to extend the use of data
>> further, then NiFi will be really useful, because it already support
>> variety of data source and protocol integrations. Also, if data schema
>> changed at different rate between these systems, then NiFi can be a
>> schema migration point.
>>
>> Data ingested into NiFi called FlowFiles, which has opaque binary
>> Content, and string key/value pairs called Attributes, very generic
>> data format.
>> Once the data is converted to a FlowFile, there's no big difference
>> where the FlowFile came from via what protocol.
>> So, If NiFi can act as a server for different protocols, then your
>> NiFi data flow can support broader type of clients.
>>
>> There's many existing processors, act as server for certain protocol,
>> such as ListenTCP, ListenSyslog, ListenHTTP/HandleHTTPRequest ... etc.
>> Once you created PutXXXX for your custom protocol, then you can use
>> NiFi to support TCP, HTTP, WebSocket ... protocols to receive data to
>> integrate with the destination using XXXX protocol, e.g. ListenHTTP ->
>> some data processing on NiFi -> PutXXXX.
>>
>> Also, MiNiFi can be helpful to bring data integration to the edge.
>>
>> Should an application/micro service worry about how it talk with NiFi?
>> My answer would be no. The service should focus and use the best
>> protocol or technology for its purpose.
>> But knowing what NiFi can help extending use cases around the service,
>> would help you to focus more on the app/service itself.
>>
>> Thanks,
>> Koji
>>
>> On Wed, Dec 7, 2016 at 6:52 PM, kant kodali <kanth...@gmail.com
>> <mailto:kanth...@gmail.com>> wrote:
>> > I am also confused a little bit since I am new to Nifi. I wonder why Nifi
>> > would act as a server? isn't Nifi a routing layer between systems? because
>> > this brings in another question about Nifi in general.
>> >
>> > When I write my applications/microservices do I need to worry about how my
>> > service would talk to Nifi or do I have the freedom of just focusing on my
>> > application and using whatever protocol I want and In the end just plugin
>> > to
>> > Nifi which would take care? other words is Nifi a tight integration with
>> > applications such that I always have to import a Nifi Library within my
>> > application/microservice ? other words do I need to worry about Nifi at
>> > programming/development time of an Application/Microservice or at
>> > deployment
>> > time?
>> >
>> > Sorry if these are naive questions. But answers to those will help greatly
>> > and prevent me from asking more questions!
>> >
>> > Thanks much!
>> > kant
>> >
>> >
>> >
>> >
>> > On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com
>> > <mailto:kanth...@gmail.com>> wrote:
>> >>
>> >> Hi Koji,
>> >>
>> >> That is an awesome explanation! I expected processors for HTTP2 at very
>> >> least since it is widely used ( the entire GRPC stack runs on that). I am
>> >> not sure how easy or hard it is to build one?
>> >>
>> >> Thanks!
>> >>
>> >> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokaruma...@gmail.com
>> >> <mailto:ijokaruma...@gmail.com>>
>> >> wrote:
>> >>>
>> >>> Hi Kant,
>> >>>
>> >>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi
>> >>> has a set of processors for WebSocket since 1.1.0.
>> >>> It enables NiFi to act as a WebSocket client to communicate with a
>> >>> remote WebSocket server, or makes NiFi a WebSocket server so that
>> >>> remote clients access to it via WebSocket protocol.
>> >>>
>> >>> I've written a blog post about how to use it, I hope it will be useful
>> >>> for your use case:
>> >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/
>> >>> <http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/>
>> >>>
>> >>> Thanks,
>> >>> Koji
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth...@gmail.com
>> >>> <mailto:kanth...@gmail.com>> wrote:
>> >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another
>> >>> > convincing reason!)
>> >>> >
>> >>> > Coming back to the problem, We use NSQ a lot (although not my favorite)
>> >>> > and
>> >>> > want to be able integrate Nifi with NSQ to other systems such as Kafka,
>> >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2
>> >>> > and
>> >>> > some micro service that uses Websockets
>> >>> >
>> >>> > (which brings in other question if Nifi has HTTP2 and Websocket
>> >>> > processors?)
>> >>> >
>> >>> > Also below is NSQ protocol spec. They have Java client library as well.
>> >>> >
>> >>> > http://nsq.io/clients/tcp_protocol_spec.html
>> >>> > <http://nsq.io/clients/tcp_protocol_spec.html>
>> >>> >
>> >>> >
>> >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky
>> >>> > <ozhurakou...@hortonworks.com <mailto:ozhurakou...@hortonworks.com>>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi Kant
>> >>> >>
>> >>> >> What you’re trying to accomplish is definitely possible, however more
>> >>> >> information may be needed from you.
>> >>> >> For example, the way I understand your statement about “integration
>> >>> >> with
>> >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is
>> >>> >> the
>> >>> >> case such integration is definitely possible with your “custom system”
>> >>> >> by
>> >>> >> developing custom Processor and/or ControllerService.
>> >>> >> Processors and ControllerServices are the two main integration points
>> >>> >> within NiFi
>> >>> >> You can definitely find may examples by looking at some of the
>> >>> >> processors
>> >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.)
>> >>> >>
>> >>> >> Let us know if you need more help to guide you through the process.
>> >>> >>
>> >>> >> Cheers
>> >>> >> Oleg
>> >>> >>
>> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth...@gmail.com
>> >>> >> > <mailto:kanth...@gmail.com>> wrote:
>> >>> >> >
>> >>> >> > HI All,
>> >>> >> >
>> >>> >> > I understand that Apache Nifi has integration with many systems but
>> >>> >> > what
>> >>> >> > If I have an application that talks a custom protocol ? How do I
>> >>> >> > integrate
>> >>> >> > Apache Nifi with the custom protocol?
>> >>> >> >
>> >>> >> > Thanks,
>> >>> >> > kant
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>>
>
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
