Hi Nishant, A HTTP load-balancer as Bryan suggests is a good option for server-side load balancing so that your client only needs to keep track of a single URL. In addition to the options Bryan mentioned, which are great, most cloud providers offer managed load balancing services, which many be convenient if you happen to be using a commercial cloud for your NiFi cluster.
Another option is client-side load balancing, in which clients of the NiFi cluster discover all available nodes in the cluster and cycle through them. There are many ways of implementing client-side load balancing, most of which involve periodically refreshing a list of server nodes/endpoints. If you want to explore this path, the NiFi API [1] makes it easy to discover all nodes known to the cluster and their status using this endpoint: GET /controller/cluster This can be called on any node in the cluster. It will return a JSON object that includes an array of nodes, each of which has an address and port you could use to build a URL as well as some other status information that might be useful for implementing dynamic load-balancing logic in the client. This could be consumed directly by the client or by an agent of a service discovery framework that would maintain a directory/registry of endpoints. Examples of frameworks that can be used as a service discovery registry include Apache Zookeeper, Consul, Netlfix Eureka, and etcd. The easiest way to get started for most is definitely the server-side load balancing approach using an HTTP load balancer. There are pros and cons to both server-side and client-side load balancing that one has to evaluate based on a particular use case. So I agree with Bryan's advice, and am only pointing out another option that may be preferable to some on this list that have a similar need. [1] https://nifi.apache.org/docs/nifi-docs/rest-api/ Cheers, Kevin On 8/29/17, 12:15, "Bryan Bende" <bbe...@gmail.com> wrote: Hello, You can run a standard HTTP load-balancer in front of ListenHTTP and have your producers use the URL of the load-balancer. Nginx or apache httpd can be used. Thanks, Bryan On Tue, Aug 29, 2017 at 11:40 AM, mayank rathi <mayank.ra...@gmail.com> wrote: > Does this help? > > [image: Inline image 1] > > On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta < > nishantgupta1...@gmail.com> wrote: > >> Hello Sir/Madam, >> >> I need to know how we can implement load balancing and single point access >> to all data producer >> >> I have suppose 100 machines that produce data.(flowfiles) >> That data is sent over HTTP from 100 machines to nifi and that we can >> access using listenHTTP processor in Nifi.(Currently I am able to do that >> for single node Nifi) >> Is there any way to provide 100 machines with a single URL (for all 4-5 >> nodes in Nifi Cluster) >> >> Can you please let me know how to achieve it.?? >> >> Thanks and Regards, >> Nishant Gupta >> > > > > -- > NOTICE: This email message is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply email > and destroy all copies of the original message. >
