David, Glad you are liking it! As you explore and learn more feel free to fire away with feedback on things you find confusing, too limiting, or awesome (not necessarily in that order).
Each node in a cluster should be able to handle 100s of thousands of operations per second on the data and several hundred MB/s of read/write throughput. So the scale you're suggesting is feasible. Of course at those numbers the details do tend to matter so if you have questions just fire away. Thanks Joe On Tue, Apr 28, 2015 at 2:33 PM, Matt Gilman <[email protected]> wrote: > David, > > Welcome and thanks for expressing interest in Apache NiFi. I just noticed > that the administrator guide [1] on our website [2] was not in its current > form so just uploaded the latest version. The document now includes a quick > explanation of our clustering capabilities and example configurations. This > would be a great place to start and become familiar with NiFi clustering. > Please let us know if you have any follow up questions. > > Also, if you had already viewed the administrator guide your browser may > have cached the older version so you may need to do a hard reload. > > [1] > https://nifi.incubator.apache.org/docs/nifi-docs/administration-guide.html > [2] https://nifi.incubator.apache.org/ > > On Tue, Apr 28, 2015 at 2:06 PM, David Klim <[email protected]> wrote: >> >> Hello, >> >> Just joined the list, I am evaluating NiFi for a large project to see if >> NiFi would fit as the main data collector. So far I am quite impressed with >> it's capabilities, the concept is just great! >> >> The project I am working on would require retrieving several hundreds of >> millions of files per day (hundreds of TB per day) so my first question is >> how to achieve distribution/clustering with NiFi, if that's possible. >> >> Thanks in advance! >> >> >> >> >> >
