Seshu, Thanks for the e-mail and for sharing your concerns!
So when we talk about combining multiple sources into a single flow, we don't mean that all data should be combined into a single flow. It absolutely makes sense to sometimes have very disparate flows! In some of the instances we've run, we have dozens or more disparate flows. The idea that I wanted to convey in the article is that just because 2 pieces of data come from different sources does not mean that they should be different flows. But if the data needs to be handled very differently then it absolutely should be two different flows. Those flows then can live side-by-side within the same instance of NiFi (generally in different Process Groups so that the graph is maintainable). The idea of how to handle security and authorization is definitely an ongoing debate. There are really two major approaches here. The first approach, which we offer today, is to have a separate instance of NiFi when different security and authorization is required. Remote Process Groups/site-to-site functionality is then used to send the data between flows. The rub here is that if you have many instances it can be different to manage them. The other approach would be to allow the security and authorization to take place at the Process Group level, rather than the Flow Controller level. This would be a very significant amount of work and may make the application more difficult to use, if the administrators then had to manage each group independently. So there are definitely trade-offs to each approach. If you have ideas about how you'd like to see it work, please share them so that we can make NiFi as useful as possible. Thanks -Mark ---------------------------------------- > From: [email protected] > To: [email protected] > Subject: Thoughts on the Blog Article [Apache NiFi: Thinking Differently > About DataFlow] > Date: Tue, 26 May 2015 20:48:16 +0000 > > Hello Folks, > > Finally got to install NiFi and got the sample flows running and read the > Blog article at https://blogs.apache.org/nifi/entry/basic_dataflow_design. > >> The question was "Is it possible to have NiFi service setup and running and >> allow for multiple dataflows to be designed and deployed (running) at the >> same time?” > > I understand the argument being made by the author on how you can use Nifi to > have a single flow with several inputs compared to several disparate flows. > But there are multiple advantages to having Nifi manage several disparate > flows. > > * Managing Flows that have very different transformations > * Security: Authorization, who has access to what flows, executing flows as a > named user instead of a super user. > * Resource Management: Scheduling the resources across disparate flows > * Etc > > Are there future plans to have Nifi Service setup and manage multiple data > flows? > > Regards > Seshu Adunuthula > > >
