If you point NiFi at a large dataset this is the expected behavior. We use micro-clusters of NiFi's (4 node cluster, 8CPU, 48GB ram). Different clusters for different business purposes/use cases.
java.arg.2=-Xms2g java.arg.3=-Xmx32g (48g machines - 20% less than system memory) 6 max threads per machine. If you max one of these clusters then we are doing something wrong or need to move those use cases to their own cluster. There is always a bottleneck somewhere and you will need to tweak the parameters of processors in your flow. The NiFi toolkit+registry has a nice commandline interface to copy your flows between different clusters. We ruthlessly automated the creation/destruction of these NiFi clusters so its trivial to bring up an ephemeral cluster, run a hungry flow, and destroy that cluster. Erik Anderson Bloomberg
