Hi All, We've got a NiFi 3 Node Cluster running on 3 x 40 CPU, 256GB RAM (32G Java Heap) servers. However, we have only been able to achieve a consumption of ~9.48GB Consumption Compressed (38.53GB Uncompressed) over 5 minutes, with a production rate of ~16.84GB out of the cluster over 5 mins. This is much lower than we were expecting based on what we have read. With this throughput we see a CPU load ~32 on all nodes, so we know there isn't much else we can get out of the CPU).
We have also tried SSDs, Raided and Unraided HDDs for the content repo storage, but they haven't made a difference to the amount we can process. The process is as follows: 1. Our flow reads from Kafka Compressed (Maximum of 2000 records per file). It then converts them from Avro to JSON. (ConsumeKafka_0_10 --> UpdateAttribute --> ConvertRecord) 2. Depending on which topic the flow file is consumed from, we then send the message to one of 10 potential process groups, each containing between 3 and 5 processors within the process groups. (RouteOnAttribute --> Relevant Processing Group containing JoltTransformJSON and several custom processors we have made). 3. Finally, we produce the flow file content back to one of several Kafka topics, based on the input topic name in Avro format with Snappy compression on the Kafka topic. Inspecting the queued message counts, it indicates that the Jolt Transforms are taking the time to process (Large queues before JOLT processors, small or no queues afterwards). But I'm not sure why this is any worse than the rest of the processors as the event duration is less than a second when inspecting in provenance? We have tuned the number of concurrent tasks, duration and schedules to get the performance we have so far. I'm not sure if there is anything anyone could recommend or suggest to try and make improvements? We need to achieve a rate around 5x of what it's currently processing with the same number of nodes. We are running out of ideas on how to accomplish this and may have to consider alternatives. Kind Regards, Nathan
