Re: Nifi - Content-repo on AWS-EBS volumes

2023-12-15 Thread Phillip Lord
I just switched a cluster using 3 EBS volumes for cont-repo from gp2 to gp3… resolved definite I/O throughput issues.  The change to gp3 was significant enough that I might actually reduce from 3 to 2 volumes, perhaps even a single volume would be sufficient. Of course every use case is unique.

Re: Nifi - Content-repo on AWS-EBS volumes

2023-12-15 Thread Gregory M. Foreman
Mark: Got it. Thank you for the help. Greg > On Dec 15, 2023, at 4:14 PM, Mark Payne wrote: > > Greg, > > Whether or not multiple content repos will have any impact depends very much > on where your system’s bottleneck is. If your bottleneck is disk I/O, it will > absolutely help. If your

Re: Nifi - Content-repo on AWS-EBS volumes

2023-12-15 Thread Mark Payne
Greg, Whether or not multiple content repos will have any impact depends very much on where your system’s bottleneck is. If your bottleneck is disk I/O, it will absolutely help. If your bottleneck is CPU, it won’t. If, for example, you’re running on bare metal and have 48 cores on your machine

Re: Nifi - Content-repo on AWS-EBS volumes

2023-12-15 Thread Gregory M. Foreman
Mark: I was just discussing multiple content repos on EBS volumes with a colleague. I found your post from a long time ago: https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv “Re #2: I don't know that i've used any SAN to back my repositories other than the EBS provided by Amazo

Re: Nifi - Content-repo on AWS-EBS volumes

2023-12-11 Thread Mark Payne
Hey Phil, NiFi will not spread the content of a single file over multiple partitions. It will write the content of FlowFile 1 to content repo 1, then write the next FlowFile to repo 2, etc. so it does round-robin but does not spread a single FlowFile across multiple repos. Thanks -Mark Sent f

Nifi - Content-repo on AWS-EBS volumes

2023-12-11 Thread Phillip Lord
Hello Nifi comrades, Here's my scenario... Let's say I have a Nifi cluster running on EC2 instances with attached EBS volumes serving as their repos. They've split up their content-repos into three content-repos per node(cont1, cont2, cont3). Each being a dedicated EBS volume. My understanding