Thanks Lee. Your response was awesome and really made me want to get hands on a set of boxes like this so we could do some testing.
Thanks Joe On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim <[email protected]> wrote: > Joe, > Good points regarding throughput on real flows and sustained basis. My test > was only pushing one aspect of the system. > > That said, I would be interested discussing/developing a more comprehensive > test flow to capture more real world use cases. I'll check to see if that > conversation has started. > > Thanks, > Lee > > > > > > Lee Laim > 610-864-1657 > > On Oct 17, 2016, at 9:55 PM, Ali Nazemian <[email protected]> wrote: > > Dear Joe, > Thank you very much. > > Best regards > > > On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt <[email protected]> wrote: >> >> Ali >> >> I suspect bottlenecks in the software itself and the flow design will >> become a factor before you 800 MB/s. You'd likely hit CPU efficiency >> issues before this caused by the flow processors themselves and due to >> garbage collection. Probably the most important factor though will be >> the transaction rate and whether the flow is configured to tradeoff >> some latency for higher throughput. So many variables at play but >> under idealized conditions and a system like you describe it is >> theoretically feasible to hit that value. >> >> Practically speaking I think you'd be looking at a couple hundred MB/s >> per server like this on real flows on a sustained basis. >> >> Thanks >> Joe >> >> On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian <[email protected]> >> wrote: >> > Dear Nifi users/developers, >> > Hi, >> > >> > I was wondering how can I calculate the theoretical throughput of a Nifi >> > server? let's suppose we can eliminate different bottlenecks such as the >> > file flow rep and provenance repo bottleneck by using a very high-end >> > SSD. >> > Moreover, assume that a very high-end network infrastructure is >> > available. >> > In this case, is it possible to reach 800MB throughput per second per >> > each >> > server? Suppose each server comes with 24 disk slots. 16 disk slots are >> > used >> > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the >> > content repo. Let's say each content repo can achieve 100 MB throughput. >> > May >> > I say the total throughput per each server can be 8x100=800MBps? Is it >> > possible to reach this amount of throughput practically? >> > Thank you very much. >> > >> > Best regards, >> > Ali > > > > > -- > A.Nazemian
