Thank you very much, Marcos. My application is real-time processing so I would say most of the times I am dealing with the "latest" message that emphasize page caching. In this case, does it mean there is no additional throughput provided by using 10k or 15k disks? What about having virtualized cluster for such a use case? Obviously, virtualization does not affect page caching significantly, and it seems I am dealing with the Memory rather than Disk most of the times. The problem I've got here is it would be more consistent to have a single hardware spec for everything and apparently 40 cores and 512GB memory would be a lot for hosting Kafka or Storm separately. So I have to choose between using bare metal and colocate Storm, Kafka, or use the virtualized at the first place. What do you think, which one do you think would be better for a real-time application with a hard SLA?
Regards, Ali On Thu, Apr 13, 2017 at 10:57 AM, Marcos Juarez <[email protected]> wrote: > Ali, > > I don't know of proper benchmarks out there, but I've done some work in > this area, when trying to determine what hardware to get for particular use > cases. My answers are in-line: > > On Mon, Apr 10, 2017 at 7:05 PM, Ali Nazemian <[email protected]> > wrote: > > > Hi all, > > > > I was wondering if there is any benchmark or any recommendation for > having > > physical HW vs virtual for the Kafka Brokers. I am trying to calculate > the > > HW requirements for a Kafka Cluster with a hard SLA. My questions are as > > follows. > > > > - What is the effect of OS disk caching for a Kafka-Broker? How much > > on-heap and off-heap memory would be required per node? > > > > If by "OS disk caching" you mean page cache, then it's huge. Kafka relies > on it to serve as much data as possible from memory directly. The actual > on-heap vs off-heap RAM requirements will be completely dependent on your > scenario. I've run Kafka brokers with heaps as little as 8GB of RAM, and > as much as 32GB of RAM each, using CMS GC with some customized tuning. The > boxes usually have much more physical RAM than that (64GB of RAM and 128GB > respectively), for page cache. > > I think the Confluent recommendation these days is to use G1 GC, but I've > got no experience using that. > > > > > > - Since Kafka read-write workload is pretty sequential which of the > > following spinning disks would be recommended? SATA 7.2k, SAS 10k, SAS > 15k? > > > > Again, that will depend on your use case. Are your consumers mostly > consuming up-to-the-second messages, or are they always connecting and > consuming from arbitrary offsets, or even from the beginning of topics? If > your consumers are always consuming latest, then actual disk IO will be > extremely low, since almost everything will be served out of page > cache/memory, and most of the activity will be fsync's every ~5 seconds or > so. On the other hand, if most of your consumption will be of old data, > then the brokers will need to read a large amount of data from disk > sequentially whenever consumers request data. That's when you'd benefit > from something like SAS 10k or SAS 15k. You'd need to run some custom > benchmarks to figure out what would work on workload that mimics your use > case. > > > > > > - Since Kafka is not CPU-intensive, how bad would be to coexist > > Kafka-Broker and a CPU-intensive workload like STORM? > > > > I wouldn't recommend that at all. I've always run Kafka on bare metal with > nothing else running on that box. Otherwise, you won't be able to identify > bottlenecks when you run into them, and both systems would be impacting > each other all the time, potentially. Hardware is cheap, not worth it to > spend time chasing issues caused by "noisy neighbors" on the boxes. > > > > > > Regards, > > Ali > > > > Marcos > -- A.Nazemian
