Re: Running a performance benchmark load test of Flink on new CPUs

2021-11-08 Thread Vijay Balakrishnan
Austin -the flink benchmark is for testing Flink on single machines and not a cluster. I did see this https://oceanrep.geomar.de/50729/1/bsc_nico_biernat_thesis.pdf but it is more for testing the Scaling of Flink instead of testing throughput and latency. On Mon, Nov 8, 2021 at 10:54 AM Vijay

Re: Running a performance benchmark load test of Flink on new CPUs

2021-11-08 Thread Vijay Balakrishnan
Thx, Austin. I was hoping there might be a newer benchmark run similar to done by dataArtisans on Flink in 2016(old). https://www.ververica.com/blog/extending-the-yahoo-streaming-benchmark Looks like Yahoo Streaming benchmark was an initial standard in 2016. Hoping to see something updated for

Re: Running a performance benchmark load test of Flink on new CPUs

2021-11-05 Thread Austin Cawley-Edwards
Hi Vijay, I'm not too familiar with the subject, but maybe you could have a look at the flink-faker[1], which generates fake data. I would think you could use it to write to kafka in one Flink job, and then have another Flink job to ingest and run your benchmarks. There is also this

Running a performance benchmark load test of Flink on new CPUs

2021-11-05 Thread Vijay Balakrishnan
Hi, I am a newbie to running a performance benchmark load test of Flink on new CPUs. Is there an* existing workload generator* that I can use with Kafka and then ingest it with Flink KafkaConnector & test the performance against various new chips on servers ? Measuring CPU performance etc, vCPU