Re: Transaction error in Kafka producer perf test

2019-10-28 Thread Anindya Haldar
I understand that we could have used batch size as 0 here, if I am interpreting the documentation correctly.We essentially want no batching, and setting any value less than 256 in this particular case would achieve that, since the message size is 256 bytes for any message produced. Sincerely,

Re: Transaction error in Kafka producer perf test

2019-10-28 Thread Anindya Haldar
We are consciously choosing the batch size in the experiment to be 1 because that is what our producer use case will need to use. Regarding the request timeout that you have mentioned, how does one set that? The command line is already passing two timeout values: ‘max.block.ms’, and

Re: Transaction error in Kafka producer perf test

2019-10-28 Thread M. Manna
Hi, Per test is based on a set of tuning parameters e.g. batch.size, axes, partitions, network latency etc. Your transactions are failing because your batch has expired, (or at least, that’s what it shows on the log). You have to tune your request timeout and batch.size correctly to improve on

Re: Transaction error in Kafka producer perf test

2019-10-28 Thread Anindya Haldar
Anyone with a pointer on this? Do transactions work reliably with Kafka perf test tools? If yes, then is there a way to make it work in this scenario? Sincerely, Anindya Haldar Oracle Responsys > On Oct 25, 2019, at 2:51 PM, Anindya Haldar wrote: > > We are evaluating Kafka for some of our

Transaction error in Kafka producer perf test

2019-10-25 Thread Anindya Haldar
We are evaluating Kafka for some of our use cases. As part of that effort I am trying to run an experiment with a cluster we have set up, and using the producer perf test tool supplied with the binaries. Here’s the cluster info: Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space