RE: Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

2024-08-22 Thread Austin Hackett
Hi Ömer Many thanks for your response. The bandwidth between the two sites is many orders of magnitude greater than the 1.5 MB throughput I’m able to achieve, so I’m confident that the issue isn’t a case of low bandwidth. I have used the -1 setting on the broker (as well as setting explicitly

Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

2024-08-22 Thread Ömer Şiar Baysal
Hi Austin, I think it is worth to mention also the network bandwidth without even Kafka involved. You can create a test bench with basic Linux toolkit like ncat. Tuning socket buffers hardly make sense if the bandwidth is already low between the producer and remote site. When OS is configured t

Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

2024-08-22 Thread Austin Hackett
Hi List I am running Kafka 3.7.1 on CentOS 7.9. I have written a Kafka producer program in Python using confluent_kafka (which uses librdkafka). When the program is running on a machine in the same data centre as the Kafka cluster, a single producer writing 1,000 byte messages to a single topi