I understand that we could have used batch size as 0 here, if I am interpreting the documentation correctly.We essentially want no batching, and setting any value less than 256 in this particular case would achieve that, since the message size is 256 bytes for any message produced.
Sincerely, Anindya Haldar Oracle Responsys > On Oct 28, 2019, at 12:30 PM, Anindya Haldar <anindya.hal...@oracle.com> > wrote: > > We are consciously choosing the batch size in the experiment to be 1 because > that is what our producer use case will need to use. > > Regarding the request timeout that you have mentioned, how does one set that? > The command line is already passing two timeout values: ‘max.block.ms’, and > ‘transaction-duration-ms’. Both are set to 1 hour, and the failure we see > happens within a few minutes. Is there a third timeout value that is expiring > and causing this issue? > > Sincerely, > Anindya Haldar > Oracle Responsys > > >> On Oct 28, 2019, at 11:52 AM, M. Manna <manme...@gmail.com> wrote: >> >> Hi, >> >> Per test is based on a set of tuning parameters e.g. batch.size, axes, >> partitions, network latency etc. Your transactions are failing because your >> batch has expired, (or at least, that’s what it shows on the log). You have >> to tune your request timeout and batch.size correctly to improve on these. >> I suggest you try and get this right first without a non-txn producer >> setup. Then attempt with txn. >> >> Perhaps you want to recheck docs and understand what goal you want to >> target (e.g. speed, consistency, balanced etc.). >> >> Regards, >> >> >> On Mon, 28 Oct 2019 at 17:58, Anindya Haldar <anindya.hal...@oracle.com> >> wrote: >> >>> Anyone with a pointer on this? Do transactions work reliably with Kafka >>> perf test tools? If yes, then is there a way to make it work in this >>> scenario? >>> >>> Sincerely, >>> Anindya Haldar >>> Oracle Responsys >>> >>> >>>> On Oct 25, 2019, at 2:51 PM, Anindya Haldar <anindya.hal...@oracle.com> >>> wrote: >>>> >>>> We are evaluating Kafka for some of our use cases. As part of that >>> effort I am trying to run an experiment with a cluster we have set up, and >>> using the producer perf test tool supplied with the binaries. >>>> >>>> Here’s the cluster info: >>>> >>>> Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation >>> for each node. >>>> 3 ZooKeeper nodes >>>> 5 Kafka nodes >>>> >>>> Here is the topic description: >>>> >>>> $ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 >>> --topic testtopic3 >>>> Topic:testtopic3 PartitionCount:5 ReplicationFactor:3 >>> Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms < >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fretention.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=d9QdJSgIZN7kFfuuRJDX3gi3J8Y7uTnC7UrLSFBdasI&e= >>>> =3600000,flush.messages=1,unclean.leader.election.enable=false >>>> Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2 >>> Isr: 1,4,2 >>>> Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3 >>> Isr: 4,2,3 >>>> Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5 >>> Isr: 2,3,5 >>>> Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1 >>> Isr: 3,5,1 >>>> Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4 >>> Isr: 5,1,4 >>>> >>>> >>>> >>>> And here is the producer test run command line and the result: >>>> >>>> $ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records >>> 1000000 --throughput -1 --record-size 256 --producer-props >>> bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092 >>> acks=all batch.size=1 max.block.ms < >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fmax.block.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=XP3cGgt3rD1lr8r0diTLpP4FP3BwZ8spcHNDNXc3i6I&e= >>>> =3600000 enable.idempotence=true max.in.flight.requests.per.connection=1 >>> retries=3 --transaction-duration-ms 3600000 >>>> 4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg >>> latency, 4892.0 ms max latency. >>>> 4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg >>> latency, 9873.0 ms max latency. >>>> 4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg >>> latency, 14849.0 ms max latency. >>>> 4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg >>> latency, 19784.0 ms max latency. >>>> 4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg >>> latency, 24763.0 ms max latency. >>>> 4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg >>> latency, 29728.0 ms max latency. >>>> 4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg >>> latency, 34703.0 ms max latency. >>>> 4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg >>> latency, 39672.0 ms max latency. >>>> 4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg >>> latency, 44640.0 ms max latency. >>>> 4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg >>> latency, 49624.0 ms max latency. >>>> 4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg >>> latency, 54574.0 ms max latency. >>>> 4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg >>> latency, 59554.0 ms max latency. >>>> 4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg >>> latency, 64524.0 ms max latency. >>>> 4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg >>> latency, 69494.0 ms max latency. >>>> 4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg >>> latency, 74481.0 ms max latency. >>>> 4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg >>> latency, 79457.0 ms max latency. >>>> 4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg >>> latency, 84449.0 ms max latency. >>>> 4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg >>> latency, 89434.0 ms max latency. >>>> 4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg >>> latency, 94394.0 ms max latency. >>>> org.apache.kafka.common.errors.ProducerFencedException: Producer >>> attempted an operation with an old epoch. Either there is a newer producer >>> with the same transactionalId, or the producer's transaction has been >>> expired by the broker. >>>> Exception in thread "main" org.apache.kafka.common.KafkaException: >>> Cannot perform send because at least one previous transactional or >>> idempotent request has failed with errors. >>>> at >>> org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357) >>>> at >>> org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341) >>>> at >>> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915) >>>> at >>> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856) >>>> at >>> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143) >>>> Caused by: org.apache.kafka.common.errors.ProducerFencedException: >>> Producer attempted an operation with an old epoch. Either there is a newer >>> producer with the same transactionalId, or the producer's transaction has >>> been expired by the broker. >>>> [2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1, >>> transactionalId=performance-producer-default-transactional-id] Aborting >>> producer batches due to fatal error >>> (org.apache.kafka.clients.producer.internals.Sender) >>>> org.apache.kafka.common.errors.ProducerFencedException: Producer >>> attempted an operation with an old epoch. Either there is a newer producer >>> with the same transactionalId, or the producer's transaction has been >>> expired by the broker. >>>> >>>> It does not make any difference whether I add a transaction id flag like >>> ‘—transactional-id mytxn’ to the command line. >>>> >>>> Is there something I am missing here? >>>> >>>> Sincerely, >>>> Anindya Haldar >>>> Oracle Responsys >>>> >>> >>> >
