thank you. Senthil , Why batch.size=65536 & retries=100, its big number right ? do we need this much for streaming applications ?
Regards, Shyam On Wed, Jul 3, 2019 at 1:30 PM SenthilKumar K <senthilec...@gmail.com> wrote: > `*Partition = -1` - *This explains why are you getting timeout error. > > Why dont you use Default Partitioner ?: > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java > > Try only with below producer properties : > bootstrap.servers=<<>>> > acks =1 > retries=100 > batch.size=65536 > linger.ms=10 > key.serializer=<<>> > value.serializer=<<>>> > compression.type=<<snappy>> > buffer.memory=104857600 > > --Senthil > > > On Wed, Jul 3, 2019 at 1:18 PM Shyam P <shyamabigd...@gmail.com> wrote: > >> Hi SenthilKumar, >> thanks a lot . >> >> Yeah I set up local set up and print the log with partition info which >> shows as below >> >> 2019-07-03 02:48:28.822 INFO 7092 --- [lt-dispatcher-2] >> c.s.c.p.p.CompanyInfoPartitioner : Topic : inbound_topic Key = 597736248- >> Entropy Cayman Solar Ltd.-null-null-null *Partition = -1* 2019-07-03 >> 02:48:28.931 ERROR 7092 --- [ad | producer-1] >> o.s.k.support.LoggingProducerListener : Exception thrown when sending a >> message with key='597736248- Entropy Cayman Solar Ltd.-null-null-null' and >> payload='com.spgmi.ca.prescore.model.Company@8b12343' to topic >> inbound_topic : >> >> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for >> inbound_topic --1: 104 ms has passed since batch creation plus linger time >> >> My topics inbound_topic has two partitions as you see below >> C:\Software\kafka\kafka_2.11-2.1.1\bin\windows>kafka-topics.bat --describe >> --zookeeper localhost:2181 --topic inbound_topic Topic:inbound_topic >> PartitionCount:2 ReplicationFactor:1 Configs: Topic: inbound_topic >> Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: inbound_topic Partition: 1 >> Leader: 0 Replicas: 0 Isr: 0 >> >> *But my producer seems to trying to send to Partition = -1.* >> >> My partition logic is as below >> >> int p = (((String)key).hashCode() * Integer.MAX_VALUE) % numPartitions; >> logger.info("Topic : "+ topic + "\t Key = " + (String)key + " >> Partition = " + p ); >> >> On key i am doing hashCode(). What need to be corrected here to avoid >> this negative number partition number ? i.e. Partition = -1 >> >> What should be my partition key logic like ? >> >> any help highly appreciated. >> Regards, >> Shyam >> >> On Tue, Jul 2, 2019 at 8:48 PM SenthilKumar K <senthilec...@gmail.com> >> wrote: >> >>> Does it happen to all partitions or only few partitions ? Can you make >>> sure your local setup working fine ? Were you able to produce using >>> console-producer ? >>> >>> Example : >>> EVERE: Expiring 7 record(s) for topic-9{partition:9}: 30022 ms has >>> passed since last append >>> Expiring 9 record(s) for topic-2{partition:2}: 30015 ms has passed >>> since batch creation plus linger time >>> >>> --Senthil >>> >>> On Tue, Jul 2, 2019 at 5:34 PM Shyam P <shyamabigd...@gmail.com> wrote: >>> >>>> Thanks a lot Senthil for quick reply. >>>> I am using kafka_2.11-2.1.1 . >>>> In your case Kafka Producer Client in One DataCenter and Kafka Broker >>>> in other DataCenter but in my case I installed Kafka on the same machine >>>> where Producer is running. >>>> i.e. currently I am in development mode , so everything now on my local >>>> for timebeing ...i.e. Kafka broker , zk and my producer code in eclipse. >>>> >>>> If is is a set up issue at least it should run fine in my local right. >>>> I tried several producer configurations combinations as explained in >>>> the SOF link. >>>> >>>> So not sure now what is the issue and how to fix it ? >>>> >>>> Is in your case the issue fixed ? >>>> >>>> Regards, >>>> Shyam >>>> >>>> On Tue, Jul 2, 2019 at 5:12 PM SenthilKumar K <senthilec...@gmail.com> >>>> wrote: >>>> >>>>> Hi Shyam, We also faced `TimeoutException: Expiring 1 record(s)` issue >>>>> in >>>>> our Kafka Producer Client. As described here >>>>> < >>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has >>>>> > >>>>> , >>>>> first we tried increasing request timeout but that didn't help. We had >>>>> setup like Kafka Producer Client in One DataCenter and Kafka Broker in >>>>> other DataCenter & thats why the producer failed to push records to >>>>> brokers >>>>> on time due to network issue. In your case , Could be setup issue ? >>>>> >>>>> --Senthil >>>>> >>>>> On Tue, Jul 2, 2019 at 3:57 PM Shyam P <shyamabigd...@gmail.com> >>>>> wrote: >>>>> >>>>> > Hi, >>>>> > I am facing the below issue. >>>>> > >>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) >>>>> > for <topic-name> 229 ms has passed since batch creation plus linger >>>>> > time >>>>> > >>>>> > >>>>> > I tried many producer configuration settings. more details below : >>>>> > >>>>> > >>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has >>>>> > >>>>> > But nothing working. >>>>> > >>>>> > Can anyone plz help me , what is wrong here and how to fix it ? >>>>> > >>>>> > thanks, >>>>> > Shyam >>>>> > >>>>> >>>>
