The RC candidate build is here: http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/ <http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/>
Eno > On 17 Apr 2017, at 17:20, Mahendra Kariya <mahendra.kar...@go-jek.com> wrote: > > Thanks! > > In the meantime, is the jar published somewhere on github or as a part of > build pipeline? > > On Mon, Apr 17, 2017 at 9:18 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this week. >> >> Eno >>> On 17 Apr 2017, at 13:25, Mahendra Kariya <mahendra.kar...@go-jek.com> >> wrote: >>> >>> Are the bug fix releases published to Maven central repo? >>> >>> On Sat, Apr 1, 2017 at 12:26 PM, Eno Thereska <eno.there...@gmail.com> >>> wrote: >>> >>>> Hi Sachin, >>>> >>>> In the bug fix release for 0.10.2 (and in trunk) we have now set >>>> max.poll.interval to infinite since from our experience with streams >> this >>>> should not be something that users set: https://github.com/apache/ >>>> kafka/pull/2770/files <https://github.com/apache/kafka/pull/2770/files >>> . >>>> >>>> We're in the process of documenting that change. For now you can >> increase >>>> the request timeout without worrying about max.poll.interval anymore. In >>>> fact I'd suggest you also increase max.poll.interval as we've done it >> above. >>>> >>>> Thanks >>>> Eno >>>> >>>>> On 1 Apr 2017, at 03:28, Sachin Mittal <sjmit...@gmail.com> wrote: >>>>> >>>>> Should this timeout be less than max poll interval value? if yes than >>>>> generally speaking what should be the ratio between two or range for >> this >>>>> timeout value . >>>>> >>>>> Thanks >>>>> Sachin >>>>> >>>>> On 1 Apr 2017 04:57, "Matthias J. Sax" <matth...@confluent.io> wrote: >>>>> >>>>> Yes, you can increase ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> >>>>> On 3/31/17 11:32 AM, Sachin Mittal wrote: >>>>>> Hi, >>>>>> So I have added the config ProducerConfig.RETRIES_CONFIG, >>>>> Integer.MAX_VALUE >>>>>> and the NotLeaderForPartitionException is gone. >>>>>> >>>>>> However we see a new exception especially under heavy load: >>>>>> org.apache.kafka.streams.errors.StreamsException: task [0_1] >> exception >>>>>> caught when producing >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals.RecordCollectorImpl. >>>>> checkForException(RecordCollectorImpl.java:119) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals. >> RecordCollectorImpl.flush( >>>>> RecordCollectorImpl.java:127) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at >>>>>> org.apache.kafka.streams.processor.internals. >>>> StreamTask$1.run(StreamTask. >>>>> java:76) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl. >>>>> measureLatencyNs(StreamsMetricsImpl.java:188) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals. >>>> StreamTask.commit(StreamTask. >>>>> java:280) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at >>>>>> org.apache.kafka.streams.processor.internals.StreamThread.commitOne( >>>>> StreamThread.java:787) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals.StreamThread.commitAll( >>>>> StreamThread.java:774) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at >>>>>> org.apache.kafka.streams.processor.internals. >> StreamThread.maybeCommit( >>>>> StreamThread.java:749) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> at >>>>>> org.apache.kafka.streams.processor.internals.StreamThread.runLoop( >>>>> StreamThread.java:671) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at >>>>>> org.apache.kafka.streams.processor.internals. >>>>> StreamThread.run(StreamThread.java:378) >>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) >>>> for >>>>>> new-part-advice-key-table-changelog-1: 30001 ms has passed since last >>>>> append >>>>>> >>>>>> So any idea as why TimeoutException is happening. >>>>>> Is this controlled by >>>>>> ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG >>>>>> >>>>>> If yes >>>>>> What should the value be set in this given that out consumer >>>>>> max.poll.interval.ms is defaul 5 minutes. >>>>>> >>>>>> Is there any other setting that we should try to avoid such errors >> which >>>>>> causes stream thread to die. >>>>>> >>>>>> Thanks >>>>>> Sachin >>>>>> >>>>>> >>>>>> On Sun, Mar 26, 2017 at 1:39 AM, Eno Thereska <eno.there...@gmail.com >>> >>>>>> wrote: >>>>>> >>>>>>> Hi Sachin, >>>>>>> >>>>>>> Not in this case. >>>>>>> >>>>>>> Thanks >>>>>>> Eno >>>>>>> >>>>>>>> On Mar 25, 2017, at 6:19 PM, Sachin Mittal <sjmit...@gmail.com> >>>> wrote: >>>>>>>> >>>>>>>> OK. >>>>>>>> I will try this out. >>>>>>>> >>>>>>>> Do I need to change anything for >>>>>>>> max.in.flight.requests.per.connection >>>>>>>> >>>>>>>> Thanks >>>>>>>> Sachin >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Mar 25, 2017 at 10:59 PM, Eno Thereska < >>>> eno.there...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Sachin, >>>>>>>>> >>>>>>>>> For this particular error, “org.apache.kafka.common.errors. >>>>>>>>> NotLeaderForPartitionException: This server is not the leader for >>>> that >>>>>>>>> topic-partition.”, could you try setting the number of retries to >>>>>>> something >>>>>>>>> large like this: >>>>>>>>> >>>>>>>>> Properties props = new Properties(); >>>>>>>>> props.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId); >>>>>>>>> ... >>>>>>>>> props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); >>>>>>>>> >>>>>>>>> This will retry the produce requests and should hopefully solve >> your >>>>>>>>> immediate problem. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Eno >>>>>>>>> >>>>>>>>> >>>>>>>>> On 25/03/2017, 08:35, "Sachin Mittal" <sjmit...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> We have encountered another case of series of errors which I would >>>>>>> need >>>>>>>>> more help in understanding. >>>>>>>>> >>>>>>>>> In logs we see message like this: >>>>>>>>> ERROR 2017-03-25 03:41:40,001 [kafka-producer-network-thread | >>>>>>>>> 85-StreamThread-3-producer]: >>>>>>>>> org.apache.kafka.streams.processor.internals.RecordCollectorImpl >> - >>>>>>>>> task >>>>>>>>> [0_1] Error sending record to topic new-part-advice-key-table- >>>>>>> changelog. >>>>>>>>> No >>>>>>>>> more offsets will be recorded for this task and the exception will >>>>>>>>> eventually be thrown >>>>>>>>> >>>>>>>>> then some millisecond later >>>>>>>>> ERROR 2017-03-25 03:41:40,149 [StreamThread-3]: >>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread - >>>>>>>>> stream-thread >>>>>>>>> [StreamThread-3] Failed while executing StreamTask 0_1 due to >> flush >>>>>>>>> state: >>>>>>>>> org.apache.kafka.streams.errors.StreamsException: task [0_1] >>>>>>> exception >>>>>>>>> caught when producing >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals.RecordCollectorImpl. >>>>>>>>> checkForException(RecordCollectorImpl.java:119) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>>>> RecordCollectorImpl.flush(RecordCollectorImpl.java:127) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>> StreamTask.flushState( >>>>>>>>> StreamTask.java:422) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >> StreamThread$4.apply( >>>>>>>>> StreamThread.java:555) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread. >>>>>>>>> performOnAllTasks(StreamThread.java:513) >>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>>>> StreamThread.flushAllState(StreamThread.java:551) >>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread. >>>>>>>>> shutdownTasksAndState(StreamThread.java:463) >>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>> StreamThread.shutdown( >>>>>>>>> StreamThread.java:408) >>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>>>> StreamThread.run(StreamThread.java:389) >>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> org.apache.kafka.common.errors.NotLeaderForPartitionException: >>>> This >>>>>>>>> server >>>>>>>>> is not the leader for that topic-partition. >>>>>>>>> >>>>>>>>> finally we get this >>>>>>>>> ERROR 2017-03-25 03:41:45,724 [StreamThread-3]: >>>>>>>>> com.advice.TestKafkaAdvice >>>>>>>>> - Uncaught exception: >>>>>>>>> org.apache.kafka.streams.errors.StreamsException: Exception >> caught >>>>>>> in >>>>>>>>> process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, >>>>>>>>> topic=advice-stream, partition=1, offset=48062286 >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>>>> StreamTask.process(StreamTask.java:216) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >> StreamThread.runLoop( >>>>>>>>> StreamThread.java:651) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>>>> StreamThread.run(StreamThread.java:378) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> Caused by: org.apache.kafka.streams.errors.StreamsException: task >>>>>>>>> [0_1] >>>>>>>>> exception caught when producing >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals.RecordCollectorImpl. >>>>>>>>> checkForException(RecordCollectorImpl.java:119) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>> RecordCollectorImpl.send( >>>>>>>>> RecordCollectorImpl.java:76) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> at >>>>>>>>> org.apache.kafka.streams.processor.internals. >>>>>>> RecordCollectorImpl.send( >>>>>>>>> RecordCollectorImpl.java:64) >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] >>>>>>>>> >>>>>>>>> >>>>>>>>> Again it is not clear why in this case we need to shut down the >>>>>>> steams >>>>>>>>> thread and eventually the application. Shouldn't we capture this >>>>>>> error >>>>>>>>> too? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Sachin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> >> >>
