Cool, thanks, shall we open a JIRA?
Eno
> On 3 May 2017, at 16:16, João Peixoto <joao.harti...@gmail.com> wrote:
>
> Actually I need to apologize, I pasted the wrong issue, I meant to paste
> https://github.com/facebook/rocksdb/issues/261.
>
> RocksDB did not produce a crash report since it didn't actually crash. I
> performed thread dumps on stale and not-stale instances which revealed the
> common behavior and I collect and plot several Kafka metrics, including
> "punctuate" durations, therefore I know it took a long time and eventually
> finished.
>
> Joao
>
> On Wed, May 3, 2017 at 6:22 AM Eno Thereska <eno.there...@gmail.com> wrote:
>
>> Hi there,
>>
>> Thanks for double checking. Does RocksDB actually crash or produce a crash
>> dump? I’m curious how you know that the issue is
>> https://github.com/facebook/rocksdb/issues/1121 <
>> https://github.com/facebook/rocksdb/issues/1121>, so just double checking
>> with you.
>>
>> If that’s indeed the case, do you mind opening a JIRA (a copy-paste of the
>> below should suffice)? Alternatively let us know and we’ll open it. Sounds
>> like we should handle this better.
>>
>> Thanks,
>> Eno
>>
>>
>>> On May 3, 2017, at 5:49 AM, João Peixoto <joao.harti...@gmail.com>
>> wrote:
>>>
>>> I believe I found the root cause of my problem. I seem to have hit this
>>> RocksDB bug https://github.com/facebook/rocksdb/issues/1121
>>>
>>> On my stream configuration I have a custom transformer used for
>>> deduplicating records, highly inspired in the
>>> EventDeduplicationLambdaIntegrationTest
>>> <
>> https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java#L161
>>>
>>> but
>>> adjusted to my use case, special emphasis on the "punctuate" method.
>>>
>>> All the stale instances had the main stream thread "RUNNING" the
>>> "punctuate" method of this transformer, which in term was running RocksDB
>>> "seekToFirst".
>>>
>>> Also during my debugging one such instance finished the "punctuate"
>> method,
>>> which took ~11h, exactly the time the instance was stuck for.
>>> Changing the backing state store from "persistent" to "inMemory" solved
>> my
>>> issue, at least after several days running, no stuck instances.
>>>
>>> This leads me to ask, shouldn't Kafka detect such a situation fairly
>>> quickly? Instead of just stopping polling? My guess is that the heartbeat
>>> thread which now is separate continues working fine, since by definition
>>> the stream runs a message through the whole pipeline this step probably
>>> just looked like it was VERY slow. Not sure what the best approach here
>>> would be.
>>>
>>> PS The linked code clearly states "This code is for demonstration
>> purposes
>>> and was not tested for production usage" so that's on me
>>>
>>> On Tue, May 2, 2017 at 11:20 AM Matthias J. Sax <matth...@confluent.io>
>>> wrote:
>>>
>>>> Did you check the logs? Maybe you need to increase log level to DEBUG to
>>>> get some more information.
>>>>
>>>> Did you double check committed offsets via bin/kafka-consumer-groups.sh?
>>>>
>>>> -Matthias
>>>>
>>>> On 4/28/17 9:22 AM, João Peixoto wrote:
>>>>> My stream gets stale after a while and it simply does not receive any
>> new
>>>>> messages, aka does not poll.
>>>>>
>>>>> I'm using Kafka Streams 0.10.2.1 (same happens with 0.10.2.0) and the
>>>>> brokers are running 0.10.1.1.
>>>>>
>>>>> The stream state is RUNNING and there are no exceptions in the logs.
>>>>>
>>>>> Looking at the JMX metrics, the threads are there and running, just not
>>>>> doing anything.
>>>>> The metric "consumer-coordinator-metrics > heartbeat-response-time-max"
>>>>> (The max time taken to receive a response to a heartbeat request) reads
>>>>> 43,361 seconds (almost 12 hours) which is consistent with the time of
>> the
>>>>> hang. Shouldn't this trigger a failure somehow?
>>>>>
>>>>> The stream configuration looks something like this:
>>>>>
>>>>> Properties props = new Properties();
>>>>> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
>>>>> CustomTimestampExtractor.class.getName());
>>>>> props.put(StreamsConfig.APPLICATION_ID_CONFIG, streamName);
>>>>> props.put(StreamsConfig.CLIENT_ID_CONFIG, streamName);
>>>>> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
>>>>> myConfig.getBrokerList());
>>>>> props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG,
>>>>> Serdes.String().getClass().getName());
>>>>> props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG,
>>>>> Serdes.ByteArray().getClass().getName());
>>>>> props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG,
>>>>> myConfig.getCommitIntervalMs()); // 5000
>>>>> props.put(StreamsConfig.METRICS_RECORDING_LEVEL_CONFIG, "DEBUG");
>>>>> props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG,
>>>>> myConfig.getStreamThreadsCount()); // 1
>>>>> props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,
>>>>> myConfig.getMaxCacheBytes()); // 524_288_000L
>>>>> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
>>>>> props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50);
>>>>>
>>>>> The stream LEFT JOINs 2 topics, one of them being a KTable, and outputs
>>>> to
>>>>> another topic.
>>>>>
>>>>> Thanks in advance for the help!
>>>>>
>>>>
>>>>
>>
>>
