Hard to say. Are you sure you query the correct instance? As you have 16 partitions, you need to ensure that the instance you query actually hosts the key. Or do you have only one instance running?
How do you know that there should be a value for the key? Note that
everything is async and continuously changing and thus it might hard to
reason about the system state?
-Matthias
On 10/8/19 7:53 AM, Stefan Hoffmeister wrote:
> Hello everybody,
>
> we seem to experience very unexpected behaviour from the Kafka Streams 2.2.0
> framework, where KeyValueState.get() returns a value of null, although it
> really should return some other value.
>
> This seems to be content corruption somewhere at / around the cache layer in
> Kafka Streams, with RocksDB delivering the persistent cache.
>
> On the implementation side, we simply
>
>
> streamsBuilder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("my-topic"),
>
> then retrieve and cache the store in init() of a Transformer, then use the
> store in transform().
>
> We see that *very rarely* KeyValueStore.get() returns null, although the
> (changelog) topic (on the Kafka broker) has clear evidence that
> KeyValueStore.get() should have returned a value.
>
> This was seen at exactly the same time for at least two different topics,
> upon starting the streaming application,
>
> a) (changelog) topic with exactly one partition, guaranteed to contain
> exactly one key - had reliably returned data for four weeks
>
> b) (changelog) topic over 16 partitions, where for about 30 calls to get(), a
> total of 20 returned data as expected, and about 10 returned an incorrect
> value of null.
>
> Has anyone seen something like that before, any pointers to root cause?
>
> The following items have made it onto our review agenda:
>
> a) virtual network instability, possibly leading to interesting RocksDB
> startup behaviour
> (org.apache.kafka.streams.processor.internals.StoreChangelogReader#initialize)
>
> b) on-disk data corruption (Linux -> NFS as persistent cache storage for
> RocksDB)
>
>
> Any insight or pointers greatly appreciated!
>
> Many thanks!
>
> Stefan
>
signature.asc
Description: OpenPGP digital signature
