Hi Kafka crew,
I'm rolling out a Kafka Streams application on a pretty large dataset, and
I see some exceptions that worry me, which I haven't seen with smaller data
sets
(Using Kafka streams trunk of a few days ago)
2017-02-15 15:25:14,431 | INFO | StreamThread-42 | StreamTask
| 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | task
[9_4] Initializing state stores
2017-02-15 15:25:14,433 | ERROR | StreamThread-42 | ConsumerCoordinator
| 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | User
provided listener
org.apache.kafka.streams.processor.internals.StreamThread$1 for group
KNVB-test-generation-175-person failed on partition assignment
org.apache.kafka.streams.errors.ProcessorStateException: Error opening
store KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION
at location
/output/env.managed.repository-master/storage/KNVB-test-generation-175-person/9_4/rocksdb/KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:175)
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:145)
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:150)
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131)
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62)
at
org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:85)
at
org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:143)
at
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
at
org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
at
org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
at
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
at
org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
at
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
Caused by: org.rocksdb.RocksDBException:
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:168)
... 22 more
I've checked the folder it points to:
[root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]# ls -l
total 223348
-rw-r--r--. 1 431 433 68153335 Feb 15 11:09 000353.sst
-rw-r--r--. 1 431 433 68207125 Feb 15 11:09 000355.sst
-rw-r--r--. 1 431 433 7241027 Feb 15 11:09 000356.sst
-rw-r--r--. 1 431 433 59070938 Feb 15 11:12 000459.sst
-rw-r--r--. 1 431 433 18783280 Feb 15 11:13 000489.sst
-rw-r--r--. 1 431 433 0 Feb 15 11:38 000502.log
-rw-r--r--. 1 431 433 7183476 Feb 15 11:38 000505.sst
-rw-r--r--. 1 431 433 16 Feb 15 11:38 CURRENT
-rw-r--r--. 1 431 433 37 Feb 15 10:57 IDENTITY
-rw-r--r--. 1 431 433 0 Feb 15 10:57 LOCK
-rw-r--r--. 1 431 433 13446 Feb 15 11:38 LOG
-rw-r--r--. 1 431 433 13130 Feb 15 10:57 LOG.old.1487158687061100
-rw-r--r--. 1 431 433 456 Feb 15 11:38 MANIFEST-000501
-rw-r--r--. 1 431 433 4275 Feb 15 10:57 OPTIONS-000005
-rw-r--r--. 1 431 433 4275 Feb 15 11:38 OPTIONS-000504
[root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]#
What strikes me as odd is that none of the files have been touched in the
last few hours (the stacktrace is from 15:25), so there doesn't appear to
have been any logging of that error (on the rocksdb side)
Any ideas? Or should I post this to RocksDb?
regards, Frank
