[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139493#comment-16139493 ] ASF GitHub Bot commented on KAFKA-5733: --- Github user bbejeck closed the pull request at: https://github.com/apache/kafka/pull/3664 > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska >Assignee: Bill Bejeck > Fix For: 1.0.0 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131353#comment-16131353 ] Guozhang Wang commented on KAFKA-5733: -- 3681 contains Bill's PR, so KAFKA-5725 should be unblocked now. > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska >Assignee: Bill Bejeck > Fix For: 1.0.0 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131305#comment-16131305 ] Eno Thereska commented on KAFKA-5733: - [~guozhang] thanks, do we still need [~bbejeck]'s PR? > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska >Assignee: Bill Bejeck > Fix For: 1.0.0 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131298#comment-16131298 ] ASF GitHub Bot commented on KAFKA-5733: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3681 > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska >Assignee: Bill Bejeck > Fix For: 1.0.0 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130778#comment-16130778 ] Guozhang Wang commented on KAFKA-5733: -- Yup, moved to 1.0.0 > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska >Assignee: Bill Bejeck > Fix For: 1.0.0 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130122#comment-16130122 ] Damian Guy commented on KAFKA-5733: --- [~enothereska] [~guozhang] this is only on trunk - right? i.e., it is due to the bulkload that is not on the 0.11 branch? If true can we move it it 1.0.0? > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska > Fix For: 0.11.0.1 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129642#comment-16129642 ] ASF GitHub Bot commented on KAFKA-5733: --- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3681 KAFKA-5733: RocksDB bulk load with lower number of levels. This is to complete Bill's PR on KAFKA-5733, incorporating the suggestion in https://github.com/facebook/rocksdb/issues/2734. Some minor changes: move `open = true` in `openDB`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K5733-rocksdb-bulk-load Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3681.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3681 commit 2cfc54c8e77b840acc0d176cea8eabda4d1c4b03 Author: Bill Bejeck Date: 2017-08-14T14:46:08Z KAFKA-5733: ensure clean RocksDB directory before setting prepareForBulkload settings commit f468147455c9f8eaf640f2821b7849b7b370968d Author: Bill Bejeck Date: 2017-08-14T16:57:29Z KAFKA-5733: address comments commit 149634b405cd5d73ec0e27c9a55b16f9971abf1f Author: Bill Bejeck Date: 2017-08-14T18:20:05Z KAFKA-5733: address additional comments, enforce not opening and closing if not a clean RocksDB dir on restore commit b4cd505b5c5cb9e57df4728414b5194a65d14b1c Author: Guozhang Wang Date: 2017-08-16T22:17:17Z Merge branch 'KAFKA-5733_rocks_db_throws_more_than_num_levels_on_restore' of https://github.com/bbejeck/kafka into K5733-rocksdb-bulk-load commit d0ada11eb8e97fbd5efdfb4c66160a81970575e4 Author: Guozhang Wang Date: 2017-08-16T22:45:16Z use compactRange to still enable bulk loading commit 08da1f16c8aeb587dd8b1ac7b5192f1ab3395cd7 Author: Guozhang Wang Date: 2017-08-16T23:53:02Z minor fixes > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska > Fix For: 0.11.0.1 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.intern
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125778#comment-16125778 ] ASF GitHub Bot commented on KAFKA-5733: --- GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/3664 KAFKA-5733: ensure clean RocksDB directory before setting prepareForB… …ulkload settings You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka KAFKA-5733_rocks_db_throws_more_than_num_levels_on_restore Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3664.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3664 commit 2cfc54c8e77b840acc0d176cea8eabda4d1c4b03 Author: Bill Bejeck Date: 2017-08-14T14:46:08Z KAFKA-5733: ensure clean RocksDB directory before setting prepareForBulkload settings > System tests get exception RocksDBException: db has more levels than > options.num_levels > --- > > Key: KAFKA-5733 > URL: https://issues.apache.org/jira/browse/KAFKA-5733 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.1 >Reporter: Eno Thereska > Fix For: 0.11.0.1 > > > New system tests as part of KAFKA-5725 with PR > https://github.com/apache/kafka/pull/3656 consistently fail like this: > [2017-08-14 10:37:57,216] ERROR User provided listener > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener > for group SmokeTest failed on partition assignment > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) > org.apache.kafka.streams.errors.ProcessorStateException: Error opening store > cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176) > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259) > at > org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74) > at > org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126) > at > org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94) > at > org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677) > Caused by: org.rocksdb.RocksDBException: db has more levels than > options.num_levels > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels
[
https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125708#comment-16125708
]
Bill Bejeck commented on KAFKA-5733:
I've been able to reproduce this in a unit-test.
Here is the issue. We have an existing RocksDB store (default num levels
setting of 7). When the broker goes down and reconnects a re-balance is
triggered and causes a state restoration. When we go to restore the state
store, we close RocksDB, we turn on {{options.prepareForBulkload}} which sets
the num_levels to 2 for optimized bulk loads, then re-opens the database.
Then we get the error because the pre-existing data is already there with
num_levels of 7, thus throwing the exception.
> System tests get exception RocksDBException: db has more levels than
> options.num_levels
> ---
>
> Key: KAFKA-5733
> URL: https://issues.apache.org/jira/browse/KAFKA-5733
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eno Thereska
> Fix For: 0.11.0.1
>
>
> New system tests as part of KAFKA-5725 with PR
> https://github.com/apache/kafka/pull/3656 consistently fail like this:
> [2017-08-14 10:37:57,216] ERROR User provided listener
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener
> for group SmokeTest failed on partition assignment
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> org.apache.kafka.streams.errors.ProcessorStateException: Error opening store
> cntStoreName at location /mnt/streams/SmokeTest/2_0/rocksdb/cntStoreName
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:206)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:176)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:259)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.access$200(RocksDBStore.java:74)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore$RocksDBBatchingRestoreCallback.onRestoreStart(RocksDBStore.java:551)
> at
> org.apache.kafka.streams.processor.internals.CompositeRestoreListener.onRestoreStart(CompositeRestoreListener.java:54)
> at
> org.apache.kafka.streams.processor.internals.StateRestorer.restoreStarted(StateRestorer.java:61)
> at
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:126)
> at
> org.apache.kafka.streams.processor.internals.TaskManager.createTasks(TaskManager.java:94)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:177)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:368)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:317)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:736)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:707)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:677)
> Caused by: org.rocksdb.RocksDBException: db has more levels than
> options.num_levels
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:231)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:199)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
