If you provide the logs from your streams application then we might have some chance of working out what is going on. Without logs then we really don't have much hope of diagnosing the problem.
On Mon, 12 Dec 2016 at 12:18 Jon Yeargers <jon.yearg...@cedexis.com> wrote: > Im running as many threads as I have partitions on this topic. Just curious > if it would make any difference to the seemingly endless rebalancing woes. > > So far no change. In fact, I'll often see all 10 partitions (plus the 2 x > 10 for the two aggregations) assigned to a single thread. > > On Mon, Dec 12, 2016 at 4:15 AM, Jon Yeargers <jon.yearg...@cedexis.com> > wrote: > > > At this moment I have 5 instances each running 2 threads. > > Single instance / machine. > > > > Define 'full logs' ? > > > > On Mon, Dec 12, 2016 at 3:54 AM, Damian Guy <damian....@gmail.com> > wrote: > > > >> Jon, > >> > >> How many StreamThreads do you have running? > >> How many application instances? > >> Do you have more than one instance per machine? If yes, are they sharing > >> the same State Directory? > >> Do you have full logs that can be provided so we can try and see > how/what > >> is happening? > >> > >> Thanks, > >> Damian > >> > >> On Mon, 12 Dec 2016 at 10:17 Jon Yeargers <jon.yearg...@cedexis.com> > >> wrote: > >> > >> > No luck here. Moved all state storage to a non-tmp folder and > restarted. > >> > Still hitting the 'No locks available' error quite frequently. > >> > > >> > On Sun, Dec 11, 2016 at 3:45 PM, Jon Yeargers < > jon.yearg...@cedexis.com > >> > > >> > wrote: > >> > > >> > > I moved the state folder to a separate drive and linked out to it. > >> > > > >> > > I'll try your suggestion and point directly. > >> > > > >> > > On Sun, Dec 11, 2016 at 2:20 PM, Matthias J. Sax < > >> matth...@confluent.io> > >> > > wrote: > >> > > > >> > >> I am not sure, but this might be related with your state directory. > >> > >> > >> > >> You use default directory that is located in /tmp -- could it be, > >> that > >> > >> /tmp gets clean up and thus you loose files/directories? > >> > >> > >> > >> Try to reconfigure your state directory via StreamsConfig: > >> > >> http://docs.confluent.io/current/streams/developer-guide. > >> > >> html#optional-configuration-parameters > >> > >> > >> > >> > >> > >> -Matthias > >> > >> > >> > >> On 12/11/16 1:28 AM, Jon Yeargers wrote: > >> > >> > Seeing this appearing somewhat frequently - > >> > >> > > >> > >> > org.apache.kafka.streams.errors.ProcessorStateException: Error > >> opening > >> > >> > store minute_agg_stream-201612100812 at location > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag > >> > >> g_stream-201612100812 > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > >> > >> (RocksDBStore.java:196) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > >> > >> (RocksDBStore.java:158) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$ > >> > >> Segment.openDB(RocksDBWindowStore.java:72) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > >> > >> getOrCreateSegment(RocksDBWindowStore.java:402) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > >> > >> putInternal(RocksDBWindowStore.java:333) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > >> > >> access$100(RocksDBWindowStore.java:51) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$ > >> > >> 2.restore(RocksDBWindowStore.java:212) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM > >> > >> anager.restoreActiveState(ProcessorStateManager.java:235) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM > >> > >> anager.register(ProcessorStateManager.java:198) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.ProcessorContex > >> > >> tImpl.register(ProcessorContextImpl.java:123) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > >> > >> init(RocksDBWindowStore.java:206) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.MeteredWindowStore. > >> > >> init(MeteredWindowStore.java:66) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.CachingWindowStore. > >> > >> init(CachingWindowStore.java:64) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.AbstractTask.in > >> > >> itializeStateStores(AbstractTask.java:81) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamTask.< > >> > >> init>(StreamTask.java:120) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread. > >> > >> createStreamTask(StreamThread.java:633) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread. > >> > >> addStreamTasks(StreamThread.java:660) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread. > >> > >> access$100(StreamThread.java:69) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread$1. > >> > >> onPartitionsAssigned(StreamThread.java:124) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina > >> > >> tor.onJoinComplete(ConsumerCoordinator.java:228) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina > >> > >> tor.joinGroupIfNeeded(AbstractCoordinator.java:313) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina > >> > >> tor.ensureActiveGroup(AbstractCoordinator.java:277) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina > >> > >> tor.poll(ConsumerCoordinator.java:259) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce( > >> > >> KafkaConsumer.java:1013) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaCo > >> > >> nsumer.java:979) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread. > >> > >> runLoop(StreamThread.java:407) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.processor.internals.StreamThread. > >> > >> run(StreamThread.java:242) > >> > >> > > >> > >> > Caused by: org.rocksdb.RocksDBException: IO error: lock > >> > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag > >> > >> g_stream-201612100812/LOCK: > >> > >> > No locks available > >> > >> > > >> > >> > at org.rocksdb.RocksDB.open(Native Method) > >> > >> > > >> > >> > at org.rocksdb.RocksDB.open(RocksDB.java:184) > >> > >> > > >> > >> > at > >> > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > >> > >> (RocksDBStore.java:189) > >> > >> > > >> > >> > ... 26 common frames omitted > >> > >> > > >> > >> > >> > >> > >> > > > >> > > >> > > > > >
