Hello Frank, How many instances do you have in your apps and how many threads did you use per thread? Note that besides the topology complexity (i.e. number of state stores, number of internal topics etc) the (re-)initialization process is depending on the underlying consumer's membership protocol, and hence its rebalance latency could be longer with larger groups.
We have been optimizing our rebalance latency due to state store migration and restoration in the latest release, but for now the re-initialization latency is still largely depends on 1) topology complexity regarding to state stores and 2) number of input topic partitions and instance / threads in the application. Guozhang On Sat, Nov 26, 2016 at 12:57 AM, Damian Guy <damian....@gmail.com> wrote: > Hi Frank, > > If you are running on a single node then the RocksDB state should be > re-used by your app. However, it relies on the app being cleanly shutdown > and the existence of ".checkpoint" files in the state directory for the > store, .i.e, /tmp/kafka-streams/application-id/0_0/.checkpoint. If the > file > doesn't exist then the entire state will be restored from the changelog - > which could take some time. I suspect this is what is happening? > > As for the RocksDB memory settings, yes the off heap memory usage does > sneak under the radar. There is a memory management story for Kafka Streams > that is yet to be started. This would involve limiting the off-heap memory > that RocksDB uses. > > Thanks, > Damian > > On Fri, 25 Nov 2016 at 21:14 Frank Lyaruu <flya...@gmail.com> wrote: > > > I'm running all on a single node, so there is no 'data mobility' > involved. > > So if Streams does not use any existing data, I might as well wipe the > > whole RocksDb before starting, right? > > > > As for the RocksDb tuning, I am using a RocksDBConfigSetter, to reduce > the > > memory usage a bit: > > > > options.setWriteBufferSize(3000000); > > options.setMaxBytesForLevelBase(30000000); > > options.setMaxBytesForLevelMultiplier(3); > > > > I needed to do this as my 16Gb machine would die otherwise but I honestly > > was just reducing values more or less randomly until it wouldn't fall > over. > > I have to say this is a big drawback of Rocks, I monitor Java memory > usage > > but this just sneaks under the radar as it is off heap, and it isn't very > > clear what the implications are of different settings, as I can't says > > something like the Xmx heap setting, meaning: Take whatever you need up > to > > this maximum. Also, if I get this right, in the long run, as the data set > > changes and grows, I can never be sure it won't take too much memory. > > > > I get the impression I'll be better off with an external store, > something I > > can monitor, tune and restart separately. > > > > But I'm getting ahead of myself. I'll wipe the data before I start, see > if > > that gets me any stability > > > > > > > > > > On Fri, Nov 25, 2016 at 4:54 PM, Damian Guy <damian....@gmail.com> > wrote: > > > > > Hi Frank, > > > > > > If you have run the app before with the same applicationId, completely > > shut > > > it down, and then restarted it again, it will need to restore all of > the > > > state which will take some time depending on the amount of data you > have. > > > In this case the placement of the partitions doesn't take into account > > any > > > existing state stores, so it might need to load quite a lot of data if > > > nodes assigned certain partitions don't have that state-store (this is > > > something we should look at improving). > > > > > > As for RocksDB tuning - you can provide an implementation of > > > RocksDBConfigSetter via config: StreamsConfig.ROCKSDB_CONFIG_ > SETTER_CLASS > > > it has a single method: > > > > > > public void setConfig(final String storeName, final Options options, > > > final Map<String, Object> configs) > > > > > > in this method you can set various options on the provided Options > > object. > > > The options that might help in this case are: > > > options.setWriteBufferSize(..) - default in streams is 32MB > > > options.setMaxWriteBufferNumer(..) - default in streams is 3 > > > > > > However, i'm no expert on RocksDB and i suggest you have look at > > > https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide for more > > > info. > > > > > > Thanks, > > > Damian > > > > > > On Fri, 25 Nov 2016 at 13:02 Frank Lyaruu <flya...@gmail.com> wrote: > > > > > > > @Damian: > > > > > > > > Yes, it ran before, and it has that 200gb blob worth of Rocksdb stuff > > > > > > > > @Svente: It's on a pretty high end san in a managed private cloud, > I'm > > > > unsure what the ultimate storage is, but I doubt there is a > performance > > > > problem there. > > > > > > > > On Fri, 25 Nov 2016 at 13:37, Svante Karlsson < > svante.karls...@csi.se> > > > > wrote: > > > > > > > > > What kind of disk are you using for the rocksdb store? ie spinning > or > > > > ssd? > > > > > > > > > > 2016-11-25 12:51 GMT+01:00 Damian Guy <damian....@gmail.com>: > > > > > > > > > > > Hi Frank, > > > > > > > > > > > > Is this on a restart of the application? > > > > > > > > > > > > Thanks, > > > > > > Damian > > > > > > > > > > > > On Fri, 25 Nov 2016 at 11:09 Frank Lyaruu <flya...@gmail.com> > > wrote: > > > > > > > > > > > > > Hi y'all, > > > > > > > > > > > > > > I have a reasonably simple KafkaStream application, which > merges > > > > about > > > > > 20 > > > > > > > topics a few times. > > > > > > > The thing is, some of those topic datasets are pretty big, > about > > > 10M > > > > > > > messages. In total I've got > > > > > > > about 200Gb worth of state in RocksDB, the largest topic is 38 > > Gb. > > > > > > > > > > > > > > I had set the MAX_POLL_INTERVAL_MS_CONFIG to one hour to cover > > the > > > > > > > initialization time, > > > > > > > but that does not seem nearly enough, I'm looking at more than > > two > > > > hour > > > > > > > startup times, and > > > > > > > that starts to be a bit ridiculous. > > > > > > > > > > > > > > Any tips / experiences on how to deal with this case? Move away > > > from > > > > > > Rocks > > > > > > > and use an external > > > > > > > data store? Any tuning tips on how to tune Rocks to be a bit > more > > > > > useful > > > > > > > here? > > > > > > > > > > > > > > regards, Frank > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- -- Guozhang
