On 2019/06/26 21:58:02, Patrik Kleindl <pklei...@gmail.com> wrote:
> Hi Kiran
> You can use the RocksDBConfigSetter and pass
>
> options.setMaxOpenFiles(100);
>
> to all RocksDBs for the Streams application which limits how many are
> kept open at the same time.
>
> best regards
>
> Patrik
>
>
> On Wed, 26 Jun 2019 at 16:14, emailtokir...@gmail.com <
> emailtokir...@gmail.com> wrote:
>
> > Hi,
> >
> > We are using Kafka streams DSL APIs for doing some counter aggregations
> > (running on OpenJDK 11.0.2). Our topology has some 400 sub topologies & we
> > are using 8 partitions in source topic. When we start pumping more load, we
> > start getting RockDBException stating "too many open files".
> >
> > Here are the stack trace samples:
> >
> > ------------------------------------------------------------------------------------------
> > Caused by: org.rocksdb.RocksDBException: while open a file for lock:
> > PPPPPPPPPPP.1512000000/LOCK: Too many open files
> > at org.rocksdb.RocksDB.open(Native Method)
> > at org.rocksdb.RocksDB.open(RocksDB.java:235)
> > at
> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156)
> > ... 24 common frames omitted
> >
> >
> > Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error
> > while executing flush from store XXXXXXXXXXX.1512000000
> > at
> > org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:397)
> > at
> > org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:388)
> > at
> > org.apache.kafka.streams.state.internals.Segments.flush(Segments.java:163)
> > at
> > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.flush(RocksDBSegmentedBytesStore.java:178)
> > at
> > org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85)
> > at
> > org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85)
> > at
> > org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:130)
> > at
> > org.apache.kafka.streams.state.internals.MeteredWindowStore.flush(MeteredWindowStore.java:177)
> > at
> > org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:217)
> > ... 10 more
> > Caused by: org.rocksdb.RocksDBException: While open a file for appending:
> > YYYYYYYYYYYYY.1512000000/000007.dbtmp: Too many open files
> > at org.rocksdb.RocksDB.flush(Native Method)
> > at org.rocksdb.RocksDB.flush(RocksDB.java:3401)
> > at org.rocksdb.RocksDB.flush(RocksDB.java:3361)
> > at
> > org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:395)
> >
> > ------------------------------------------------------------------------------------------
> >
> > We tried increasing the open files limit at OS level to some decent
> > number.. but still no luck. Obviously we don't want to have boundless open
> > files..
> >
> > We also tried to play with commit interval(kafka.commit.interval.ms) and
> > cache size (kafka.cache.max.bytes.buffering) .. but no luck there either.
> >
> > KAFKA-3904 talks about it.. but it was resolved long back..
> >
> > Any other config tuning we have to do?
> >
> > Appreciate any help in this regard!
> >
> > Thanks,
> > Kiran
> >
> >
>
Hi Patrik/All,
Thanks for providing some valuable pointer!
I did that & it doesn't seems to work.
Here is how my custom config setter looks like:
----------------------------------------------------------------------------------------------------
@Override
public void setConfig(final String storeName, final Options options,
final Map<String, Object> configs) {
// See #1 below.
BlockBasedTableConfig tableConfig = new
org.rocksdb.BlockBasedTableConfig();
tableConfig.setBlockCacheSize(16 * 1024 * 1024L);
// See #2 below.
tableConfig.setBlockSize(16 * 1024L);
// See #3 below.
tableConfig.setCacheIndexAndFilterBlocks(false);
// tableConfig.setPinL0FilterAndIndexBlocksInCache(true);
options.setMaxOpenFiles(-1);
options.setTableFormatConfig(tableConfig);
// See #4 below.
options.setMaxWriteBufferNumber(2);
}
----------------------------------------------------------------------------------------------------
I tried many options with this:
1. tableConfig.setCacheIndexAndFilterBlocks(true); ----> as per docs
(https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks)
if we set to true, the max_open_files shouldn't play a role. But I was still
getting too many open files exception from Rocksdb
2. tableConfig.setCacheIndexAndFilterBlocks(true);
tableConfig.setPinL0FilterAndIndexBlocksInCache(true); ----> no luck; same
exception
3. tableConfig.setCacheIndexAndFilterBlocks(false); and
options.setMaxOpenFiles(50000); -----> This also resulted with same
exception.. with java process having ~24K open files
4. tableConfig.setCacheIndexAndFilterBlocks(false); and
options.setMaxOpenFiles(-1); -----> This also resulted with same
exception.. with java process having ~24K ope files. As per the doc, if we set
to -1, it means infinite and controlled by underlying OS limit.
I am using MacOS Mojave (10.14.4) and OpenJDK 11.0.2. At OS level, I have
bumped the open files limit to 1 million.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1000000
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1418
virtual memory (kbytes, -v) unlimited
Am I missing some other config here?
Thanks,
Kiran
