[
https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henry Cai updated KAFKA-3904:
-
Description:
I noticed when my application was running long (> 1 day), I will get 'Too many
open files' error.
I used 'lsof' to list all the file descriptors used by the process, it's over
32K, but most of them belongs to the .lock file, e.g. this same lock file shows
2700 times.
I looked at the code, I think the problem is in:
File lockFile = new File(stateDir, ProcessorStateManager.LOCK_FILE_NAME);
FileChannel channel = new RandomAccessFile(lockFile, "rw").getChannel();
Each time new RandomAccessFile is called, a new fd will be created, we probably
should either close or reuse this RandomAccessFile object.
lsof result:
java14799 hcai *740u REG9,00 2415928585
/mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
java14799 hcai *743u REG9,00 2415928585
/mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
java14799 hcai *746u REG9,00 2415928585
/mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
java14799 hcai *755u REG9,00 2415928585
/mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
hcai@teststream02001:~$ lsof -p 14799 | grep lock | grep 0_16 | wc
2709 24381 319662
was:
Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, or
the default values are directly used. We need to make them configurable for
advanced users. For example, some default values may not work perfectly for
some scenarios:
https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar
to "StreamsConfig", which defines all related rocksDB options configs, that can
be passed as key-value pairs to "StreamsConfig".
> File descriptor leaking (Too many open files) for long running stream process
> -
>
> Key: KAFKA-3904
> URL: https://issues.apache.org/jira/browse/KAFKA-3904
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Henry Cai
>Assignee: Henry Cai
> Labels: api, newbie
>
> I noticed when my application was running long (> 1 day), I will get 'Too
> many open files' error.
> I used 'lsof' to list all the file descriptors used by the process, it's over
> 32K, but most of them belongs to the .lock file, e.g. this same lock file
> shows 2700 times.
> I looked at the code, I think the problem is in:
> File lockFile = new File(stateDir, ProcessorStateManager.LOCK_FILE_NAME);
> FileChannel channel = new RandomAccessFile(lockFile, "rw").getChannel();
> Each time new RandomAccessFile is called, a new fd will be created, we
> probably should either close or reuse this RandomAccessFile object.
> lsof result:
> java14799 hcai *740u REG9,00 2415928585
> /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
> java14799 hcai *743u REG9,00 2415928585
> /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
> java14799 hcai *746u REG9,00 2415928585
> /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
> java14799 hcai *755u REG9,00 2415928585
> /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock
> hcai@teststream02001:~$ lsof -p 14799 | grep lock | grep 0_16 | wc
>2709 24381 319662
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
