Gwen, Is there any particular reason why "inactive" (no consumers or producers for a topic) files need to be open?
Chris -- Learn microservices - http://learnmicroservices.io Microservices application platform http://eventuate.io On Fri, Jul 29, 2016 at 6:33 PM, Gwen Shapira <g...@confluent.io> wrote: > woah, it looks like you have 15,000 replicas per broker? > > You can go into the directory you configured for kafka's log.dir and > see how many files you have there. Depending on your segment size and > retention policy, you could have hundreds of files per partition > there... > > Make sure you have at least that many file handles and then also add > handles for the client connections. > > 1 million file handles sound like a lot, but you are running lots of > partitions per broker... > > We normally don't see more than maybe 4000 per broker and most > clusters have a lot fewer, so consider adding brokers and spreading > partitions around a bit. > > Gwen > > On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues > <kessi...@callinize.com> wrote: > > Hi guys, > > > > I have been experiencing some issues on kafka, where its throwing too > many open files. > > > > I have around of 6k topics and 5 partitions each. > > > > My cluster was made with 6 brokers. All of them are running Ubuntu 16 > and the file limits settings are: > > > > `cat /proc/sys/fs/file-max` > > 2000000 > > > > `ulimit -n` > > 1000000 > > > > Anyone has experienced it before? >
