Re: .deleted file descriptors
I see DEL file descriptors as well. java2978 root DELREG 202,16 1074192884 /mnt/data1/kafka/flirt_tasks-1/000143874814.index.deleted Using java from Oracle java version "1.7.0_72" Java(TM) SE Runtime Environment (build 1.7.0_72-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode) On Sun, Mar 1, 2015 at 11:14 PM, Jaikiran Pai wrote: > One thing to remember is that the .index files are memory-mapped [1] which > in Java means that the file descriptors may not be released even when the > program is done using it. A garbage collection is expected to close such > resources, but forcing a System.gc() is only a hint and thus doesn't > guarantee that it will trigger the garbage collection and close that > resource. More details in [2]. In order to confirm that this is what you > are running into, can you print the output of the following: > > lsof -p | grep .deleted > > Before running that command do wait for at least the retention timeout and > file deletion delay to have actually passed. The interesting part in that > output would be the "type" column. For example, in my case it shows DEL as > the value for that column: > > java7518 jaikiran DELREG8,6 1453306 > /tmp/kafka-logs/hey-0/.index.deleted > > when the index file has really been deleted but the JVM hasn't yet let go > off the file descriptor resource. I did try to run a gc() from the jconsole > MBean for that process, but that too didn't free this resource, but I > didn't really expect it to, since there's not much control on GC itself. > > By the way which exact vendor and version of Java do you use? [3] suggests > that an update in Oracle Java 1.6 has a workaround to this problem, but > that workaround only comes into picture when a subsequent FileChannel.map > calls results in a OOM. > > [1] http://docs.oracle.com/javase/7/docs/api/java/nio/ > MappedByteBuffer.html > [2] http://bugs.java.com/view_bug.do?bug_id=4724038 > [3] http://bugs.java.com/view_bug.do?bug_id=6417205 > > -Jaikiran > > > On Monday 02 March 2015 10:30 AM, Guangle Fan wrote: > >> Slightly different from what I observed. >> >> Broker box has 800GB disk space. By setting the appropriate log retention, >> it's supposed to hold the log size. But then the usage of disk hits 90%, >> and by doing nothing but restarting broker server. It free 40% disk space. >> It's for sure the speed of the traffic won't be able to fill 40% disk >> space >> within one minute period (log.delete.delay.ms). >> >> The obvious change before and after restart broker server is broker server >> frees tons of file descriptors of .index. Most of those file descriptors >> are very old. >> >> lsof -p | grep .deleted >> >> Don't know how come kafka didn't release those file descriptors and what's >> the dial of it. >> >> >> >> >> >> >> >> >> On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat < >> gharatmayures...@gmail.com >> >>> wrote: >>> Also I suppose when the broker starts up it will remove the files that >>> are >>> marked with suffix .deleted and that's why you can see the free disk >>> space >>> on restarting. Guozhang can correct me if I am wrong. >>> >>> Thanks, >>> >>> Mayuresh >>> >>> On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang >>> wrote: >>> >>> Guangle, The deletion of the segment log / index files are async, i.e. when Kafka decide to clean the logs, it only adds a suffix ".deleted" to the files such that it will not be access any more by other Kafka threads. The >>> actual >>> file deletion will be executed later, with period controlled by " file.delete.delay.ms" (default 1 minute). On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan >>> wrote: >>> Hi, > > After Kafka cleaned .log / .index files based on topic retention. I can > still lsof a lot of .index.deleted files. And df shows usage on disk > space > is accumulated to full. > > When this happened, just by restarting broker, it will immediately free > those disk space. I seems to me kafka after cleaning expired files > still >>> hold file descriptors which lead to disk space still being held. > > How do you config kafka to let kafka release file descriptors in this > case > ? > > Using kafka 0.8.1.1 > > Regards, > > Guangle > > -- -- Guozhang >>> >>> -- >>> -Regards, >>> Mayuresh R. Gharat >>> (862) 250-7125 >>> >>> >
Re: .deleted file descriptors
One thing to remember is that the .index files are memory-mapped [1] which in Java means that the file descriptors may not be released even when the program is done using it. A garbage collection is expected to close such resources, but forcing a System.gc() is only a hint and thus doesn't guarantee that it will trigger the garbage collection and close that resource. More details in [2]. In order to confirm that this is what you are running into, can you print the output of the following: lsof -p | grep .deleted Before running that command do wait for at least the retention timeout and file deletion delay to have actually passed. The interesting part in that output would be the "type" column. For example, in my case it shows DEL as the value for that column: java7518 jaikiran DELREG8,6 1453306 /tmp/kafka-logs/hey-0/.index.deleted when the index file has really been deleted but the JVM hasn't yet let go off the file descriptor resource. I did try to run a gc() from the jconsole MBean for that process, but that too didn't free this resource, but I didn't really expect it to, since there's not much control on GC itself. By the way which exact vendor and version of Java do you use? [3] suggests that an update in Oracle Java 1.6 has a workaround to this problem, but that workaround only comes into picture when a subsequent FileChannel.map calls results in a OOM. [1] http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html [2] http://bugs.java.com/view_bug.do?bug_id=4724038 [3] http://bugs.java.com/view_bug.do?bug_id=6417205 -Jaikiran On Monday 02 March 2015 10:30 AM, Guangle Fan wrote: Slightly different from what I observed. Broker box has 800GB disk space. By setting the appropriate log retention, it's supposed to hold the log size. But then the usage of disk hits 90%, and by doing nothing but restarting broker server. It free 40% disk space. It's for sure the speed of the traffic won't be able to fill 40% disk space within one minute period (log.delete.delay.ms). The obvious change before and after restart broker server is broker server frees tons of file descriptors of .index. Most of those file descriptors are very old. lsof -p | grep .deleted Don't know how come kafka didn't release those file descriptors and what's the dial of it. On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat wrote: Also I suppose when the broker starts up it will remove the files that are marked with suffix .deleted and that's why you can see the free disk space on restarting. Guozhang can correct me if I am wrong. Thanks, Mayuresh On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang wrote: Guangle, The deletion of the segment log / index files are async, i.e. when Kafka decide to clean the logs, it only adds a suffix ".deleted" to the files such that it will not be access any more by other Kafka threads. The actual file deletion will be executed later, with period controlled by " file.delete.delay.ms" (default 1 minute). On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan wrote: Hi, After Kafka cleaned .log / .index files based on topic retention. I can still lsof a lot of .index.deleted files. And df shows usage on disk space is accumulated to full. When this happened, just by restarting broker, it will immediately free those disk space. I seems to me kafka after cleaning expired files still hold file descriptors which lead to disk space still being held. How do you config kafka to let kafka release file descriptors in this case ? Using kafka 0.8.1.1 Regards, Guangle -- -- Guozhang -- -Regards, Mayuresh R. Gharat (862) 250-7125
Re: .deleted file descriptors
Did you check if log.delete.delay.ms is set to 6? Guozhang On Sun, Mar 1, 2015 at 9:00 PM, Guangle Fan wrote: > Slightly different from what I observed. > > Broker box has 800GB disk space. By setting the appropriate log retention, > it's supposed to hold the log size. But then the usage of disk hits 90%, > and by doing nothing but restarting broker server. It free 40% disk space. > It's for sure the speed of the traffic won't be able to fill 40% disk space > within one minute period (log.delete.delay.ms). > > The obvious change before and after restart broker server is broker server > frees tons of file descriptors of .index. Most of those file descriptors > are very old. > > lsof -p | grep .deleted > > Don't know how come kafka didn't release those file descriptors and what's > the dial of it. > > > > > > > > > On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Also I suppose when the broker starts up it will remove the files that > are > > marked with suffix .deleted and that's why you can see the free disk > space > > on restarting. Guozhang can correct me if I am wrong. > > > > Thanks, > > > > Mayuresh > > > > On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang > wrote: > > > > > Guangle, > > > > > > The deletion of the segment log / index files are async, i.e. when > Kafka > > > decide to clean the logs, it only adds a suffix ".deleted" to the files > > > such that it will not be access any more by other Kafka threads. The > > actual > > > file deletion will be executed later, with period controlled by " > > > file.delete.delay.ms" (default 1 minute). > > > > > > On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan > > wrote: > > > > > > > Hi, > > > > > > > > After Kafka cleaned .log / .index files based on topic retention. I > can > > > > still lsof a lot of .index.deleted files. And df shows usage on disk > > > space > > > > is accumulated to full. > > > > > > > > When this happened, just by restarting broker, it will immediately > free > > > > those disk space. I seems to me kafka after cleaning expired files > > still > > > > hold file descriptors which lead to disk space still being held. > > > > > > > > How do you config kafka to let kafka release file descriptors in this > > > case > > > > ? > > > > > > > > Using kafka 0.8.1.1 > > > > > > > > Regards, > > > > > > > > Guangle > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > -- -- Guozhang
Re: .deleted file descriptors
Slightly different from what I observed. Broker box has 800GB disk space. By setting the appropriate log retention, it's supposed to hold the log size. But then the usage of disk hits 90%, and by doing nothing but restarting broker server. It free 40% disk space. It's for sure the speed of the traffic won't be able to fill 40% disk space within one minute period (log.delete.delay.ms). The obvious change before and after restart broker server is broker server frees tons of file descriptors of .index. Most of those file descriptors are very old. lsof -p | grep .deleted Don't know how come kafka didn't release those file descriptors and what's the dial of it. On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat wrote: > Also I suppose when the broker starts up it will remove the files that are > marked with suffix .deleted and that's why you can see the free disk space > on restarting. Guozhang can correct me if I am wrong. > > Thanks, > > Mayuresh > > On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang wrote: > > > Guangle, > > > > The deletion of the segment log / index files are async, i.e. when Kafka > > decide to clean the logs, it only adds a suffix ".deleted" to the files > > such that it will not be access any more by other Kafka threads. The > actual > > file deletion will be executed later, with period controlled by " > > file.delete.delay.ms" (default 1 minute). > > > > On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan > wrote: > > > > > Hi, > > > > > > After Kafka cleaned .log / .index files based on topic retention. I can > > > still lsof a lot of .index.deleted files. And df shows usage on disk > > space > > > is accumulated to full. > > > > > > When this happened, just by restarting broker, it will immediately free > > > those disk space. I seems to me kafka after cleaning expired files > still > > > hold file descriptors which lead to disk space still being held. > > > > > > How do you config kafka to let kafka release file descriptors in this > > case > > > ? > > > > > > Using kafka 0.8.1.1 > > > > > > Regards, > > > > > > Guangle > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
Re: .deleted file descriptors
Also I suppose when the broker starts up it will remove the files that are marked with suffix .deleted and that's why you can see the free disk space on restarting. Guozhang can correct me if I am wrong. Thanks, Mayuresh On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang wrote: > Guangle, > > The deletion of the segment log / index files are async, i.e. when Kafka > decide to clean the logs, it only adds a suffix ".deleted" to the files > such that it will not be access any more by other Kafka threads. The actual > file deletion will be executed later, with period controlled by " > file.delete.delay.ms" (default 1 minute). > > On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan wrote: > > > Hi, > > > > After Kafka cleaned .log / .index files based on topic retention. I can > > still lsof a lot of .index.deleted files. And df shows usage on disk > space > > is accumulated to full. > > > > When this happened, just by restarting broker, it will immediately free > > those disk space. I seems to me kafka after cleaning expired files still > > hold file descriptors which lead to disk space still being held. > > > > How do you config kafka to let kafka release file descriptors in this > case > > ? > > > > Using kafka 0.8.1.1 > > > > Regards, > > > > Guangle > > > > > > -- > -- Guozhang > -- -Regards, Mayuresh R. Gharat (862) 250-7125
Re: .deleted file descriptors
Guangle, The deletion of the segment log / index files are async, i.e. when Kafka decide to clean the logs, it only adds a suffix ".deleted" to the files such that it will not be access any more by other Kafka threads. The actual file deletion will be executed later, with period controlled by " file.delete.delay.ms" (default 1 minute). On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan wrote: > Hi, > > After Kafka cleaned .log / .index files based on topic retention. I can > still lsof a lot of .index.deleted files. And df shows usage on disk space > is accumulated to full. > > When this happened, just by restarting broker, it will immediately free > those disk space. I seems to me kafka after cleaning expired files still > hold file descriptors which lead to disk space still being held. > > How do you config kafka to let kafka release file descriptors in this case > ? > > Using kafka 0.8.1.1 > > Regards, > > Guangle > -- -- Guozhang
.deleted file descriptors
Hi, After Kafka cleaned .log / .index files based on topic retention. I can still lsof a lot of .index.deleted files. And df shows usage on disk space is accumulated to full. When this happened, just by restarting broker, it will immediately free those disk space. I seems to me kafka after cleaning expired files still hold file descriptors which lead to disk space still being held. How do you config kafka to let kafka release file descriptors in this case ? Using kafka 0.8.1.1 Regards, Guangle