fangwentong created FILEUPLOAD-288: -------------------------------------- Summary: Linux dentry cache memory leak due to inappropriate temporary file detection Key: FILEUPLOAD-288 URL: https://issues.apache.org/jira/browse/FILEUPLOAD-288 Project: Commons FileUpload Issue Type: Improvement Environment: OS version: CentOS release 6.7 (Final) Kenel: 2.6.32-573.26.1.el6.x86_64 JVM: Oracle 1.8.0_102-b14 Tomcat: 8.0.33.0 commons-fileupload : 1.3.3 Reporter: fangwentong
We use commons-fileupload:1.3.3 to handle multipart request, param {{file-size-threshold}} was set to 200K. Our API handle multipart request in high QPS, the size of request body is about 20KB, which means those parts was saved in memory, not written to disk temporary file. But, in our production environment, size of dentry cache grows up in high speed. here are some analysis: in {{strace}} result, plenty of {{stat}} system call invoked for tmp files which not exists. {noformat} [pid 111310] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 [pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624923.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624922.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624921.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624920.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624919.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624918.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 69798] stat("/tmp", <unfinished ...> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624917.tmp", <unfinished ...> [pid 69798] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0 [pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624916.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624915.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624914.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624913.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624912.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624911.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624910.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624909.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624908.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624907.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624906.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624905.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624904.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624903.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624902.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624901.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624900.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624899.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624898.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624897.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624896.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624895.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624894.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624893.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624892.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624891.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624890.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624889.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624888.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624887.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624886.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624885.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624884.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory) [pid 78454] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=57810, si_uid=605} ---{noformat} in {{slabtop}} result , dentry cache taken up 22GB+ memory. {noformat} Active / Total Objects (% used) : 233715492 / 234153507 (99.8%) Active / Total Slabs (% used) : 7858579 / 7858606 (100.0%) Active / Total Caches (% used) : 108 / 200 (54.0%) Active / Total Size (% used) : 29322726.34K / 29380106.43K (99.8%) Minimum / Average / Maximum Object : 0.02K / 0.12K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 116095303 116095295 26% 0.06K 1967717 59 7870868K size-64 116090260 116090260 26% 0.19K 5804513 20 23218052K dentry {noformat} finally, I think the root cause is:in temporary file cleanup policy defined at {{DiskFileItem#finalize}}, call {{outputFile.exists()}} for every File, even if the part not write to disk. {code:java} @Override protected void finalize() { if (dfos == null) { return; } File outputFile = dfos.getFile(); if (outputFile != null && outputFile.exists()) { outputFile.delete(); } } {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)