Hi,all Maybe this is unix domain socket's bug.I found that qemu send the fd to vapp the fd's f_count inc twice in kernel.
1.kernel calls when we call send. unix_stream_sendmsg -> unix_scm_to_skb -> unix_attach_fds -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) Maybe should't inc the f_count when call send. 2.kernel calls when we call recv unix_stream_recvmsg -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) On 2014/10/20 14:26, Wen Congyang wrote: > On 10/20/2014 02:17 PM, Linhaifeng wrote: >> >> >> On 2014/10/20 13:32, Wen Congyang wrote: >>> On 10/20/2014 12:48 PM, Linhaifeng wrote: >>>> >>>> >>>> On 2014/10/20 10:12, Wen Congyang wrote: >>>>> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>>>>> >>>>>> >>>>>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>>>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng....@huawei.com >>>>>>>>> wrote: >>>>>>>>>> From: linhaifeng <haifeng....@huawei.com> >>>>>>>>>> >>>>>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>>>>> qemu should close all shared fd when exit. >>>>>>>>>> >>>>>>>>>> Signed-off-by: linhaifeng <haifeng....@huawei.com> >>>>>>>>> >>>>>>>>> Err, all file descriptors are closed automatically when a process >>>>>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>>>>> functional effect on a resource leak. >>>>>>>>> >>>>>>>>> If QEMU has sent the FD to another process, that process has a >>>>>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>>>>> not close the FD in the other process. You need the other process >>>>>>>>> to exit for the copy to be closed. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Daniel >>>>>>>>> >>>>>>>> Hi,daniel >>>>>>>> >>>>>>>> QEMU send the fd by unix domain socket.unix domain socket just install >>>>>>>> the fd to >>>>>>>> other process and inc the f_count,if qemu not close the fd the f_count >>>>>>>> is not dec. >>>>>>>> Then the other process even close the fd the hugepage would not freed >>>>>>>> whise the other process exit. >>>>>>> >>>>>>> The kernel always closes all FDs when a process exits. So if this FD is >>>>>>> not being correctly closed then it is a kernel bug. There should never >>>>>>> be any reason for an application to do close(fd) before exiting. >>>>>>> >>>>>>> Regards, >>>>>>> Daniel >>>>>>> >>>>>> Hi,daniel >>>>>> >>>>>> I don't think this is kernel's bug.May be this a problem about usage. >>>>>> If you open a file you should close it too. >>>>> >>>>> If you don't close it, the kernel will help you when the program exits. >>>>> >>>> Yes,when the hugepage is only used for qemu,the kernel will free the file >>>> object.If the hugepage shared for other process,when qemu exit the kernel >>>> will not free the file. >>> >>> Even if the hugepage is shared with the other process, the kernel will auto >>> close the fd when qemu >>> exits. If the kernel doesn't do it, it is a kernel bug. >>> >> Kernel supply close to fix this bug.If you call open you must call close. >> If not, the result is unpredictability. > > No, if the program exists, the kernel must close all fd used by the program. > So, there is no need to close fd before program exists. > > Thanks > Wen Congyang > >>>>>> >>>>>> This is <<linux man page>>about how to free resource of file. >>>>>> http://linux.die.net/man/2/close >>>>>> >>>>>> >>>>>> I'm trying to describe my problem. >>>>>> >>>>>> For example, there are 2 VMs run with hugepage and the hugepage only for >>>>>> QEMU to use. >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the two VMs.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage.After this step the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> 5.shutdown VM with signal 15 without close(fd).After this step the >>>>>> meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Yes,it works well,like you said the kernel recycle all resources. >>>>>> >>>>>> For another example,there are 2 VMs run with hugepage and share the >>>>>> hugepage with vapp(a vhost-user application). >>>>> >>>>> The vapp is your internal application? >>>>> >>>> Yes vapp is a application to share the QEMU's hugepage.So threr are two >>>> process use the hugepage. >>>> >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the first VM.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage and send the fd to vapp with unix domain >>>>>> socket.After this step the meminfo is: >>> >>> Do you modify qemu? >>> >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the second VM.After this step the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Then I want to close the first VM and run another VM.After close the >>>>>> first VM and close the fd in vapp the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>> >>>>> Does the qemu still run after you close the first VM? If the qemu exits, >>>>> the fd will be closed by the kernel, so this >>>>> bug is very strange. >>>>> >>>> qemu is not run when close the first VM.If other process used the file >>>> will be closed by kernel too? >>> >>> If qeum doesn't run after the first vm is closed, the fd should be closed >>> even if another process uses the file. >>> >>>> >>>>>> >>>>>> So failed to run the third VM because the first VM have not free the >>>>>> hugepage.After apply this patch the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> So i can run the third VM success. >>>>>> >>>>> >>>>> >>>>> . >>>>> >>>> >>> >>> >>> . >>> >> > > > . > -- Regards, Haifeng