I know the code is a bit messy, but do you still see an issue compiling
linux kernel after commit 5c2efa8f077fafa82023f5aec5e2c474c5ed2fdf?
Regards, Malahal.
On Thu, Oct 19, 2017 at 2:07 PM, LUCAS Patrice <patrice.lu...@cea.fr> wrote:
> On 10/18/17 15:40, Frank Filz wrote:
>
>> Hmm, this discussion got stalled, but Patrice reminded me that we need to
>> continue it...
>>
>> On 09/21/2017 07:45 PM, Frank Filz wrote:
>>>
>>>> Philippe discovered that recent Ganesha will no longer allow compiling
>>>> the linux kernel due to dangling open file descriptors.
>>>>
>>>> I'm not sure if there is any true leak, the simple test of echo foo >
>>>> /mnt/foo does show a remaining open fd for /mnt/foo, however that is
>>>> the global fd opened in the course of doing a getattrs on FSAL_ VFS.
>>>>
>>>> We have been talking about how the current management of open file
>>>> descriptors doesn't really work, so I have a couple proposals:
>>>>
>>>> 1. We really should have a limit on the number of states we allow. Now
>>>> that NLM locks and shares also have a state_t, it would be simple to
>>>> have a count of how many are in use, and return a resource error if an
>>>> operation requires creating a new one past the limit. This can be a
>>>> hard limit with no grace, if the limit is hit, then alloc_state fails.
>>>>
>>> This I agree with.
>>>
>>> 2. Management of the global fd is more complex, so here goes:
>>>>
>>>> Part of the proposal is a way for the FSAL to indicate that an FSAL
>>>> call used the global fd in a way that consumes some kind of resource
>>>> the FSAL would like managed.
>>>>
>>>> FSAL_PROXY should never indicate that (anonymous I/O should be done
>>>> using a special stateid, and a simple file create should result in the
>>>> open stateid immediately being closed, if that's not the case, then
>>>> it's easy enough to indicate use of a limited resource.
>>>>
>>>> FSAL_VFS would indicate use of the resource any time it utilizes the
>>>> global fd. If it uses a temp fd that is closed after performing the
>>>> operation, it would not indicate use of the limited resource.
>>>>
>>>> FSAL_GPFS, FSAL_GLUSTER, and FSAL_CEPH should all be similar to
>>>>
>>> FSAL_VFS.
>>>
>>>> FSAL_RGW only has a global fd, and I don't quite understand how it is
>>>> managed.
>>>>
>>> If only PROXY doesn't set this, then maybe it's added complexity we don't
>>> need. Just assume it's set.
>>>
>> Matt, could you chime in on RGW? It sounds like FSAL_RGW and/or the RGW
>> library really manage the open/close state of the file. If so, then you
>> don't need hints from MDCACHE LRU...
>>
>> The main part of the proposal is to actually create a new LRU queue
>>>> for objects that are using the limited resource.
>>>>
>>>> If we are at the hard limit on the limited resource and an entry that
>>>> is not already in the LRU uses the resource, then we would reap an
>>>> existing entry and call fsal_close on it to release the resource. If
>>>> an entry was not available to be reaped, we would temporarily exceed
>>>> the limit just like we do with mdcache entries.
>>>>
>>>> If an FSAL call resulted in use of the resource and the entry was
>>>> already in the resource LRU, then it would be bumped to MRU of L1.
>>>>
>>>> The LRU run thread for the resource would demote objects from LRU L1
>>>> to MRU of L2, and call fsal_close and remove objects from LRU of L2. I
>>>> think it should work to close any files that have not been used in the
>>>> amount of time, really using the L1 and L2 to give a shorter life to
>>>> objects for which the resource is used once and then not used again,
>>>> whereas a file that is accessed multiple times would have more
>>>> resistance to being closed. I think the exact mechanics here may need
>>>>
>>> some tuning, but that's the general idea.
>>>
>>>> The idea here is to be constantly closing files that have not been
>>>> accessed recently, and also to better manage a count of the files for
>>>> which we are actually using the resources, and not keep a file open
>>>> just because for some reason we do lots of lookups or stats of it (we
>>>> might have to open it for getattrs, but then we might serve a bunch of
>>>> cached attrs, which doesn't go to disk, might as well close the fd).
>>>>
>>> This sounds almost exactly like the existing LRU thread, except that it
>>>
>> ignores
>>
>>> refcount. If you remove global FD from the obj_handle, then the LRU as
>>> it
>>> currently exists becomes unnecessary for MDCACHE entries, as they only
>>> need a simple, single-level LRU based only on initial refcounts. The
>>>
>> current,
>>
>>> multi-level LRU only exists to close the global FD when transitioning LRU
>>> levels.
>>>
>> The multi-level LRU for handle cache still have some value for scan
>> resistance.
>>
>> So, what it sounds like to me is that you're splitting the LRU for entries
>>>
>> from
>>
>>> the LRU for global FDs. Is this correct? If so, I think this
>>> complicates
>>>
>> the two
>>
>>> sets of LRU transitions, but probably not insurmountably so.
>>>
>>> I also propose making the limit for the resource configurable
>>>> independent of the ulimit for file descriptors, though if an FSAL is
>>>> loaded that actually uses file descriptors for open files should check
>>>> that the ulimit is big enough, it should also include the limit on
>>>> state_t also. Of course it will be impossible to account for file
>>>> descriptors used for sockets, log files, config files, or random
>>>>
>>> libraries that
>>
>>> like to open files...
>>>
>>> Hmmm... I don't think we can do any kind of checking, if we're not going
>>>
>> to
>>
>>> use ulimit by default, since it depends on which FSALs are in use at any
>>>
>> given
>>
>>> time. I say we either default the limits to ulimit, or just ignore
>>> ulimit
>>>
>> entirely
>>
>>> and log an appropriate error when EMFILE is returned.
>>>
>> Yea, that certainly would be the simplest mechanism. Along with
>> documentation. Specific products which know which FSAL they are using and
>> may know other quirks could always set ulimit to the config value + some
>> reasonable amount to cover things like log file open/close and sockets
>> (hmm,
>> do we have a configurable limit on the number of TCP connections we
>> accept?).
>>
>> Frank
>>
>>
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> https://www.avast.com/antivirus
>>
>>
> Isn't it only a question of layering the code ? If limited ressources are
> used by any fsal, why don't letting each FSAL managing it ?
>
>
> The FSAL-API seems to be designed without any reference to the "global fd"
> concept whereas some parts of the code outside of any fsals seem to be very
> link to this concept. Using a "global fd", isn't-it only a FSAL deal ? A
> very common deal (this choice seems to be done by vfs, ceph, gpfs and
> gluster, fsals), but only at fsal level. It looks like using a global fd is
> "hardcoded" outside of the FSAL whereas there is no track of it in the
> FSAL-API.
>
>
> Best regards,
>
> --
> Patrice LUCAS
> Ingenieur-Chercheur, CEA-DAM/DSSI/SISR/LA2S
> tel : +33 (0)1 69 26 47 86
> e-mail : patrice.lu...@cea.fr
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
