On 2010-10-24 16:18, Francois Deppierraz wrote: > The same problem could also be reproduced by repeatedly listing a single > directory. Your suspicion that the ResponseCache object is living longer > than expected sounds correct. > > Each read operation on a mutable file adds a new object into the > ResponseCache even when there's already an entry for this file. This > makes me think that this ResponseCache is not actually being used. > > Would it be possible that some field of the verinfo tuple (such as > offset) are different between calls to mutable.filenode._add_to_cache() > and mutable.filenode._read_from_cache()? > > On 10/22/2010 09:23 AM, Brian Warner wrote: >> Or, maybe we should get rid of ResponseCache >> altogether: I don't remember what performance improvement it tries to >> provide.
If there were no cache at all for mutable files, each operation on a mutable directory would redownload shares from the storage servers (note that DirectoryNode._read in dirnode.py appears to be called for each directory operation). > I gave a try to this approach with the following change and the memory > leak indeed disappeared! > > http://github.com/ctrlaltdel/tahoe-lafs/commit/3a4dbdee1a3ea8a30559dd49f51dba40ba1661fb Even if it were working as intended, the ResponseCache stores shares as retrieved from storage servers. Wouldn't it be more efficient, in both memory and compute cycles, to cache the reconstructed file rather than the shares? Ah, I see that is ticket #465. Since I think the main performance issue here is for directories, another possibility is to cache the decoded directory contents in DirectoryNode, rather than trying to cache mutable file shares in general. (nodemaker.py create_from_cap will ensure that a NodeMaker creates only one node object for a given cap. However, there will be separate DirectoryNode objects and separate underlying MutableFileNode objects for read and write caps to a directory -- see also ticket #1105. But this would not cause a performance regression since the two MutableFileNode objects would currently have separate ResponseCaches.) There is no additional security problem created by cacheing plaintext in memory, since the corresponding filecap is already in memory. -- David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
signature.asc
Description: OpenPGP digital signature
_______________________________________________ tahoe-dev mailing list tahoe-dev@tahoe-lafs.org http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
