On Tuesday, March 13, 2018 at 9:38:05 AM UTC, Dima Pasechnik wrote: > > > >> I can imagine a small function constructing a matrix (for which it needs >> to construct the MatrixSpace) to do some computation but not keeping >> that matrix in memory. For example, the output of that function could be >> the determinant of the matrix. If this function is called from a loop, >> you end up in the scenario that I described. >> > > this example, assuming the matrix size is a parameter, would easily become > a memory leak, > just imagine calling this function millions of times, for 10 different > fields, with random matrix sizes. > > Indeed. The strong LRU/FIFO cache needs to be placed very carefully. It produces the kind of optimization that a memory pool gives. In the latter there's a natural granularity: block size. Here it's just time since use.
The random matrix sizes give a good example where this caching is counterproductive: by keeping parents around that aren't used anymore, you're preventing quick memory reuse so one would actually be deteriorating runtime (even before leaking becomes a real problem) because you'd be making life harder for the memory cache of the processor. Note that, with the missing LRU/FIFO there's a viable strategy to counteracting the overhead: profile the code, note that some parent construction is a bottleneck, identify which parent (that would be clear from the call graph), and construct that parent before the loop and keep a strong reference. Counterproductive behaviour of the LRU/FIFO would be much harder to detect and even harder to work around. I recall that for exactly the reason of avoiding expensive parent creation, we used to have deferred full parent initialization of matrix spaces; exactly for the scenario where a matrix only gets created as a temporary thing, for instance for taking its determinant. The advantage of that approach is that you even gain on the first ephemeral use. Did we lose that for a reason? Perhaps full parent initialization isn't as expensive as it used to be? (I have trouble believing that, with the introduction of the category framework). So: the implementation of the CachedWeakValueDictionary on the ticket is fine, but I doubt that it's actually the appropriate solution for the problems sketched. We'd need to have actual instances of the problem to see. Probably some modular forms code somewhere will be quite linear algebra intensive. The code on https://trac.sagemath.org/ticket/15113 would be another example. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
