On Tue, Sep 25, 2012 at 11:49 AM, Bruno Girin <[email protected]> wrote: True but then shouldn't it release memory allocated to thumbnails for a given BatchImportJob once that job is finished? By the looks of it, it doesn't always do that as the memory usage only goes up.
Yes, but the bottleneck at the end of the process causes the Pixbufs to queue up, leading to constant growth in the footprint. When the import does complete the memory is released. At least, it's *supposed* to, if not, that's a memory leak, but I don't think that's what we're seeing here. It's unfortunate that the thumbnail generation has to happen at the tail of the process. It's a result of an early design decision that the thumbnails are named by their corresponding photo's rowid in the SQLite database. Obviously you don't know the rowid until the row has been inserted, hence the thumbnails are the last step in the process. Also remember that GLib uses a slice allocator that will grow a pool of available memory so speed up malloc's. It's a kind of garbage collection, and means that the total memory Shotwell is using (as seen by the system) is not necessarily a reflection of how much *active* memory Shotwell as allocated at any moment. The way the F-Spot import was (re-)designed it creates a series of small BatchImportJob's rather than a single massive one to enable the core BatchImport class to clean up after each job. The idea was that even with a massive database to import, it should never really spike but should rather plateau with processed jobs being removed from the head of queue while new ones are added to the tail. Note that this is not necessarily the most efficient design in terms of processor performance because it results in quite a few threads but the size of each individual jobs could be quite easily adjusted. Yes, this is a solid improvement, because it allows for BatchImport to smooth out the tasks between the various stages. Import has proven to be of the trickiest beasts in Shotwell. It takes too long, so you increase parallelization, then it takes too much CPU/RAM/disk activity, so you back off the parallelization. Each system is different, so finding that sweet spot is difficult. All this said, it does sound like there's an infinite loop here, and it sounds like it's at the end of the import. If it's in the main loop, that would block completion of background import tasks, which also leads to what looks like a memory leak. -- Jim _______________________________________________ Shotwell mailing list [email protected] http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
