Jason Tackaberry wrote: > On Sat, 2006-07-15 at 20:33 +0200, Dirk Meyer wrote: >> scanning my whole media dir (2500 files, most of them images) and >> creating thumbnails, the main process is 30MB and the thumbnailer >> 20MB. Were does the memory go? > > Could also be all those cyclic refs you've got in beacon. :) What > happens if you run gc.collect() periodically and then check memory > usage? Does collect() return any unreachable objects?
beacon runs gc.collect() in a timer, no unreachable objects.
>> 5. I looked in kaa.db and found something:
>> | self._cursor.execute("PRAGMA cache_size=50000")
>> Looking at sqlite doc:
>> | PRAGMA cache_size;
>> | PRAGMA cache_size = Number-of-pages;
>> |
>> | Query or change the maximum number of database disk pages that
>> | SQLite will hold in memory at once. Each page uses about 1.5K of
>> | memory. The default cache size is 2000. If you are doing UPDATEs
>> | or DELETEs that change many rows of a database and you do not
>> | mind if SQLite uses more memory, you can increase the cache size
>> | for a possible speed improvement.
>> 50000 * 1.5k == 75MB. Tack, are you kidding me? Don't you think it
>> is one zero too much?
>
> Ah, I didn't read this email fully before replying before. Actually
> this is somewhat leftover from when I was benchmarking kaa.db. I wanted
> to make sure all indexes would be in memory, since if they weren't it
> would taint my benchmarks. Cache size is definitely something we should
> expose to the user as a pref, but 75MB is obviously too much. Maybe a
> value like 10000.
I vote for half of it. 7.5 MB sounds enough for me and it is more than
twice as much as sqlite has as default.
>> 6. I reduced the db cache to 50 (==150kb) and removed all the scanning
>> and mtime check code in the crawler (keeping the old db file). I
>> still see memory growing. And this doesn't explain the problem in
>> the thumbnailer.
>
> Is there a way to stress test the thumbnailer standalone, outside of
> beacon? This will help troubleshooting.
Well, I tried to reproduce it with the thumbnailer alone, I did not
find a problem. But no, there is no real stress test.
>> OK, that's it for today, I'm sick of hunting these bugs. Maybe someone
>> else can take a look at it or knows better ways to debug python code.
>
> I can help out with this problem this week.
That would be great, a fresh pair of eyes it what I need here.
Dischi
--
A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1...
pgphNUygyV6KW.pgp
Description: PGP signature
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________ Freevo-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/freevo-devel
