Hi, maybe someone (Tack?) can help me out here, maybe I'm blind. I tried for hours to find some bugs in beacon and I don't see them.
We have two processes, the main server and the thumbnail/hwmon
process. When beacon is crawling and you press C-c one of the
following happens:
1. beacon stops. Doesn't happen most of the time
2. gtk is raising some error. Looks like the second process gets my
keyboard events and the gtk main loop (used by the hal code in
hwmon) doesn't react on this as it should.
3. Most of the time, main stops and waits for the child which never
dies. It could be because the child is much slower when processing
images and did not get the shutdown() rpc. But a os.kill also
doesn't help. In that case, the child dies and both processes print
a last debug message. But I still have one process left (main)
which is at the end of script, doing nothing anymore. Well, nothing
is not correct, it prints
Exception exceptions.TypeError: 'unsubscriptable object' in <bound method
WeakSocketDispatcher._weakref_destroyed of <WeakSocketDispatcher for
_handle_data>> ignored
and then it just hangs and waits for a second C-c. First of all,
which _handle_data are we talking about here? And why does it
block?
Besides the problem above, we have a memory problem. On startup main
uses 13MB and thumbnail uses 17MB (I guess because of gtk). After
scanning my whole media dir (2500 files, most of them images) and
creating thumbnails, the main process is 30MB and the thumbnailer
20MB. Were does the memory go?
1. I added a counter to Item with +1 on __init__ and -1 on __del__. At
the end most items are gone. we only have a small number left in
the crawler (the starting points) and in the media (the root
dir). No bug here.
2. I did the same with Callback and Signal in notifier. The number of
these objects in constant. No bug here.
3. Inotify has constant references, but removing the inotify code did
not help. No bug here.
4. I did some extreme tests with kaa.rpc, no memory leak.
5. I looked in kaa.db and found something:
| self._cursor.execute("PRAGMA cache_size=50000")
Looking at sqlite doc:
| PRAGMA cache_size;
| PRAGMA cache_size = Number-of-pages;
|
| Query or change the maximum number of database disk pages that
| SQLite will hold in memory at once. Each page uses about 1.5K of
| memory. The default cache size is 2000. If you are doing UPDATEs
| or DELETEs that change many rows of a database and you do not
| mind if SQLite uses more memory, you can increase the cache size
| for a possible speed improvement.
50000 * 1.5k == 75MB. Tack, are you kidding me? Don't you think it
is one zero too much?
6. I reduced the db cache to 50 (==150kb) and removed all the scanning
and mtime check code in the crawler (keeping the old db file). I
still see memory growing. And this doesn't explain the problem in
the thumbnailer.
OK, that's it for today, I'm sick of hunting these bugs. Maybe someone
else can take a look at it or knows better ways to debug python code.
Dischi
--
Murphy's Laws:
(1) If anything can go wrong, it will.
(2) Nothing is as easy as it looks.
(3) Everything takes longer than you think it will.
pgpWAMH4Gk2s4.pgp
Description: PGP signature
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________ Freevo-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/freevo-devel
