-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I've been debugging the python parser a bit more with some local
files a user referred me to for a medical reference guide. There are 26522
straight .html files, and I'm not doing anything exotic with them, and all
of these reference local files. --maxdepth is specified to only include
those local files. This process has currently been running on an extremely
fast/large memory box for 3 days, and it's only reached...
---- 6924 collected, 17293 to do ----
Processing file:/tmp/biam/biam/Spe105.html...
Retrieved ok.
Parsed ok.
---- 6925 collected, 17292 to do ----
Processing file:/tmp/biam/biam/Spe2586.html...
Retrieved ok.
Parsed ok.
..so far. I just attached gdb to the process, and found that it is
sitting on the following:
#0 0x080bbf10 in PyList_SetItem ()
#1 0x080bdab7 in PyList_AsTuple ()
#2 0x08078026 in PyEval_CallObjectWithKeywords ()
At this point, it parses 1 page every 5-6 hours or so. In the
beginning, it ripped through the first few thousand pages no problem, but as
the hours wore on, it slowed down considerably.
I just cancelled it, and will be installing oprofile and trace on
the box to see if I can get deeper into the reasons why this is happening.
The python process currently is using 995 megs of ram and 396 megs of swap,
but the processor load is well under 1%.
Once the new Plucker website is complete and launched (a week or two
more, rough estimate), I'm diving full-time back into the perl parser.
There's some places in perl with string and text and associative array
handling which may be a bit more robust than the current (2.1) python stuff.
Note: This is not a dig on the Python parser or the work Bill and Holger
have done in writing it. I just can't debug these problems because the
language is very unfamiliar to me.
Bill, can you point me to some places to peek/poke some code to try
to expose this a bit more?
d.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE8+7kmkRQERnB1rkoRApgYAJ9oZbIq5rTXNZfIFBWfSy34K7jizQCcCKqb
hbmoYUnichBsGWuWmIilECA=
=goks
-----END PGP SIGNATURE-----
