Hi there! I'm having a little trouble figuring out why my parsing / database application is consuming vast amounts of memory.
I'll preface this by saying that perhaps SA isn't the best solution for my problem. The app basically parses through hundreds of megabytes of text files looking for various regex patterns. When it finds what its looking for it creates a database object at one of three different levels in a heirarchical model. (Objects at level A with children at level B, level B has children at level C). All of these objects are linked with SA relations, with cascade and delete- orphan options. I understand why SA might not be the best solution. I'm creating tens of thousands of database objects, which should have a short in-memory half life and then my program never uses them again. I attempt to get rid of them by doing a cascaded expunge right after I do a session.flush. Right after that I do a delete. This is what happens to memory: Each loop (job) through the application consumes approximately 6MB which isnt being released to use on the next loop. You can see that after 50 jobs this might be an issue. Even worse, about 1MB out of the 6MB after each loop is never heard from again -- even after python exits. I can't even begin to imagine what might be causing this. I thought it might be the mysql database drivers, but it happens with my sqlite3 test database also. I've been searching all over the Internet to see if someone else has run into this problem, but have had relatively little luck. I did see articles discussing python memory deallocation problems, but nothing about failure to de-allocate after python exits. Even so, most of those problems were solved by Python 2.5 which is what I'm using. Let me also say that SA and Elixir are great projects and I really hope its something else that were doing, but I'm at a loss. I'd like to set my program up as a cron job, but honestly I can't put it into production until I nail this issue down. We can't run more than about 250-300 jobs through the system without being forced to reboot since were leaking memory like a sieve. Has anyone else run into these types of issues? Any help greatly appreciated. Thanks, Jim Burnes Boulder, CO --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
