[sqlalchemy] pylons SQLAlchemy memory
And now for a question about a completely different app (no sharding, very simple). I haven't got a sufficient response from the pylons group, so I'm trying here. The question: http://groups.google.com/group/pylons-discuss/browse_thread/thread/cb48d0ea2b084159 Things I've tried/considered: --- mysql utf8 memory issues --- different mapping approaches (base declarative, reflective, etc) --- various debug options and non-production standard settings --- beaker (although I'm not sure that I successfully turned it all off, I'll keep trying) --- indexes --- added tests for old-school classes (ones that don't extend object and aren't garbage collected) --- I've played with dozer (added to middleware.py, didn't find much -- old-school classed, maybe?) You wouldn't know it from my deluge of recent questions, but in all my years at this I've only asked these three threads worth of questions, plus an X11 one years and years ago on the the OpenBSD group. Anyway, I do apologize for having to ask so much from you and this group -- perhaps I'll be able to start giving back soon by answering some of the questions. Sorry, --diana -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] pylons SQLAlchemy memory
diana wrote: And now for a question about a completely different app (no sharding, very simple). I haven't got a sufficient response from the pylons group, so I'm trying here. The question: http://groups.google.com/group/pylons-discuss/browse_thread/thread/cb48d0ea2b084159 Things I've tried/considered: --- mysql utf8 memory issues --- different mapping approaches (base declarative, reflective, etc) --- various debug options and non-production standard settings --- beaker (although I'm not sure that I successfully turned it all off, I'll keep trying) --- indexes --- added tests for old-school classes (ones that don't extend object and aren't garbage collected) --- I've played with dozer (added to middleware.py, didn't find much -- old-school classed, maybe?) You wouldn't know it from my deluge of recent questions, but in all my years at this I've only asked these three threads worth of questions, plus an X11 one years and years ago on the the OpenBSD group. Anyway, I do apologize for having to ask so much from you and this group -- perhaps I'll be able to start giving back soon by answering some of the questions. the Python VM doesn't shrink very much in size once it's grown. GC can let go of everything but the VM size is there for good. Your query of 90K rows is growing it because MySQLdb fully buffers result sets before making them available. this is a MySQLdb limitation. Sorry, --diana -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] pylons SQLAlchemy memory
Le lundi 11 janvier 2010 à 15:55 -0800, diana a écrit : And now for a question about a completely different app (no sharding, very simple). I haven't got a sufficient response from the pylons group, so I'm trying here. The question: http://groups.google.com/group/pylons-discuss/browse_thread/thread/cb48d0ea2b084159 Well if you only want to count entries, use Query.count(), not Query.all(). It will be much more efficient, both on the DB side and on the Python side. Even if you use the entries one by one but don't need to keep them in memory afterwards, just use the iterative form (`for row in query: ...`). Regardless, you are fetching 9 objects and witnessing a 160MB increase in memory. This gives approximately 1.7KB per objects. Depending on the size and complexity of each row this is not necessarily surprising. Python will generally not be as memory-efficient as hand-tailored structures written in C, since there is a lot of genericity and flexibility in most Python datatypes. Objects in general can be quite big, because they are based on dictionaries (dict objects) which are themselves big. As for releasing memory, try to call gc.collect() after you have released the last reference to the result set. I'd be a bit surprised if SQLAlchemy created reference cycles, though -- and would be inclined to consider it a bug ;-) -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] pylons SQLAlchemy memory
On Mon, Jan 11, 2010 at 7:07 PM, Antoine Pitrou solip...@pitrou.net wrote: Le lundi 11 janvier 2010 à 15:55 -0800, diana a écrit : Well if you only want to count entries, use Query.count(), not Query.all(). Yup, I don't actually do this in a real app. I was just doing this (in a hello world app) as an exercise to illustrate a point, and to better understand pylons (and, as I'm just learning, the Python VM). Thanks Antoine, --diana -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] pylons SQLAlchemy memory
we have a full set of tests that ensure SQLA itself has no unreleased memory issues or excessive cycles and they've been in our trunk for several years, and we also nailed a few remaining corner cases over the past year which correspond to highly unusual usage patterns, so I'm very confident that there's no issues within SQLA itself. On Jan 11, 2010, at 7:17 PM, Diana Clarke wrote: On Mon, Jan 11, 2010 at 7:07 PM, Antoine Pitrou solip...@pitrou.net wrote: Le lundi 11 janvier 2010 à 15:55 -0800, diana a écrit : Well if you only want to count entries, use Query.count(), not Query.all(). Yup, I don't actually do this in a real app. I was just doing this (in a hello world app) as an exercise to illustrate a point, and to better understand pylons (and, as I'm just learning, the Python VM). Thanks Antoine, --diana -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] pylons SQLAlchemy memory
I never really suspected that this was a SQLAlchemy issue, which was why I didn't originally post this question to the SQLAlchemy group. I apologize if it came across that way. Time for me to do my python VM homework... My apologies, --diana On Mon, Jan 11, 2010 at 8:15 PM, Michael Bayer mike...@zzzcomputing.com wrote: we have a full set of tests that ensure SQLA itself has no unreleased memory issues or excessive cycles and they've been in our trunk for several years, and we also nailed a few remaining corner cases over the past year which correspond to highly unusual usage patterns, so I'm very confident that there's no issues within SQLA itself. On Jan 11, 2010, at 7:17 PM, Diana Clarke wrote: On Mon, Jan 11, 2010 at 7:07 PM, Antoine Pitrou solip...@pitrou.net wrote: Le lundi 11 janvier 2010 à 15:55 -0800, diana a écrit : Well if you only want to count entries, use Query.count(), not Query.all(). Yup, I don't actually do this in a real app. I was just doing this (in a hello world app) as an exercise to illustrate a point, and to better understand pylons (and, as I'm just learning, the Python VM). Thanks Antoine, --diana -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
