how are you measuring memory ? is that a Python memory profiler? note that the Python interpreter on the outside is not very consistent about returning memory to the OS. if this is within Python then you'd just have a garbage collection issue and you'd want to look at GC debugging. as always, the process is to create a script that looks similar to what you have, show that it does not have the memory issue, then incrementally add elements to it from your memory-leaking script until it does. a self-contained script which illustrates the issue can be shared here.
On Wed, Mar 4, 2020, at 12:01 PM, Adrien Blin wrote: > Hello, > > I'm having troubles understanding how to deal with memory management with > sqlalchemy. > > Here is my issue. I have a big script requesting data from a PostgreSQL DB. I > process this data, and insert the generated data in my DB. > > On the beggining of the process, I request an object from my DB : > > Line # Mem usage Increment Line Contents > ================================================ > 223 144.2 MiB 144.2 MiB @profile(stream=fp) > 224 def compute(): > 226 155.4 MiB 11.2 MiB obj= > session.query(Table).filter(Table.idTable == some_id).scalar() > > As you can see, this object takes roughly 11Mb in my RAM. At the end of my > script, I try to free the memory from this object, but nothing seems to work > (other requests are done using the session.query() or engine.execute(), the > issue is the same for them. This example is the most significant). > > 589 161.4 MiB 0.0 MiB session.flush(obj) > 590 161.4 MiB 0.0 MiB session.expunge_all() > 591 161.4 MiB 0.0 MiB session.close() > 592 161.4 MiB 0.0 MiB s.close_all_sessions() > 593 161.4 MiB 0.0 MiB obj= None > 594 161.4 MiB 0.0 MiB gc.collect() > > > I call this function several thousand times in the process, which makes the > RAM increase infinitely until it breaks. > I also tried expiring the object through the session, deleting its reference > and garbage collecting it. > > How could I request this object, extract the information I need from it, and > delete it from my memory, so that my process won't run out of memory ? > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sqlalchemy/94afa0c9-7b4b-432c-8c8a-709ae366522f%40googlegroups.com > > <https://groups.google.com/d/msgid/sqlalchemy/94afa0c9-7b4b-432c-8c8a-709ae366522f%40googlegroups.com?utm_medium=email&utm_source=footer>. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/5382a28c-2481-47c0-ad3d-794056ad92a0%40www.fastmail.com.
