Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-05 Thread Mike Bayer
On Thu, Mar 5, 2020, at 10:17 AM, Adrien Blin wrote: > I tried strong referencing the objects stored in the session using : > > *def *strong_reference_session(session): > @event.listens_for(session, "pending_to_persistent") > @event.listens_for(session, "deleted_to_persistent") > @ev

Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-05 Thread Adrien Blin
I tried strong referencing the objects stored in the session using : def strong_reference_session(session): @event.listens_for(session, "pending_to_persistent") @event.listens_for(session, "deleted_to_persistent") @event.listens_for(session, "detached_to_persistent") @event.listens

Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-05 Thread Mike Bayer
On Thu, Mar 5, 2020, at 5:17 AM, Adrien Blin wrote: > > > I'll try to recreate the issue for you to test it, but my question really is > : Is there something I didn't get about memory usage in sqlalchemy, meaning, > is it normal for it to retain objects in memory, and if so, how can we get >

Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-05 Thread Adrien Blin
I am using memory_profiler (https://pypi.org/project/memory-profiler/) to measure the memory usage. I reduced the code to just the request to see ifthe issue only comes from here : import gc import sqlalchemy.orm.session as s from MyDatabase.model.Table import Table from memory_profiler impor

Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-04 Thread Mike Bayer
how are you measuring memory ? is that a Python memory profiler? note that the Python interpreter on the outside is not very consistent about returning memory to the OS. if this is within Python then you'd just have a garbage collection issue and you'd want to look at GC debugging. as always, th

[sqlalchemy] Memory management in sqlalchemy

2020-03-04 Thread Adrien Blin
Hello, I'm having troubles understanding how to deal with memory management with sqlalchemy. Here is my issue. I have a big script requesting data from a PostgreSQL DB. I process this data, and insert the generated data in my DB. On the beggining of the process, I request an object from my DB