Re: [sqlalchemy] Memory management in sqlalchemy

2020-03-04 Thread Mike Bayer
how are you measuring memory ? is that a Python memory profiler? note that the Python interpreter on the outside is not very consistent about returning memory to the OS. if this is within Python then you'd just have a garbage collection issue and you'd want to look at GC debugging. as always, th

[sqlalchemy] Memory management in sqlalchemy

2020-03-04 Thread Adrien Blin
Hello, I'm having troubles understanding how to deal with memory management with sqlalchemy. Here is my issue. I have a big script requesting data from a PostgreSQL DB. I process this data, and insert the generated data in my DB. On the beggining of the process, I request an object from my DB

Re: [sqlalchemy] SQLAlchemy Memory Leak

2020-03-04 Thread Mike Bayer
produce a script that reproduces the same results and share it here. this means writing a small script that looks like the same thing you are doing, but without your whole environment. if it does not leak memory, then you have to slowly change one element at a time until it looks more and more l

Re: [sqlalchemy] No handlers could be found for logger "sqlalchemy.pool.impl.QueuePool"

2020-03-04 Thread Mike Bayer
the pool might be trying to emit an info message, not really sure, there is some complex process by which Python logging either auto-configures itself, or doesnt. Since Alembic has the env.py call upon fileConfig(), that would be the logging config here. Add a logging config for it to your alemb

Re: [sqlalchemy] SQLAlchemy Memory Leak

2020-03-04 Thread Ahmed Cheikh
I tried the obj.expire(obj) and it did unfortunately not change anything. I still have accumulating used memory for whatever reason. When I did profile the code I saw that sometime engine objects use memory and sometimes session. Do any of you guys have any other suggestions Le jeudi 27 févri