Hi Massimo, Thanks for looking into this. I just ran a few tests and I can confirm that the leak does repro on Mac OS X El Capitan (10.11.6 (15G1004)) with the latest version of pyDAL (16.9) and the HEAD and latest tags for web2py repo (https://github.com/web2py/web2py/releases). The leak did not repro from the last pypi version of web2py (2.1.1). I ran the program I listed in PyCharm on Mac OSX and watched the memory usage using htop installed on the Mac using Homebrew.
Here are the detailed results: 1. pyDAL 16.9 from pydal import DAL def open_close_dal(): sql = DAL() sql.close() if __name__ == '__main__': while True: open_close_dal() Memory usage at ~4 mins: 193M (https://www.dropbox.com/s/vj5v6hqya4mot2v/Screenshot%202016-11-08%2011.43.28.png?dl=0) Memory usage at ~11 mins: 432M (https://www.dropbox.com/s/hac3monmy134b8f/Screenshot%202016-11-08%2011.50.51.png?dl=0) 2. web2py 2.1.1 (pip install web2py==2.1.1) from gluon import DAL def open_close_dal(): sql = DAL() sql.close() if __name__ == '__main__': while True: open_close_dal() Memory usage at ~5 mins: 16.7M (https://www.dropbox.com/s/yjy8vv7fy7i79b0/Screenshot%202016-11-08%2012.14.52.png?dl=0) Memory usage at ~11 mins: 16.7M (https://www.dropbox.com/s/49laoi3tk74qhvp/Screenshot%202016-11-08%2012.21.32.png?dl=0) 3. web2py cloned @latest tag from https://github.com/web2py/web2py.git and copied the gluon directory to my project in PyCharm. from gluon import DAL def open_close_dal(): sql = DAL() sql.close() if __name__ == '__main__': while True: open_close_dal() Memory usage at ~4 mins: 187M (https://www.dropbox.com/s/5hx01xve1mpemg5/Screenshot%202016-11-08%2012.32.54.png?dl=0) Memory usage at ~11 mins: 398M (https://www.dropbox.com/s/wcnpz15cor4g698/Screenshot%202016-11-08%2012.40.32.png?dl=0) Thanks, Sukhmeet On Monday, November 7, 2016 at 5:52:17 PM UTC-8, Massimo Di Pierro wrote: > > I run your code on a Mac and I was unable to reproduce the leak. I invite > others to try with different OSes and DAL version. It will help debug it. > > On Monday, 7 November 2016 19:48:35 UTC-6, Massimo Di Pierro wrote: >> >> Thank you for your help. We will investigate.... >> Any chance you can try if the leak exists with the DAL in web2py stable? >> >> On Friday, 4 November 2016 21:34:52 UTC-5, Sukhmeet Toor wrote: >>> >>> Hi web2py-users, >>> >>> I have an application that runs on CherryPy (v8.1.2) on Ubuntu (14.04.5 >>> LTS) and python 2.7.6. I use standalone pyDAL (v16.09) to connect to my >>> MySQL DB on AWS RDS. I'm seeing a constant memory leak on my servers which >>> causes the process to use up 100% of RAM in a few hours and eventual die a >>> painful death (no SWAP enabled; enabling SWAP pushes back certain death a >>> little). >>> >>> I have managed to reproduce the leak with a very small loop that >>> repeatedly opens and closes the DAL object. I confirmed that memory >>> continues to grow using htop and will eventually end up using 100% of RAM. >>> I thought it was an MySQL adapter specific problem but I was able to repro >>> it with the default sqlite adapter as well. >>> >>> from pydal import DAL >>> >>> def open_close_dal(): >>> sql = DAL() >>> sql.close() >>> >>> if __name__ == '__main__': >>> while True: >>> open_close_dal() >>> >>> >>> I read some really old threads on this issue that suggested calling >>> sql._adapter.close() or sql._adapter.close_all_instances() or del sql but >>> it doesn't seem to make a difference. I've tried tracking down the memory >>> leak and managed to confirm that it is the guts of the DAL that are >>> leaking. One iteration of the loop, with pympler.tracker enabled reports >>> these objects created: >>> >>> >>> types | # objects | total size >>> ============================================ | =========== | ============ >>> dict | 672 | 423.38 KB >>> list | 3525 | 354.35 KB >>> str | 2851 | 162.49 KB >>> <class 'collections.OrderedDict | 129 | 135.02 KB >>> code | 585 | 73.12 KB >>> <class 'pydal.dialects.MetaDialect | 32 | 28.25 KB >>> <class 'pydal.parsers.MetaParser | 17 | 15.01 KB >>> type | 16 | 14.12 KB >>> <class 'pydal.representers.MetaRepresenter | 16 | 14.12 KB >>> tuple | 168 | 12.85 KB >>> <class 'pydal.dialects.sqltype_for | 178 | 11.12 KB >>> weakref | 89 | 7.65 KB >>> int | 258 | 6.05 KB >>> <class 'pydal.representers.for_type | 49 | 3.06 KB >>> function (wrap) | 25 | 2.93 KB >>> >>> >>> Is this a known issue? I've been pulling out my hair with this for a few >>> days now and would love any insight into it. >>> >>> >>> Thanks in advance, >>> >>> Sukhmeet >>> >>> >>> p.s. Older threads on this topic, which no solution that worked for me: >>> >>> https://groups.google.com/forum/#!newtopic/web2py/web2py/hmsupVHdDHo >>> >>> http://stackoverflow.com/questions/5539722/standalone-web2py-dal-with-pyramid-requests-threads-and-memory >>> >>> >>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.