Hi all, We (@3liz.com) use QGIS Server since a long time to provide hosting services for our clients, with Lizmap Web Client (LWC) as the web map builder.
Having hundred LWC instances, we pay obviously attention to metrics, and monitor the CPU, memory, requests response time, etc. We sometimes encounter heavy memory consumption for specific projects, for example : https://github.com/qgis/QGIS/issues/43787 which put a lot of strain to the servers. It is often very hard to find the project(s) responsible for a high memory usage, in a production context, when many requests are run in parallel for many projects. I created a POC processing script to audit all the QGIS projects stored in a folder: https://gist.github.com/mdouchin/cd89a259d3560635d4bb8708ad430caa/ which helps a bit to gather some key figures, such as the number of layers, the sum of the images used in map layouts, etc. This can help, but often the memory consumption is not related to the number of layers, and we use a "find a needle in the haystack" approach to find the problematic project(s). For example, I spent once a whole day finding that a too big JPEG used in a print layout causes the heavy memory usage: * look at the memory curve, * find the moment when the memory increased a lot * use the command line find to get which QGIS project files has changed during this period (for example because it has been uploaded by the user) * look inside QGIS logs to find which project has been loaded during this period, and compare * if you are lucky, there are not so many projects involved... * download the project, open it in QGIS Desktop to observe and confirm the memory issue * try to understand which causes the issue * ... We are also building a REST API over py-qgis-server ( https://github.com/3liz/py-qgis-server ) which will allow to get information on the loaded projects, to ease the auditing process in real-time. (It will be opensource). In the processing script, and in this API tool, we still miss a very important metric: how much each project consumes the system memory (RAM). I am not sure it could be easily done, but it will be very handy to: * find heavy memory consumer projects * see how the memory evolves * compare the memory usage for each QGIS versions (with the same project and layers) * help to find the possible memory leaks This message is not a feature request, I would like to get your experience and opinion regarding this memory usage issue. Any contribution welcome ;-) Thanks in advance Michaël
_______________________________________________ QGIS-Developer mailing list [email protected] List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
