Re: [Qgis-developer] shapefile editing memory issue
Thanks for the quick response. Since my previous post, I have modified the code to eliminate all the calls to startEditing() and commitChanges(). I thought the memory problems might have been related to memory leaks in those functions. Now, I'm implementing changeAttributeValues() through the vector data provider to modify the attribute table directly without messing with the layer editing buffer, as recommended here: http://lists.osgeo.org/pipermail/qgis-developer/2011-July/015385.html However, some major memory leaks remain. As recommended in the previous post, I implemented my plug-in under valgrind on a Linux machine. Some output below highlights the changeAttributeValues() function as a possible culprit: 500,000 bytes 'definitely' lost with only a couple iterations of my algorithm. It's easy to see how gobs of memory can be lost after many thousand iterations. Any suggestions for a better way to update an attribute table repeatedly without losing so much memory? Thanks, Kelly ==31858== 576,500 (23,040 direct, 553,460 indirect) bytes in 360 blocks are definitely lost in loss record 14,582 of 14,586 ==31858==at 0x4A059DC: operator new(unsigned long) (vg_replace_malloc.c:220) ==31858==by 0x86CB51D: SHPReadOGRFeature(SHPInfo*, DBFInfo*, OGRFeatureDefn*, int, SHPObject*) (in /usr/lib64/libgdal.so.1.14.3) ==31858==by 0x86A99D4: OGRShapeLayer::GetFeature(long) (in /usr/lib64/libgdal.so.1.14.3) ==31858==by 0x19A72375: QgsOgrProvider::changeAttributeValues(QMapint, QMapint, QVariant const) (in /usr/lib64/qgis/libogrprovider.so) ==31858==by 0x1F002421: ??? (in /usr/lib64/python2.7/site-packages/qgis/core.so) ==31858==by 0x1BFE665A: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BFE804C: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BF6EC61: ??? (in /usr/lib64/libpython2.7.so.1.0) ==31858==by 0x1BF45FC2: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) -Original Message- From: Tim Sutton [mailto:li...@linfiniti.com] Sent: Thursday, July 12, 2012 12:59 PM To: Thorp, Kelly Cc: qgis-developer@lists.osgeo.org Subject: Re: [Qgis-developer] shapefile editing memory issue Hi On Thu, Jul 12, 2012 at 8:00 PM, Thorp, Kelly kelly.th...@ars.usda.gov wrote: Hello List: I am developing a QGIS plug-in for the purpose of running simulation models for unique land units, as represented by polygons in a polygon shapefile. Essentially, the plug-in uses spatial information in the polygon layer to write a simulation model input file. It then uses Python's 'subprocess' module to call the model executable and run the simulation. It then reads an output file generated by the model, sets the polygon shapefile to be editable, changes some attributes based on the model output, and then commits those changes. Then repeating this in a loop for each feature in the polygon shapefile. I've also included an optimization routine for the purpose of model calibration. This essentially means doing the above procedures repeatedly tens or hundreds of thousands of times. After each model simulation, the updated 'simulated' attribute values are compared with other attributes that contain 'observed' data. The optimization then repeatedly adjusts the model parameters until simulated and observed data are in best agreement. This process can take many hours or days, while I am repeatedly editing the shapefile and commiting the edits. However, I am not adding or removing any features or attributes. Problem is the memory slowly creeps up and if the optimization takes too long, I'll run out of memory and QGIS will crash. I notice that if my optimization routine does finish, I can completely shut down my plugin's dialog window, but the memory usage remains high. This makes me think the problem is not in my plug-in. Any ideas for me? You would probably need to run your plugin under a memory profiler like valgrind to see where the leaks are occurring. My apologies if that is too low level an answer. Regards Tim Thanks, Kelly ~ Kelly Robert Thorp, Ph.D., P.E. Research Agricultural Engineer USDA-ARS U.S. Arid Land Agricultural Research Center 21881 N Cardon Lane Maricopa, AZ 85138 520-316-6375 (Phone) 520-316-6330 (Fax) kelly.th...@ars.usda.gov ~ This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe
[Qgis-developer] shapefile editing memory issue
Hello List: I am developing a QGIS plug-in for the purpose of running simulation models for unique land units, as represented by polygons in a polygon shapefile. Essentially, the plug-in uses spatial information in the polygon layer to write a simulation model input file. It then uses Python's 'subprocess' module to call the model executable and run the simulation. It then reads an output file generated by the model, sets the polygon shapefile to be editable, changes some attributes based on the model output, and then commits those changes. Then repeating this in a loop for each feature in the polygon shapefile. I've also included an optimization routine for the purpose of model calibration. This essentially means doing the above procedures repeatedly tens or hundreds of thousands of times. After each model simulation, the updated 'simulated' attribute values are compared with other attributes that contain 'observed' data. The optimization then repeatedly adjusts the model parameters until simulated and observed data are in best agreement. This process can take many hours or days, while I am repeatedly editing the shapefile and commiting the edits. However, I am not adding or removing any features or attributes. Problem is the memory slowly creeps up and if the optimization takes too long, I'll run out of memory and QGIS will crash. I notice that if my optimization routine does finish, I can completely shut down my plugin's dialog window, but the memory usage remains high. This makes me think the problem is not in my plug-in. Any ideas for me? Thanks, Kelly ~ Kelly Robert Thorp, Ph.D., P.E. Research Agricultural Engineer USDA-ARS U.S. Arid Land Agricultural Research Center 21881 N Cardon Lane Maricopa, AZ 85138 520-316-6375 (Phone) 520-316-6330 (Fax) kelly.th...@ars.usda.gov ~ This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] shapefile editing memory issue
Hi On Thu, Jul 12, 2012 at 8:00 PM, Thorp, Kelly kelly.th...@ars.usda.gov wrote: Hello List: I am developing a QGIS plug-in for the purpose of running simulation models for unique land units, as represented by polygons in a polygon shapefile. Essentially, the plug-in uses spatial information in the polygon layer to write a simulation model input file. It then uses Python's 'subprocess' module to call the model executable and run the simulation. It then reads an output file generated by the model, sets the polygon shapefile to be editable, changes some attributes based on the model output, and then commits those changes. Then repeating this in a loop for each feature in the polygon shapefile. I've also included an optimization routine for the purpose of model calibration. This essentially means doing the above procedures repeatedly tens or hundreds of thousands of times. After each model simulation, the updated 'simulated' attribute values are compared with other attributes that contain 'observed' data. The optimization then repeatedly adjusts the model parameters until simulated and observed data are in best agreement. This process can take many hours or days, while I am repeatedly editing the shapefile and commiting the edits. However, I am not adding or removing any features or attributes. Problem is the memory slowly creeps up and if the optimization takes too long, I'll run out of memory and QGIS will crash. I notice that if my optimization routine does finish, I can completely shut down my plugin's dialog window, but the memory usage remains high. This makes me think the problem is not in my plug-in. Any ideas for me? You would probably need to run your plugin under a memory profiler like valgrind to see where the leaks are occurring. My apologies if that is too low level an answer. Regards Tim Thanks, Kelly ~ Kelly Robert Thorp, Ph.D., P.E. Research Agricultural Engineer USDA-ARS U.S. Arid Land Agricultural Research Center 21881 N Cardon Lane Maricopa, AZ 85138 520-316-6375 (Phone) 520-316-6330 (Fax) kelly.th...@ars.usda.gov ~ This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
