Re: [Qgis-developer] memory layer performance while adding many features?
Hi Denis, Adding all features at once, or 1000 by 1000, speeds up enormously. It only takes a few minutes, at a steady speed of 480 fpm. Where adding the same dataset 1 by 1 took about 5 hours. Thanks for pointing this out! Raymond On 30-07-15 09:34, Denis Rouzaud wrote: From your code [0] , I would see 2 options: * add features all at once on the provider using QgsVectorDataProvider::addFeatures (the method you currently use) * work on the layer level rather than the provider, which I believe is the recommended way for plugins. Make the layer editable, and add features to the edit buffer and commit at once. Or maybe commit 1000 by 1000. I hope this helps. [0] https://github.com/opengeogroep/AERIUS-QGIS-plugins/blob/master/ImaerReader/imaer_reader.py#L228 On 07/30/2015 09:20 AM, Raymond Nijssen wrote: Hi Denis, The code is in the ImaerReader plugin in the qgis repo and in github: https://github.com/opengeogroep/AERIUS-QGIS-plugins I'm adding the features one by one. Regards, Raymond On 30-07-15 08:46, Denis Rouzaud wrote: Hi Raymond, Can you show us the code doing this? Do you commit features one by one ar all at once? Best wishes, Denis On 07/29/2015 09:04 AM, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond -- Terglobo Ampèrestraat 110 5223 CT 's-Hertogenbosch 06-25314983 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
Hi all, in general I think we should advice to use the layer.addFeature() method with a layer.commitChanges() in the end. It has the advantage that it does batch updates on the provider automatically and proper signals are sent so the UI (canvas, attribute table...) can reflect changes immediately. The layer.dataProvider().addFeatures() method delegates all this to the developer. Cheers, Matthias On 08/05/2015 08:28 PM, Raymond Nijssen wrote: Hi Denis, Adding all features at once, or 1000 by 1000, speeds up enormously. It only takes a few minutes, at a steady speed of 480 fpm. Where adding the same dataset 1 by 1 took about 5 hours. Thanks for pointing this out! Raymond On 30-07-15 09:34, Denis Rouzaud wrote: From your code [0] , I would see 2 options: * add features all at once on the provider using QgsVectorDataProvider::addFeatures (the method you currently use) * work on the layer level rather than the provider, which I believe is the recommended way for plugins. Make the layer editable, and add features to the edit buffer and commit at once. Or maybe commit 1000 by 1000. I hope this helps. [0] https://github.com/opengeogroep/AERIUS-QGIS-plugins/blob/master/ImaerReader/imaer_reader.py#L228 On 07/30/2015 09:20 AM, Raymond Nijssen wrote: Hi Denis, The code is in the ImaerReader plugin in the qgis repo and in github: https://github.com/opengeogroep/AERIUS-QGIS-plugins I'm adding the features one by one. Regards, Raymond On 30-07-15 08:46, Denis Rouzaud wrote: Hi Raymond, Can you show us the code doing this? Do you commit features one by one ar all at once? Best wishes, Denis On 07/29/2015 09:04 AM, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond signature.asc Description: OpenPGP digital signature ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
Merci Denis, I will try both options later today. Raymond On 30-07-15 09:34, Denis Rouzaud wrote: From your code [0] , I would see 2 options: * add features all at once on the provider using QgsVectorDataProvider::addFeatures (the method you currently use) * work on the layer level rather than the provider, which I believe is the recommended way for plugins. Make the layer editable, and add features to the edit buffer and commit at once. Or maybe commit 1000 by 1000. I hope this helps. [0] https://github.com/opengeogroep/AERIUS-QGIS-plugins/blob/master/ImaerReader/imaer_reader.py#L228 On 07/30/2015 09:20 AM, Raymond Nijssen wrote: Hi Denis, The code is in the ImaerReader plugin in the qgis repo and in github: https://github.com/opengeogroep/AERIUS-QGIS-plugins I'm adding the features one by one. Regards, Raymond On 30-07-15 08:46, Denis Rouzaud wrote: Hi Raymond, Can you show us the code doing this? Do you commit features one by one ar all at once? Best wishes, Denis On 07/29/2015 09:04 AM, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond -- Terglobo Ampèrestraat 110 5223 CT 's-Hertogenbosch 06-25314983 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
Hi Raymond, Can you show us the code doing this? Do you commit features one by one ar all at once? Best wishes, Denis On 07/29/2015 09:04 AM, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
Hi Raymond, On 29/07/2015 09:04, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? If I am correct the spatial index is created only if you ask so (by passing index=yes). Do you have the same results without spatial index ? ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
Hi Hugo, I'm not specifically adding an index. So if you are right, I'm not using that. Regards, Raymond On 30-07-15 09:10, Hugo Mercier wrote: Hi Raymond, On 29/07/2015 09:04, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? If I am correct the spatial index is created only if you ask so (by passing index=yes). Do you have the same results without spatial index ? ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Terglobo Ampèrestraat 110 5223 CT 's-Hertogenbosch 06-25314983 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
From your code [0] , I would see 2 options: * add features all at once on the provider using QgsVectorDataProvider::addFeatures (the method you currently use) * work on the layer level rather than the provider, which I believe is the recommended way for plugins. Make the layer editable, and add features to the edit buffer and commit at once. Or maybe commit 1000 by 1000. I hope this helps. [0] https://github.com/opengeogroep/AERIUS-QGIS-plugins/blob/master/ImaerReader/imaer_reader.py#L228 On 07/30/2015 09:20 AM, Raymond Nijssen wrote: Hi Denis, The code is in the ImaerReader plugin in the qgis repo and in github: https://github.com/opengeogroep/AERIUS-QGIS-plugins I'm adding the features one by one. Regards, Raymond On 30-07-15 08:46, Denis Rouzaud wrote: Hi Raymond, Can you show us the code doing this? Do you commit features one by one ar all at once? Best wishes, Denis On 07/29/2015 09:04 AM, Raymond Nijssen wrote: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] memory layer performance while adding many features?
No idea why that might be happeningbut hats off to you for such a well documented report. I had to say it :-) 2015-07-29 9:04 GMT+02:00 Raymond Nijssen r.nijs...@terglobo.nl: Dear developers, ;) A plugin of mine imports data from a gml file into a memory layer. Works fine for tiny gml files, but takes forever on huge ones. Sounds plausible maybe, but the relation is not linear. So I did did some tests and found out that reading and parsing the gml and creating the features always goes at the same speed of about 400 features a second (fps). But when adding these to my memory layer, the process slows down tremendously. I output the fps for every 1000 features and it gave me this graph: http://terglobo.nl/downloads/memory-layer-performance.png At the end of the importing process I'm processing just 3 features per second. Is this expected behaviour? Is a memory layer not meant for something like this? Or is this a bug? In the source code I found that for every feature I add to a memory layer the index is updated. Could that be inefficient? Hoping anyone can explain! Regards, Raymond -- Terglobo Ampèrestraat 110 5223 CT 's-Hertogenbosch The Netherlands +31 6 25 31 49 83 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
