subject:"\[Qgis\-user\] Qgis low CPU and Memory load"

Re: [Qgis-user] Qgis low CPU and Memory load

2021-04-19 Thread Matt Boyd

I use multiproc / proc to speed things up in python instead of a for
loop, each step needs to be independent of the other steps for this to
work.
On an i7 with 4 cores I can process 8 layers at a time and hit 100% on
all of them. I haven't used this inside qgis yet but for the
relatively simple work I'm doing it works well.

Matt

On Sat, Apr 17, 2021 at 2:12 AM Wolfgang Meinolf
 wrote:
>
> Thanks for the excellent feedback and hints. First of all, my QGIS qork is 
> mainly with vector layers. Therefore unfortunately the OpenCL capabilities 
> don’t help. To my understanding this is mainly for accelerating heavy raster 
> calculations. Secondly and as already mentioned, my most urgent need is to 
> accelerate the execution of GRASS v.net.steiner and v.clean as part of my 
> toolbox processing scripts for a number of given route networks in separate 
> areas.
>
>
>
> However, on you feedback I searched a bit and found, that obviously  since 
> QGIS3 some parallelization seems to be supported. In the developers coockbook 
> here: 
> https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/tasks.html#task-from-a-processing-algorithm
>   I found some promising info including the link to here: 
> https://www.opengis.ch/2018/06/22/threads-in-pyqgis3/. I will spend some time 
> to modify my loop through my separate network areas to try, to start python 
> jobs from inside the loop, which are then collected at the end.
>
>
>
> Thanks again for your good ideas and have a good weekend
>
> Br
>
> Wolfgang
>
> -
>
>
>
> Hi,
>
>
>
> You can also activate the OpenCL option.  Some algorithms have been optimized 
> (I believe it’s still experimental).  If you have a compatible  OpenCL video 
> card, it will run these on the card.  If not, it can probably run on 
> compatible CPUs.
>
>
>
> https://www.itopen.it/opencl-acceleration-now-available-in-qgis/
>
>
>
> I find it’s worth testing equivalent algorithms from different sources in the 
> Processing plugin is worth the trouble.  Some Saga algorithm are faster.  
> Merging vector lines, for example, could take a long time using the QGIS 
> algorithm (talking hours) and was much faster (a few minutes) using the Saga 
> version of the tool.  I also found that some algorithms are much faster to 
> handle when the files are not open beforehand in the layer manager.  For 
> example, the Processing plugin can sometimes grab the needed files directly 
> on the hard drive rather than picking one that is already open in the layer 
> manager. Looking at the processor, memory and disk usage can also help 
> identify bottle necks in the hardware.
>
> Nicolas Cadieux
>
> https://gitlab.com/njacadieux
>
>
>
> Le 16 avr. 2021 à 03:34, Francesco Pelullo  a écrit :
>
> 
>
>
>
> Il ven 16 apr 2021, 02:56 Stewart Holt  ha scritto:
>
>  Parallelizing script execution is complex and I doubt that it is currently 
> done in QGIS.
>
>
>
> That's true.
>
>
>
> In Settings / Options / Rendering / there's a checkbox for render layers in 
> parallel using as many CPU cores you want, but It Is enabled by default so I 
> suppose that there are no others settings that could make QGIS more 
> aggressive in CPU usage.
>
>
>
> Regards
>
>
>
>
>
>
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Qgis low CPU and Memory load

2021-04-16 Thread Wolfgang Meinolf

Thanks for the excellent feedback and hints. First of all, my QGIS qork is
mainly with vector layers. Therefore unfortunately the OpenCL capabilities
don’t help. To my understanding this is mainly for accelerating heavy raster
calculations. Secondly and as already mentioned, my most urgent need is to
accelerate the execution of GRASS v.net.steiner and v.clean as part of my
toolbox processing scripts for a number of given route networks in separate
areas.

However, on you feedback I searched a bit and found, that obviously since
QGIS3 some parallelization seems to be supported. In the developers coockbook
here:
https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/tasks.html#task-from-a-processing-algorithm
I found some promising info including the link to here:
https://www.opengis.ch/2018/06/22/threads-in-pyqgis3/. I will spend some time
to modify my loop through my separate network areas to try, to start python
jobs from inside the loop, which are then collected at the end.

Thanks again for your good ideas and have a good weekend
Br
Wolfgang
-

Hi,

You can also activate the OpenCL option. Some algorithms have been optimized
(I believe it’s still experimental). If you have a compatible OpenCL video
card, it will run these on the card. If not, it can probably run on compatible
CPUs.

https://www.itopen.it/opencl-acceleration-now-available-in-qgis/

I find it’s worth testing equivalent algorithms from different sources in the
Processing plugin is worth the trouble. Some Saga algorithm are faster.
Merging vector lines, for example, could take a long time using the QGIS
algorithm (talking hours) and was much faster (a few minutes) using the Saga
version of the tool. I also found that some algorithms are much faster to
handle when the files are not open beforehand in the layer manager. For
example, the Processing plugin can sometimes grab the needed files directly on
the hard drive rather than picking one that is already open in the layer
manager. Looking at the processor, memory and disk usage can also help identify
bottle necks in the hardware.
Nicolas Cadieux
https://gitlab.com/njacadieux

Le 16 avr. 2021 à 03:34, Francesco Pelullo
mailto:f.pelu...@gmail.com>> a écrit :

Il ven 16 apr 2021, 02:56 Stewart Holt
mailto:stewartbh...@gmail.com>> ha scritto:
Parallelizing script execution is complex and I doubt that it is currently
done in QGIS.

That's true.

In Settings / Options / Rendering / there's a checkbox for render layers in
parallel using as many CPU cores you want, but It Is enabled by default so I
suppose that there are no others settings that could make QGIS more aggressive
in CPU usage.

Regards

___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

smime.p7s
Description: S/MIME cryptographic signature
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Qgis low CPU and Memory load

2021-04-16 Thread Nicolas Cadieux

Hi,

You can also activate the OpenCL option.  Some algorithms have been optimized 
(I believe it’s still experimental).  If you have a compatible  OpenCL video 
card, it will run these on the card.  If not, it can probably run on compatible 
CPUs. 

https://www.itopen.it/opencl-acceleration-now-available-in-qgis/

I find it’s worth testing equivalent algorithms from different sources in the 
Processing plugin is worth the trouble.  Some Saga algorithm are faster.  
Merging vector lines, for example, could take a long time using the QGIS 
algorithm (talking hours) and was much faster (a few minutes) using the Saga 
version of the tool.  I also found that some algorithms are much faster to 
handle when the files are not open beforehand in the layer manager.  For 
example, the Processing plugin can sometimes grab the needed files directly on 
the hard drive rather than picking one that is already open in the layer 
manager. Looking at the processor, memory and disk usage can also help identify 
bottle necks in the hardware.  

Nicolas Cadieux
https://gitlab.com/njacadieux

> Le 16 avr. 2021 à 03:34, Francesco Pelullo  a écrit :
> 
> 
> 
> 
> Il ven 16 apr 2021, 02:56 Stewart Holt  ha scritto:
>>  Parallelizing script execution is complex and I doubt that it is currently 
>> done in QGIS.
> 
> 
> That's true.
> 
> In Settings / Options / Rendering / there's a checkbox for render layers in 
> parallel using as many CPU cores you want, but It Is enabled by default so I 
> suppose that there are no others settings that could make QGIS more 
> aggressive in CPU usage.
> 
> 
> Regards
> 
> 
> 
>> 
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Qgis low CPU and Memory load

2021-04-16 Thread Francesco Pelullo

Il ven 16 apr 2021, 02:56 Stewart Holt  ha scritto:

>  Parallelizing script execution is complex and I doubt that it is
> currently done in QGIS.
>

That's true.

In Settings / Options / Rendering / there's a checkbox for render layers in
parallel using as many CPU cores you want, but It Is enabled by default so
I suppose that there are no others settings that could make QGIS more
aggressive in CPU usage.

>
Regards



>
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Qgis low CPU and Memory load

2021-04-15 Thread Stewart Holt

Your I-7 likely has 4 or more logical processors. The only way to go over
100/N where N is the number of processors is to have 2 or more things
(continuously) running simultaneously (like loading a complex web page in a
browser). Scripts are going to run as a single stream which will bounce
around between processors, giving an average user of 100/N% If you have 4
processors, 25% would be from the script and another 5% handling disk I/O,
network, and other Windows tasks. That would result in the 30% you see.
Parallelizing script execution is complex and I doubt that it is currently
done in QGIS.

On Thu, Apr 15, 2021 at 1:03 PM Wolfgang Meinolf <
wmein...@bbv-deutschland.de> wrote:

> Dear Community,
>
> I use Qgis quite a while under win10 and ubuntu (via Oracle VM Box). My
> FTTH scripts include GRASS i.e. v.net.steiner algorithm, which are quite
> CPU and memory intensive. However I never observed more than 30% CPU and
> never more than appx 4GB mem usage. My Hardware is Intel I7, 32GB Ram and
> 1000TB SSD. At same time I don’t observe any other activities i.e. on
> network. I installed OsGEO4W the 64bit Version. I’. currently on 3.18.1
>
>
>
> I have however the impression to use a 32bit version.
>
>
>
> I would appreciate any hint, how I can increase the CPU and MEM Load, to
> accelerate my scripts, which run usually quite a while. What do I wrong?
>
>
>
> Thanks in advance and best regards from Bavaria.
>
> Wolfgang
>
> 
>
>
>
> QGIS-Version
>
> 3.18.1-Zürich
>
> QGIS-Codeversion
>
> 202f1bf7e5 
>
> Kompiliert gegen Qt
>
> 5.11.2
>
> Laufendes Qt
>
> 5.11.2
>
> Kompiliert mit GDAL/OGR
>
> 3.1.4
>
> Läuft mit GDAL/OGR
>
> 3.1.4
>
> Kompiliert mit GEOS
>
> 3.8.1-CAPI-1.13.3
>
> Läuft mit GEOS
>
> 3.8.1-CAPI-1.13.3
>
> Kompiliert mit SQLite
>
> 3.29.0
>
> Läuft mit SQLite
>
> 3.29.0
>
> PostgreSQL-Client-Version
>
> 11.5
>
> SpatiaLite-Version
>
> 4.3.0
>
> QWT-Version
>
> 6.1.3
>
> QScintilla2-Version
>
> 2.10.8
>
> Kompiliert mit PROJ
>
> 6.3.2
>
> Läuft mit PROJ
>
> Rel. 6.3.2, May 1st, 2020
>
> BS-Version
>
> Windows 10 (10.0)
>
> Aktive Python-Erweiterungen
>
> alkisplugin;
> closest_points;
> disconnected-islands;
> ImportPhotos;
> joinmultiplelines;
> lizmap;
> meinotestplugin;
> mmqgis;
> networks;
> ORStools;
> pluginbuilder3;
> qchainage;
> qgis-bulk-nominatim-master;
> Qgis2threejs;
> qgis2web;
> QuickOSM;
> quick_map_services;
> refFunctions;
> db_manager;
> MetaSearch;
> processing
>
>
>
>
>
>
>
>
> ___
> Qgis-user mailing list
> Qgis-user@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

[Qgis-user] Qgis low CPU and Memory load

2021-04-15 Thread Wolfgang Meinolf

Dear Community,
I use Qgis quite a while under win10 and ubuntu (via Oracle VM Box). My FTTH 
scripts include GRASS i.e. v.net.steiner algorithm, which are quite CPU and 
memory intensive. However I never observed more than 30% CPU and never more 
than appx 4GB mem usage. My Hardware is Intel I7, 32GB Ram and 1000TB SSD. At 
same time I don't observe any other activities i.e. on network. I installed 
OsGEO4W the 64bit Version. I'. currently on 3.18.1

I have however the impression to use a 32bit version.

I would appreciate any hint, how I can increase the CPU and MEM Load, to 
accelerate my scripts, which run usually quite a while. What do I wrong?

Thanks in advance and best regards from Bavaria.
Wolfgang



QGIS-Version

3.18.1-Zürich

QGIS-Codeversion

202f1bf7e5

Kompiliert gegen Qt

5.11.2

Laufendes Qt

5.11.2

Kompiliert mit GDAL/OGR

3.1.4

Läuft mit GDAL/OGR

3.1.4

Kompiliert mit GEOS

3.8.1-CAPI-1.13.3

Läuft mit GEOS

3.8.1-CAPI-1.13.3

Kompiliert mit SQLite

3.29.0

Läuft mit SQLite

3.29.0

PostgreSQL-Client-Version

11.5

SpatiaLite-Version

4.3.0

QWT-Version

6.1.3

QScintilla2-Version

2.10.8

Kompiliert mit PROJ

6.3.2

Läuft mit PROJ

Rel. 6.3.2, May 1st, 2020

BS-Version

Windows 10 (10.0)

Aktive Python-Erweiterungen

alkisplugin;
closest_points;
disconnected-islands;
ImportPhotos;
joinmultiplelines;
lizmap;
meinotestplugin;
mmqgis;
networks;
ORStools;
pluginbuilder3;
qchainage;
qgis-bulk-nominatim-master;
Qgis2threejs;
qgis2web;
QuickOSM;
quick_map_services;
refFunctions;
db_manager;
MetaSearch;
processing






smime.p7s
Description: S/MIME cryptographic signature
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Qgis low CPU and Memory load

Re: [Qgis-user] Qgis low CPU and Memory load

Re: [Qgis-user] Qgis low CPU and Memory load

Re: [Qgis-user] Qgis low CPU and Memory load

Re: [Qgis-user] Qgis low CPU and Memory load

[Qgis-user] Qgis low CPU and Memory load

6 matches

Site Navigation

Mail list logo

Footer information