Re: [QGIS-Developer] Multiprocessing QGIS

[Sebastian M. Ernst]

Hi all,

Am 29.03.21 um 01:17 schrieb Nyall Dawson:
> [...] So it's quite straightforward to use
> multiprocessing in PyQGIS via the Qt methods and do things like
> calculating intersections for different objects across multiple
> threads at once without having to worry about the GIL at all...

I have just experimented a little bit with `multiprocessing` inside QGIS
(the GUI app) across multiple platforms. All performed tests are common
text-book examples for the basic use of `multiprocessing` (completely
ignoring QGIS' own internal mechanisms for similar-ish tasks). Tests 1
to 3 are process-based, i.e. they `fork` on Unix-like systems. Tests 4
and 5 are thread-based. All 5 tests work on Unix-like systems *and*
Windows inside a regular Python interpreter (without the involvement of
QGIS). If someone wants to replicate my experiments, here is the code:

https://gist.github.com/s-m-e/65197151fd3ddc74eee319252b4a9d6e

All 5 tests work flawlessly in QGIS 3.14.0 on Linux. I was lacking the
time to go through younger versions but I'd expect no difference.

While tests 4 and 5 do work, tests 1 to 3, the process-based ones, fail
in QGIS 3.18.1 (202f1bf7e5) on Windows 10 (OSGeo4W). Actually the way
they are failing is interesting. The failure happens even at an earlier
stage than I expected:

`AttributeError: module 'sys' has no attribute 'argv'`

It basically means that QGIS' C++ code does not forward `argv` to the
Python interpreter. On Linux is does not matter, but on Windows it does:
`multiprocessing` prepares to start new worker processes from scratch
(because there is no `fork`) and tries to access the main process'
`argv`, so it can eventually forward it to the workers. The error
therefore happens before it is even attempted to start worker processes
which is where I was originally expecting a failure. I think the `argv`
issue is a trivial thing to fix. Right after the Python interpreter gets
initialized ...

https://github.com/qgis/QGIS/blob/70eac1c97255ea31e3e5d57fc7cea721d080c8e8/src/python/qgspythonutilsimpl.cpp#L158

... a call to `PySys_SetArgv` is required:

https://docs.python.org/3/c-api/init.html?highlight=pysys_setargv#c.PySys_SetArgv

Once it is fixed, the question then becomes what happens / fails next. I
do not have an operational Windows build-environment at the moment -
does someone who has one like to try this? I'd be really interested in
the result.

Bottom line: Simple process-based parallelism with `multiprocessing`
**on Windows**, which is actually very common in the Python world, has
definately been broken in QGIS for several versions.

Best regards,
Sebastian
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Multiprocessing QGIS

[Pedro Camargo]

Hi Nyall,

I have seen that, and that's why I prefaced with an observation that this seems 
to be handled pretty well in the C++ world inside QGIS. 

However, there are methods/functions/algorithms that are not available in QGIS 
and that would benefit from being run in multiple threads (if the user knows 
how to go down to C/C++ and release the GIL themselves) or to just brute-force 
it with Multiprocessing.

One might ask why is somebody doing it inside QGIS, then?  Although that is a 
fair question, I would say that this is a testament to how good the QGIS 
interface and tools are, as people want to do things within its boundaries.  

AequilibraE itself, which is a package for transportation modelling, takes its 
cues from what is available in the QGIS packaging for Windows so it can be used 
by non-programmers and has all its core functionality based solely on the 
packages available by default (some functionality depends on the user 
installing further packages).

So, although it is fair to have a stance that QGIS is NOT intended to be a 
virtual environment that comes with great GIS software, I would say it is 
understandable that some people would like it to be.

Cheers,
Pedro

PS - And I promise I won't cause any more steer (this week), Nyall!  :-D 


  On Mon, 29 Mar 2021 09:17:08 +1000 Nyall Dawson  
wrote 
 > On Mon, 29 Mar 2021 at 09:00, Pedro Camargo  wrote:
 > >
 > > Florian,
 > >
 > >   I see that this is all very well developed in the C++ world, 
 > > but I would like to add one question and one comment.
 > >
 > > Question:
 > >
 > > - Is there anything similar to Python Multiprocessing that can be used?  
 > > Sometimes releasing the GIL is hard (requires going to C++/C/Cython).  The 
 > > original question may have been on this, I guess.
 > 
 > As I've mentioned previously on this thread, this isn't an issue with
 > PyQGIS. The GIL is automatically released whenever you call a PyQGIS
 > method, unless it's absolutely certain that the method will be
 > near-instant to call. So it's quite straightforward to use
 > multiprocessing in PyQGIS via the Qt methods and do things like
 > calculating intersections for different objects across multiple
 > threads at once without having to worry about the GIL at all...
 > 
 > Nyall
 > 
 > 
 > >
 > > - Using Python threads works well when you do everything in Cython (for 
 > > example) and release the GIL
 > >
 > > Cheers,
 > > Pedro
 > >
 > >
 > >   On Mon, 29 Mar 2021 05:00:01 +1000  
 > >  wrote 
 > >  > Send QGIS-Developer mailing list submissions to
 > >  > qgis-developer@lists.osgeo.org
 > >  >
 > >  > To subscribe or unsubscribe via the World Wide Web, visit
 > >  > https://lists.osgeo.org/mailman/listinfo/qgis-developer
 > >  > or, via email, send a message with subject or body 'help' to
 > >  > qgis-developer-requ...@lists.osgeo.org
 > >  >
 > >  > You can reach the person managing the list at
 > >  > qgis-developer-ow...@lists.osgeo.org
 > >  >
 > >  > When replying, please edit your Subject line so it is more specific
 > >  > than "Re: Contents of QGIS-Developer digest..."
 > >  >
 > >  >
 > >  > Today's Topics:
 > >  >
 > >  >1. Re: Multiprocessing QGIS (Florian El Ahdab)
 > >  >
 > >  >
 > >  > --
 > >  >
 > >  > Message: 1
 > >  > Date: Sun, 28 Mar 2021 16:48:12 +0200
 > >  > From: Florian El Ahdab 
 > >  > To: Joao Folgado 
 > >  > Cc: David Marteau , qgis-developer
 > >  > 
 > >  > Subject: Re: [QGIS-Developer] Multiprocessing QGIS
 > >  > Message-ID:
 > >  > 
 > >  > Content-Type: text/plain; charset="utf-8"
 > >  >
 > >  > Hi.
 > >  >
 > >  > I am posting my reply again because I missed the qgis-developer mail 
 > > list
 > >  > first...
 > >  >
 > >  > It is indeed possible to achieve multiprocessing in QGIS.
 > >  >
 > >  > But you should rely on Qt multi thread mechanisms rather than on the 
 > > python
 > >  > multiprocessing module.
 > >  >
 > >  > Qgis is written in C++, based on the Qt library. Using the Qt
 > >  > possibilities, even if it is thru python, is less likely to trigger
 > >  > problems...
 > >  >
 > >  > QThread is your friend here.
 > >  > You can also use classes that are designed in QGis to ease things up, 
 > > like
 > >  > QgsTask (https://n

Re: [QGIS-Developer] Multiprocessing QGIS

[Nyall Dawson]

On Mon, 29 Mar 2021 at 09:00, Pedro Camargo  wrote:
>
> Florian,
>
>   I see that this is all very well developed in the C++ world, 
> but I would like to add one question and one comment.
>
> Question:
>
> - Is there anything similar to Python Multiprocessing that can be used?  
> Sometimes releasing the GIL is hard (requires going to C++/C/Cython).  The 
> original question may have been on this, I guess.

As I've mentioned previously on this thread, this isn't an issue with
PyQGIS. The GIL is automatically released whenever you call a PyQGIS
method, unless it's absolutely certain that the method will be
near-instant to call. So it's quite straightforward to use
multiprocessing in PyQGIS via the Qt methods and do things like
calculating intersections for different objects across multiple
threads at once without having to worry about the GIL at all...

Nyall


>
> - Using Python threads works well when you do everything in Cython (for 
> example) and release the GIL
>
> Cheers,
> Pedro
>
>
>   On Mon, 29 Mar 2021 05:00:01 +1000  
>  wrote 
>  > Send QGIS-Developer mailing list submissions to
>  > qgis-developer@lists.osgeo.org
>  >
>  > To subscribe or unsubscribe via the World Wide Web, visit
>  > https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > or, via email, send a message with subject or body 'help' to
>  > qgis-developer-requ...@lists.osgeo.org
>  >
>  > You can reach the person managing the list at
>  > qgis-developer-ow...@lists.osgeo.org
>  >
>  > When replying, please edit your Subject line so it is more specific
>  > than "Re: Contents of QGIS-Developer digest..."
>  >
>  >
>  > Today's Topics:
>  >
>  >1. Re: Multiprocessing QGIS (Florian El Ahdab)
>  >
>  >
>  > ------
>  >
>  > Message: 1
>  > Date: Sun, 28 Mar 2021 16:48:12 +0200
>  > From: Florian El Ahdab 
>  > To: Joao Folgado 
>  > Cc: David Marteau , qgis-developer
>  > 
>  > Subject: Re: [QGIS-Developer] Multiprocessing QGIS
>  > Message-ID:
>  > 
>  > Content-Type: text/plain; charset="utf-8"
>  >
>  > Hi.
>  >
>  > I am posting my reply again because I missed the qgis-developer mail list
>  > first...
>  >
>  > It is indeed possible to achieve multiprocessing in QGIS.
>  >
>  > But you should rely on Qt multi thread mechanisms rather than on the python
>  > multiprocessing module.
>  >
>  > Qgis is written in C++, based on the Qt library. Using the Qt
>  > possibilities, even if it is thru python, is less likely to trigger
>  > problems...
>  >
>  > QThread is your friend here.
>  > You can also use classes that are designed in QGis to ease things up, like
>  > QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).
>  >
>  > Regards
>  > Florian.
>  >
>  > Le sam. 27 mars 2021 ? 13:48, Joao Folgado  a ?crit :
>  >
>  > > Hi guys,
>  > >
>  > > First of all thank you but, David Marteua and Sebastian Ernst, for the
>  > > answer.
>  > > So is impossible to use multiprocessing in Qgis plugin?
>  > > This is a enormous limitations because Qgis have a lot of data  that
>  > >  needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
>  > > and we can not use them in a plugin for example is bad. If some one have
>  > > any idea who we can use multiprocessing in a plugin i will be thankful.
>  > >
>  > > Best regards,
>  > > Jo?o Folgado
>  > >
>  > > David Marteau  escreveu em sex., 26/03/2021 ?s 08:57 :
>  > >
>  > >> > I am not entirely sure what QGIS' (intended) behavior is in a case 
> like
>  > >> > this. From the top of my head, having read the relevant portion of its
>  > >> > source code, I'd expect at least a second application window likely
>  > >> > followed by non-deterministic behavior (again, depending on the
>  > >> > use-case, likely resulting in a crash). Can someone elaborate?
>  > >>
>  > >> I can confirm that the result is non-deterministic even with headless
>  > >> PyQgis programs.
>  > >>
>  > >> For programs relying heavily on multiprocessing the best strategy is to
>  > >> use a `fork-server` process
>  > >> started at the very beginning (before initializing Qgis) managing the
>  > >> pool of child processes, so that each forked
>  > >> proces

Re: [QGIS-Developer] Multiprocessing QGIS

[Pedro Camargo]

Florian,

  I see that this is all very well developed in the C++ world, but 
I would like to add one question and one comment.

Question:

- Is there anything similar to Python Multiprocessing that can be used?  
Sometimes releasing the GIL is hard (requires going to C++/C/Cython).  The 
original question may have been on this, I guess.

- Using Python threads works well when you do everything in Cython (for 
example) and release the GIL

Cheers,
Pedro


  On Mon, 29 Mar 2021 05:00:01 +1000  
 wrote 
 > Send QGIS-Developer mailing list submissions to
 > qgis-developer@lists.osgeo.org
 > 
 > To subscribe or unsubscribe via the World Wide Web, visit
 > https://lists.osgeo.org/mailman/listinfo/qgis-developer
 > or, via email, send a message with subject or body 'help' to
 > qgis-developer-requ...@lists.osgeo.org
 > 
 > You can reach the person managing the list at
 > qgis-developer-ow...@lists.osgeo.org
 > 
 > When replying, please edit your Subject line so it is more specific
 > than "Re: Contents of QGIS-Developer digest..."
 > 
 > 
 > Today's Topics:
 > 
 >1. Re: Multiprocessing QGIS (Florian El Ahdab)
 > 
 > 
 > --
 > 
 > Message: 1
 > Date: Sun, 28 Mar 2021 16:48:12 +0200
 > From: Florian El Ahdab 
 > To: Joao Folgado 
 > Cc: David Marteau , qgis-developer
 > 
 > Subject: Re: [QGIS-Developer] Multiprocessing QGIS
 > Message-ID:
 > 
 > Content-Type: text/plain; charset="utf-8"
 > 
 > Hi.
 > 
 > I am posting my reply again because I missed the qgis-developer mail list
 > first...
 > 
 > It is indeed possible to achieve multiprocessing in QGIS.
 > 
 > But you should rely on Qt multi thread mechanisms rather than on the python
 > multiprocessing module.
 > 
 > Qgis is written in C++, based on the Qt library. Using the Qt
 > possibilities, even if it is thru python, is less likely to trigger
 > problems...
 > 
 > QThread is your friend here.
 > You can also use classes that are designed in QGis to ease things up, like
 > QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).
 > 
 > Regards
 > Florian.
 > 
 > Le sam. 27 mars 2021 ? 13:48, Joao Folgado  a ?crit :
 > 
 > > Hi guys,
 > >
 > > First of all thank you but, David Marteua and Sebastian Ernst, for the
 > > answer.
 > > So is impossible to use multiprocessing in Qgis plugin?
 > > This is a enormous limitations because Qgis have a lot of data  that
 > >  needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
 > > and we can not use them in a plugin for example is bad. If some one have
 > > any idea who we can use multiprocessing in a plugin i will be thankful.
 > >
 > > Best regards,
 > > Jo?o Folgado
 > >
 > > David Marteau  escreveu em sex., 26/03/2021 ?s 08:57 :
 > >
 > >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
 > >> > this. From the top of my head, having read the relevant portion of its
 > >> > source code, I'd expect at least a second application window likely
 > >> > followed by non-deterministic behavior (again, depending on the
 > >> > use-case, likely resulting in a crash). Can someone elaborate?
 > >>
 > >> I can confirm that the result is non-deterministic even with headless
 > >> PyQgis programs.
 > >>
 > >> For programs relying heavily on multiprocessing the best strategy is to
 > >> use a `fork-server` process
 > >> started at the very beginning (before initializing Qgis) managing the
 > >> pool of child processes, so that each forked
 > >> process starts in the same state.
 > >>
 > >> David,
 > >>
 > >> Le 23/03/2021 ? 19:22, Sebastian M. Ernst a ?crit :
 > >> > Hi Jo?o, all,
 > >> >
 > >> > really interesting question.
 > >> >
 > >> >> however when i run the plugin a new instance of application is open
 > >> >> and give me an error.
 > >> > I assume that you are using `multiprocessing` [1] from Python's standard
 > >> > library. Its default behavior is to use process-based parallelism (as
 > >> > the title in the documentation says). I am assuming that you have not
 > >> > altered its default behavior (e.g. by configuring it to use threads
 > >> > instead). This is why you get at least another "worker" process if you
 > >> > e.g. create a process pool. I have successfully used this method with
 > >> > QGIS before, but always on Linux and 

Re: [QGIS-Developer] Multiprocessing QGIS

[Florian El Ahdab]

Hi.

I am posting my reply again because I missed the qgis-developer mail list
first...

It is indeed possible to achieve multiprocessing in QGIS.

But you should rely on Qt multi thread mechanisms rather than on the python
multiprocessing module.

Qgis is written in C++, based on the Qt library. Using the Qt
possibilities, even if it is thru python, is less likely to trigger
problems...

QThread is your friend here.
You can also use classes that are designed in QGis to ease things up, like
QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).

Regards
Florian.

Le sam. 27 mars 2021 à 13:48, Joao Folgado  a écrit :

> Hi guys,
>
> First of all thank you but, David Marteua and Sebastian Ernst, for the
> answer.
> So is impossible to use multiprocessing in Qgis plugin?
> This is a enormous limitations because Qgis have a lot of data  that
>  needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
> and we can not use them in a plugin for example is bad. If some one have
> any idea who we can use multiprocessing in a plugin i will be thankful.
>
> Best regards,
> João Folgado
>
> David Marteau  escreveu em sex., 26/03/2021 às 08:57 :
>
>> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>> > this. From the top of my head, having read the relevant portion of its
>> > source code, I'd expect at least a second application window likely
>> > followed by non-deterministic behavior (again, depending on the
>> > use-case, likely resulting in a crash). Can someone elaborate?
>>
>> I can confirm that the result is non-deterministic even with headless
>> PyQgis programs.
>>
>> For programs relying heavily on multiprocessing the best strategy is to
>> use a `fork-server` process
>> started at the very beginning (before initializing Qgis) managing the
>> pool of child processes, so that each forked
>> process starts in the same state.
>>
>> David,
>>
>> Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :
>> > Hi João, all,
>> >
>> > really interesting question.
>> >
>> >> however when i run the plugin a new instance of application is open
>> >> and give me an error.
>> > I assume that you are using `multiprocessing` [1] from Python's standard
>> > library. Its default behavior is to use process-based parallelism (as
>> > the title in the documentation says). I am assuming that you have not
>> > altered its default behavior (e.g. by configuring it to use threads
>> > instead). This is why you get at least another "worker" process if you
>> > e.g. create a process pool. I have successfully used this method with
>> > QGIS before, but always on Linux and OS X. I am just guessing that you
>> > are doing this on Windows, right?
>> >
>> > Depending on the operating system, `multiprocessing` uses different
>> > methods to create worker processes. On Linux and OS X, it simply forks
>> > [2] the main process. This way, you do not see a second application
>> > window popping up. You "simply" get a second, third, fourth, ... process
>> > "in the background". On Windows, the `fork` syscall does not exist,
>> > neither does something similar [3]. `multiprocessing` therefore starts a
>> > complete new process from scratch. If it was pure Python, it would
>> > simply start a new Python interpreter process. Because it is QGIS, it
>> > opens another instance of QGIS.
>> >
>> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>> > this. From the top of my head, having read the relevant portion of its
>> > source code, I'd expect at least a second application window likely
>> > followed by non-deterministic behavior (again, depending on the
>> > use-case, likely resulting in a crash). Can someone elaborate?
>> >
>> > Best regards,
>> > Sebastian
>> >
>> >
>> > 1: https://docs.python.org/3/library/multiprocessing.html
>> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
>> > 3: https://stackoverflow.com/q/985281/1672565
>> >
>> >
>> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
>> >> Hi everyone,
>> >>
>> >> I had created  a plugin for Qgis and I trying to optimize it. So i use
>> >> in my code the library python that have Queue whit multiprocessing
>> >> however when i run the plugin a new instance of application is open and
>> >> give me an error.
>> >> I had early use this type of multiprocessing before in simple projects
>> >> in university and it works ok.
>> >> Can someone help ? I researched in many forums, i see people whit the
>> >> same question but not with the answer.
>> >> Thank you very much.
>> >>
>> >> Best regards,
>> >> João Folgado
>> >> Portugal
>> >> --
>> >> JOÃO FOLGADO
>> >>
>> >> ___
>> >> QGIS-Developer mailing list
>> >> QGIS-Developer@lists.osgeo.org
>> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >>
>> > ___
>> > QGIS-Developer mailing list
>> > 

Re: [QGIS-Developer] Multiprocessing QGIS

[Joao Folgado]

Hi guys,

First of all thank you but, David Marteua and Sebastian Ernst, for the
answer.
So is impossible to use multiprocessing in Qgis plugin?
This is a enormous limitations because Qgis have a lot of data  that  needs
to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores and we
can not use them in a plugin for example is bad. If some one have any idea
who we can use multiprocessing in a plugin i will be thankful.

Best regards,
João Folgado

David Marteau  escreveu em sex., 26/03/2021 às 08:57 :

> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > this. From the top of my head, having read the relevant portion of its
> > source code, I'd expect at least a second application window likely
> > followed by non-deterministic behavior (again, depending on the
> > use-case, likely resulting in a crash). Can someone elaborate?
>
> I can confirm that the result is non-deterministic even with headless
> PyQgis programs.
>
> For programs relying heavily on multiprocessing the best strategy is to
> use a `fork-server` process
> started at the very beginning (before initializing Qgis) managing the
> pool of child processes, so that each forked
> process starts in the same state.
>
> David,
>
> Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :
> > Hi João, all,
> >
> > really interesting question.
> >
> >> however when i run the plugin a new instance of application is open
> >> and give me an error.
> > I assume that you are using `multiprocessing` [1] from Python's standard
> > library. Its default behavior is to use process-based parallelism (as
> > the title in the documentation says). I am assuming that you have not
> > altered its default behavior (e.g. by configuring it to use threads
> > instead). This is why you get at least another "worker" process if you
> > e.g. create a process pool. I have successfully used this method with
> > QGIS before, but always on Linux and OS X. I am just guessing that you
> > are doing this on Windows, right?
> >
> > Depending on the operating system, `multiprocessing` uses different
> > methods to create worker processes. On Linux and OS X, it simply forks
> > [2] the main process. This way, you do not see a second application
> > window popping up. You "simply" get a second, third, fourth, ... process
> > "in the background". On Windows, the `fork` syscall does not exist,
> > neither does something similar [3]. `multiprocessing` therefore starts a
> > complete new process from scratch. If it was pure Python, it would
> > simply start a new Python interpreter process. Because it is QGIS, it
> > opens another instance of QGIS.
> >
> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > this. From the top of my head, having read the relevant portion of its
> > source code, I'd expect at least a second application window likely
> > followed by non-deterministic behavior (again, depending on the
> > use-case, likely resulting in a crash). Can someone elaborate?
> >
> > Best regards,
> > Sebastian
> >
> >
> > 1: https://docs.python.org/3/library/multiprocessing.html
> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
> > 3: https://stackoverflow.com/q/985281/1672565
> >
> >
> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
> >> Hi everyone,
> >>
> >> I had created  a plugin for Qgis and I trying to optimize it. So i use
> >> in my code the library python that have Queue whit multiprocessing
> >> however when i run the plugin a new instance of application is open and
> >> give me an error.
> >> I had early use this type of multiprocessing before in simple projects
> >> in university and it works ok.
> >> Can someone help ? I researched in many forums, i see people whit the
> >> same question but not with the answer.
> >> Thank you very much.
> >>
> >> Best regards,
> >> João Folgado
> >> Portugal
> >> --
> >> JOÃO FOLGADO
> >>
> >> ___
> >> QGIS-Developer mailing list
> >> QGIS-Developer@lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>
> > ___
> > QGIS-Developer mailing list
> > QGIS-Developer@lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-- 
JOÃO FOLGADO
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Multiprocessing QGIS

[David Marteau]

I am not entirely sure what QGIS' (intended) behavior is in a case like
this. From the top of my head, having read the relevant portion of its
source code, I'd expect at least a second application window likely
followed by non-deterministic behavior (again, depending on the
use-case, likely resulting in a crash). Can someone elaborate?


I can confirm that the result is non-deterministic even with headless 
PyQgis programs.


For programs relying heavily on multiprocessing the best strategy is to 
use a `fork-server` process
started at the very beginning (before initializing Qgis) managing the 
pool of child processes, so that each forked

process starts in the same state.

David,

Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :

Hi João, all,

really interesting question.


however when i run the plugin a new instance of application is open
and give me an error.

I assume that you are using `multiprocessing` [1] from Python's standard
library. Its default behavior is to use process-based parallelism (as
the title in the documentation says). I am assuming that you have not
altered its default behavior (e.g. by configuring it to use threads
instead). This is why you get at least another "worker" process if you
e.g. create a process pool. I have successfully used this method with
QGIS before, but always on Linux and OS X. I am just guessing that you
are doing this on Windows, right?

Depending on the operating system, `multiprocessing` uses different
methods to create worker processes. On Linux and OS X, it simply forks
[2] the main process. This way, you do not see a second application
window popping up. You "simply" get a second, third, fourth, ... process
"in the background". On Windows, the `fork` syscall does not exist,
neither does something similar [3]. `multiprocessing` therefore starts a
complete new process from scratch. If it was pure Python, it would
simply start a new Python interpreter process. Because it is QGIS, it
opens another instance of QGIS.

I am not entirely sure what QGIS' (intended) behavior is in a case like
this. From the top of my head, having read the relevant portion of its
source code, I'd expect at least a second application window likely
followed by non-deterministic behavior (again, depending on the
use-case, likely resulting in a crash). Can someone elaborate?

Best regards,
Sebastian


1: https://docs.python.org/3/library/multiprocessing.html
2: https://en.wikipedia.org/wiki/Fork_(system_call)
3: https://stackoverflow.com/q/985281/1672565


Am 23.03.21 um 13:08 schrieb Joao Folgado:

Hi everyone,

I had created  a plugin for Qgis and I trying to optimize it. So i use
in my code the library python that have Queue whit multiprocessing
however when i run the plugin a new instance of application is open and
give me an error.
I had early use this type of multiprocessing before in simple projects
in university and it works ok.
Can someone help ? I researched in many forums, i see people whit the
same question but not with the answer.
Thank you very much.

Best regards,
João Folgado
Portugal
--
JOÃO FOLGADO

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Multiprocessing QGIS

[Sebastian M. Ernst]

Hi João, all,

really interesting question.

> however when i run the plugin a new instance of application is open
> and give me an error.

I assume that you are using `multiprocessing` [1] from Python's standard
library. Its default behavior is to use process-based parallelism (as
the title in the documentation says). I am assuming that you have not
altered its default behavior (e.g. by configuring it to use threads
instead). This is why you get at least another "worker" process if you
e.g. create a process pool. I have successfully used this method with
QGIS before, but always on Linux and OS X. I am just guessing that you
are doing this on Windows, right?

Depending on the operating system, `multiprocessing` uses different
methods to create worker processes. On Linux and OS X, it simply forks
[2] the main process. This way, you do not see a second application
window popping up. You "simply" get a second, third, fourth, ... process
"in the background". On Windows, the `fork` syscall does not exist,
neither does something similar [3]. `multiprocessing` therefore starts a
complete new process from scratch. If it was pure Python, it would
simply start a new Python interpreter process. Because it is QGIS, it
opens another instance of QGIS.

I am not entirely sure what QGIS' (intended) behavior is in a case like
this. From the top of my head, having read the relevant portion of its
source code, I'd expect at least a second application window likely
followed by non-deterministic behavior (again, depending on the
use-case, likely resulting in a crash). Can someone elaborate?

Best regards,
Sebastian


1: https://docs.python.org/3/library/multiprocessing.html
2: https://en.wikipedia.org/wiki/Fork_(system_call)
3: https://stackoverflow.com/q/985281/1672565


Am 23.03.21 um 13:08 schrieb Joao Folgado:
> Hi everyone,
> 
> I had created  a plugin for Qgis and I trying to optimize it. So i use
> in my code the library python that have Queue whit multiprocessing
> however when i run the plugin a new instance of application is open and
> give me an error.
> I had early use this type of multiprocessing before in simple projects
> in university and it works ok.
> Can someone help ? I researched in many forums, i see people whit the
> same question but not with the answer.
> Thank you very much.
> 
> Best regards,
> João Folgado
> Portugal
> -- 
> JOÃO FOLGADO
> 
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> 
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Multiprocessing qgis console

[Shane Carey]

Ok cool - thanks

On Céad 11 Noll 2019 at 09:07, Luigi Pirelli  wrote:

> Not yet... but consider that GUI works in the main thread => if you need
> to post messages in the console you should need to create producer/consumer
> queues (see use of Queue module).
> I would suggest to simplify everything using QgsTask.
>
> cheers
>
> Luigi Pirelli
>
>
> **
> * LinkedIn: https://www.linkedin.com/in/luigipirelli
> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
> * GitHub: https://github.com/luipir
> * Book: Mastering QGIS3 - 3rd Edition
> 
> * Hire a team: http://www.qcooperative.net
>
> **
>
>
> On Wed, 11 Dec 2019 at 08:33, Shane Carey  wrote:
>
>> Hi,
>>
>> Has anyone to successfully setup multiprocessing through the the qgis
>> console?
>> Thanks
>> --
>> Le gach dea ghui,
>> *Shane Carey*
>> *GIS and Data Solutions Consultant*
>>
> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
Le gach dea ghui,
*Shane Carey*
*GIS and Data Solutions Consultant*
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Multiprocessing qgis console

[Luigi Pirelli]

Not yet... but consider that GUI works in the main thread => if you need to
post messages in the console you should need to create producer/consumer
queues (see use of Queue module).
I would suggest to simplify everything using QgsTask.

cheers

Luigi Pirelli

**
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition

* Hire a team: http://www.qcooperative.net
**


On Wed, 11 Dec 2019 at 08:33, Shane Carey  wrote:

> Hi,
>
> Has anyone to successfully setup multiprocessing through the the qgis
> console?
> Thanks
> --
> Le gach dea ghui,
> *Shane Carey*
> *GIS and Data Solutions Consultant*
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer