On Mon, 29 Mar 2021 at 09:00, Pedro Camargo <c...@margo.co> wrote: > > Florian, > > I see that this is all very well developed in the C++ world, > but I would like to add one question and one comment. > > Question: > > - Is there anything similar to Python Multiprocessing that can be used? > Sometimes releasing the GIL is hard (requires going to C++/C/Cython). The > original question may have been on this, I guess.
As I've mentioned previously on this thread, this isn't an issue with PyQGIS. The GIL is automatically released whenever you call a PyQGIS method, unless it's absolutely certain that the method will be near-instant to call. So it's quite straightforward to use multiprocessing in PyQGIS via the Qt methods and do things like calculating intersections for different objects across multiple threads at once without having to worry about the GIL at all... Nyall > > - Using Python threads works well when you do everything in Cython (for > example) and release the GIL > > Cheers, > Pedro > > > ---- On Mon, 29 Mar 2021 05:00:01 +1000 > <qgis-developer-requ...@lists.osgeo.org> wrote ---- > > Send QGIS-Developer mailing list submissions to > > qgis-developer@lists.osgeo.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.osgeo.org/mailman/listinfo/qgis-developer > > or, via email, send a message with subject or body 'help' to > > qgis-developer-requ...@lists.osgeo.org > > > > You can reach the person managing the list at > > qgis-developer-ow...@lists.osgeo.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of QGIS-Developer digest..." > > > > > > Today's Topics: > > > > 1. Re: Multiprocessing QGIS (Florian El Ahdab) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Sun, 28 Mar 2021 16:48:12 +0200 > > From: Florian El Ahdab <felah...@gmail.com> > > To: Joao Folgado <jfolgad...@gmail.com> > > Cc: David Marteau <dmart...@3liz.com>, qgis-developer > > <qgis-developer@lists.osgeo.org> > > Subject: Re: [QGIS-Developer] Multiprocessing QGIS > > Message-ID: > > <caeeudys4cjkzb9ltrjwyj7fovkgzpno_hdw66eijgqxbwo6...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi. > > > > I am posting my reply again because I missed the qgis-developer mail list > > first... > > > > It is indeed possible to achieve multiprocessing in QGIS. > > > > But you should rely on Qt multi thread mechanisms rather than on the python > > multiprocessing module. > > > > Qgis is written in C++, based on the Qt library. Using the Qt > > possibilities, even if it is thru python, is less likely to trigger > > problems... > > > > QThread is your friend here. > > You can also use classes that are designed in QGis to ease things up, like > > QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html). > > > > Regards > > Florian. > > > > Le sam. 27 mars 2021 ? 13:48, Joao Folgado <jfolgad...@gmail.com> a ?crit : > > > > > Hi guys, > > > > > > First of all thank you but, David Marteua and Sebastian Ernst, for the > > > answer. > > > So is impossible to use multiprocessing in Qgis plugin? > > > This is a enormous limitations because Qgis have a lot of data that > > > needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores > > > and we can not use them in a plugin for example is bad. If some one have > > > any idea who we can use multiprocessing in a plugin i will be thankful. > > > > > > Best regards, > > > Jo?o Folgado > > > > > > David Marteau <dmart...@3liz.com> escreveu em sex., 26/03/2021 ?s 08:57 : > > > > > >> > I am not entirely sure what QGIS' (intended) behavior is in a case > like > > >> > this. From the top of my head, having read the relevant portion of its > > >> > source code, I'd expect at least a second application window likely > > >> > followed by non-deterministic behavior (again, depending on the > > >> > use-case, likely resulting in a crash). Can someone elaborate? > > >> > > >> I can confirm that the result is non-deterministic even with headless > > >> PyQgis programs. > > >> > > >> For programs relying heavily on multiprocessing the best strategy is to > > >> use a `fork-server` process > > >> started at the very beginning (before initializing Qgis) managing the > > >> pool of child processes, so that each forked > > >> process starts in the same state. > > >> > > >> David, > > >> > > >> Le 23/03/2021 ? 19:22, Sebastian M. Ernst a ?crit : > > >> > Hi Jo?o, all, > > >> > > > >> > really interesting question. > > >> > > > >> >> however when i run the plugin a new instance of application is open > > >> >> and give me an error. > > >> > I assume that you are using `multiprocessing` [1] from Python's > standard > > >> > library. Its default behavior is to use process-based parallelism (as > > >> > the title in the documentation says). I am assuming that you have not > > >> > altered its default behavior (e.g. by configuring it to use threads > > >> > instead). This is why you get at least another "worker" process if you > > >> > e.g. create a process pool. I have successfully used this method with > > >> > QGIS before, but always on Linux and OS X. I am just guessing that you > > >> > are doing this on Windows, right? > > >> > > > >> > Depending on the operating system, `multiprocessing` uses different > > >> > methods to create worker processes. On Linux and OS X, it simply forks > > >> > [2] the main process. This way, you do not see a second application > > >> > window popping up. You "simply" get a second, third, fourth, ... > process > > >> > "in the background". On Windows, the `fork` syscall does not exist, > > >> > neither does something similar [3]. `multiprocessing` therefore > starts a > > >> > complete new process from scratch. If it was pure Python, it would > > >> > simply start a new Python interpreter process. Because it is QGIS, it > > >> > opens another instance of QGIS. > > >> > > > >> > I am not entirely sure what QGIS' (intended) behavior is in a case > like > > >> > this. From the top of my head, having read the relevant portion of its > > >> > source code, I'd expect at least a second application window likely > > >> > followed by non-deterministic behavior (again, depending on the > > >> > use-case, likely resulting in a crash). Can someone elaborate? > > >> > > > >> > Best regards, > > >> > Sebastian > > >> > > > >> > > > >> > 1: https://docs.python.org/3/library/multiprocessing.html > > >> > 2: https://en.wikipedia.org/wiki/Fork_(system_call) > > >> > 3: https://stackoverflow.com/q/985281/1672565 > > >> > > > >> > > > >> > Am 23.03.21 um 13:08 schrieb Joao Folgado: > > >> >> Hi everyone, > > >> >> > > >> >> I had created a plugin for Qgis and I trying to optimize it. So i > use > > >> >> in my code the library python that have Queue whit multiprocessing > > >> >> however when i run the plugin a new instance of application is open > and > > >> >> give me an error. > > >> >> I had early use this type of multiprocessing before in simple > projects > > >> >> in university and it works ok. > > >> >> Can someone help ? I researched in many forums, i see people whit the > > >> >> same question but not with the answer. > > >> >> Thank you very much. > > >> >> > > >> >> Best regards, > > >> >> Jo?o Folgado > > >> >> Portugal > > >> >> -- > > >> >> JO?O FOLGADO > > >> >> > > >> >> _______________________________________________ > > >> >> QGIS-Developer mailing list > > >> >> QGIS-Developer@lists.osgeo.org > > >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> >> > > >> > _______________________________________________ > > >> > QGIS-Developer mailing list > > >> > QGIS-Developer@lists.osgeo.org > > >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> _______________________________________________ > > >> QGIS-Developer mailing list > > >> QGIS-Developer@lists.osgeo.org > > >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > >> > > > -- > > > JO?O FOLGADO > > > _______________________________________________ > > > QGIS-Developer mailing list > > > QGIS-Developer@lists.osgeo.org > > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210328/84f82603/attachment-0001.html> > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > QGIS-Developer mailing list > > QGIS-Developer@lists.osgeo.org > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer > > > > > > ------------------------------ > > > > End of QGIS-Developer Digest, Vol 185, Issue 60 > > *********************************************** > > > _______________________________________________ > QGIS-Developer mailing list > QGIS-Developer@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer _______________________________________________ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
