Re: [Qgis-developer] qgis-2.18.4 denies using QGIS_PLUGINPATH?
Hello Alessandro, thank you for the fast answer, but.. Am 20.03.2017 um 11:33 schrieb Alessandro Pasotti: > On Mon, Mar 20, 2017 at 11:17 AM, Jörg Habenicht <mailto:j...@mwerk.net>> wrote: > > Hi all, > > I need some help setting up a separate path to the qgis plugin libs. > I think the environment variable QGIS_PLUGINPATH or the --configpath > setting is not respected during the run of qgis_mapserv.fcgi? > > > QGIS_PLUGINPATH allows to specify a Python plugins path (yes, bad > naming, totally my fault), what you are looking for is probably > QGIS_PREFIX_PATH. I have tried this, but the behaviour hasn't changed: nohup bash -c "LD_DEBUG=files LD_LIBRARY_PATH=/usr/local/lib/qgis-2.18.4/lib QGIS_PLUGINPATH=/usr/local/lib/qgis-2.18.4/lib QGIS_PREFIX_PATH=/usr/local/lib/qgis-2.18.4/lib QT_PLUGIN_PATH=/usr/local/lib/qgis-2.18.4/lib /usr/local/lib/qgis-2.18.4/bin/qgis_mapserv.fcgi" # grep /usr/lib/qgis nohup.out | grep -A1 libarcgisfeatureserverprovider.so 23672: file=/usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so [0]; dynamically loaded by /usr/local/lib/qgis-2.18.4/lib/libQtCore.so.4 [0] 23672: file=/usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so [0]; generating link map 23672: file=libqjson.so.0 [0]; needed by /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so [0] 23672: calling init: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so 23672: opening file=/usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so [0]; direct_opencount=1 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: type (fatal) 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: databaseDrivers (fatal) 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: directoryDrivers (fatal) 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: protocolDrivers (fatal) 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: fileVectorFilters (fatal) 23672: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so: error: symbol lookup error: undefined symbol: buildSupportedRasterFileFilter (fatal) 23672: file=/usr/lib/qgis/plugins/libarcgismapserverprovider.so [0]; dynamically loaded by /usr/local/lib/qgis-2.18.4/lib/libQtCore.so.4 [0] -- 23672: calling fini: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so [0] 23672: calling fini: /usr/lib/qgis/plugins/libarcgismapserverprovider.so [0] There are 20 libs or plugins called from /usr/lib/qgis/plugins/ # grep /usr/lib/qgis/plugins/ nohup.out | grep "calling init" | wc -l 20 > > Background: > I try to setup multiple versions of qgis on our server in parallel. > We need this on our server to run multiple qgis configurations (round > about 30 card configs) on our server. To ease the upgrade testing we > decided to provide an individual qgis version setting for each card. > [...] > Is this a bug or do I need to use a different variable? > > > Using qgis-2.18.4 on Ubuntu 14.04.4 LTS > > > > best regards > Jörg > -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] qgis-2.18.4 denies using QGIS_PLUGINPATH?
Hi all, I need some help setting up a separate path to the qgis plugin libs. I think the environment variable QGIS_PLUGINPATH or the --configpath setting is not respected during the run of qgis_mapserv.fcgi? Background: I try to setup multiple versions of qgis on our server in parallel. We need this on our server to run multiple qgis configurations (round about 30 card configs) on our server. To ease the upgrade testing we decided to provide an individual qgis version setting for each card. This should work as follows: 1) Copy all libraries needed to a separate directory. 2) Run the binary with hints to use the new directory path LD_LIBRARY_PATH=/usr/local/lib/qgis-2.18.4/lib QGIS_PLUGINPATH=/usr/local/lib/qgis-2.18.4/lib QT_PLUGIN_PATH=/usr/local/lib/qgis-2.18.4/lib /usr/local/lib/qgis-2.18.4/bin/qgis_mapserv.fcgi The error: If I call the fcgi program with LD_DEBUG=files LD_LIBRARY_PATH=... I get the following message: calling init: /usr/lib/qgis/plugins/libarcgisfeatureserverprovider.so But this is the wrong directory. Even though I provided the QGIS plugin path with QGIS_PLUGINPATH=/usr/local/lib/qgis-2.18.4/lib, --configpath /usr/local/lib/qgis-2.18.4/lib or --configpath=/usr/local/lib/qgis-2.18.4/lib the lib gets loaded from /usr/lib/qgis/plugins instead of /usr/local/lib/qgis-2.18.4/lib Is this a bug or do I need to use a different variable? Using qgis-2.18.4 on Ubuntu 14.04.4 LTS best regards Jörg -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [Server] performance questions
Hello Andreas, I did forward the question to our internal development, returned some answers for clarity: Am 22.04.2016 um 16:45 schrieb Neumann, Andreas: > Hi Jörg, > > Unfortunately I am not the suitable person to ask. In only occasionally > develop Python and don't know C/C++. However, I am a long-time QGIS > server user. I have big projects as well (with 80/90 layers). They > usually render in 1-2 seconds, not 20 seconds like you cite. So maybe > there are other issues/bottlenecks? It seems so. > Do you have an exceptional number of features in your layers? Yes. > Do you use scale thresholds to filter features. No. > Typically it doesn't make any sense to render too many features at small > scales, because you can't depict them anyway if there are too many. > > I assume you use Postgis as a data source - since you talk about DB > queries. Ah yes, correct. > Do you have the proper spatial index and other indexes? We do have indices installed along the tables, sometimes more than one. We did not analyse each and every table, but the handling should perform sufficiently. > Do you have a recent version of QGIS server? Fairly recent, version 9.4 > I think version 2.10 and later > introduced filtering inside the DB (instead of filtering in QGIS), which > should come with a performance improvement. No, we dont use DB server filtering. > 20 seconds is definitely too long a wait for a map rendering, but I > guess you can bring this down to 1-2 seconds if properly optimized (both > in DB and in QGIS). > > It may well be that there is room for improvements in rendering speed of > QGIS server, but multi-threading is probaby the most complicated way of > improving performance in QGIS server. When I asked Marco Hugentobler, > the main QGIS Server developer, he told me it would be almost impossible > to make QGIS server multi-threaded. 3LIZ is also into QGIS server > development since some time. So these are the guys to ask. > > And then - the other issue is that you want to reserve your other CPU > cores for parallel requests as well. You usually don't want to dedicate > all your CPU power to a single http request. Oh yes certainly I want to. If an eight core server uses only one core to calculate the result, we could have a speed improvement factor 2-6 (my guess) for the single answer. With negligible slowdown of the application because of the additional handling of multi threading. So in this case the server delivers the resulting picture faster than before. And if the same server already has a multi threaded request running on eight cores, then the linux kernel divides the cpu calculation. So that a single request still gains an estimated improvement of 1-3. So in the result we have no significant speed loss in case of a fully loaded server. But much gain in case of an idle server. > > But if there is a way to improve speed in QGIS server I am all for it ... > > Greetings, > > Andreas > [snip] cu Jörg -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [Server] performance questions
Am 23.04.2016 um 11:47 schrieb Matthias Kuhn: > Hi Jörg, > > On 22/04/16 18:24, Jörg Habenicht wrote: >> Hi Matthias, >> [snip] >> It seems to me I have to extract the code inside the loop (in >> QgsMapRenderer::render(), "while ( li.hasPrevious() )") into a separate >> method and call QtConcurrent on that. > You shouldn't have to deal directly with QtConcurrent, this code is > alredy present. > > https://qgis.org/api/classQgsMapRendererJob.html#details > > Have a look at QgsMapCanvas::refreshMap() which sets up a job and > connects the signal finished() to a local slot and continues execution. > On the server instead you probably want to use a QFutureWatcher with > waitForFinished() to wait for the painting job (with its threads) finish > before responding with the freshly painted image. Oh yes, good hint. thank you > > Matthias cu Jörg -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [Server] performance questions
Hello Marco. Am 22.04.2016 um 17:44 schrieb Marco Hugentobler: > Hi > > Yes, rendering multithreaded in the server is possible (but needs some > work of course). > > What I said was that the effect is probably not that big, because the > fcgi module already distributes concurrent requests to different > processes. we already got this technology, but in those cases described below the work distribution does not help. Currently we got around 10 clicks/second at max, but the user has to wait an awful long time for the map to update in his/her session. We have measured the average response time which is around 1 second. But using FCGI and parallel QGis-server processes does not help in this case. > In the special case of very rare requests to the server, it > might have a considerable effect. But even then it is very unlikely that > a 20sec request will render in 1-2 sec (what you need for interactive > web maps). I'd estimate it to be nearly linear for painting the pictures (and maybe also for calling the DB). On your server we can see one core to be fully utilized calculating with low IO and all others idling around. I'd like to give it a try in the below mentioned method. Do I have to expect serious side effects? Do I have to consider something I forgot in my design? > > Regards, > Marco cu Jörg > [snip] >>> Il 22 apr 2016 3:27 PM, "Jörg Habenicht" >> <mailto:j...@mwerk.net>> ha scritto: >>> >>> Hi Andreas, dear list, >>> [snip] >>> >>> I have scanned the git-Qgis sources (commit >>> 81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable >>> multithreading >>> entry. Found QgsMapRenderer::render() because of the line "while ( >>> li.hasPrevious() )". It seems in this loop the layers are painted in >>> serial onto the resulting picture. >>> If I can build an auto parallel computing with QtConcurrent in this >>> place there seems to be the most gain. I mean to parallelize the loop >>> with concurrent DB requests, concurrent painting in separate pictures >>> and in serial map the single pictures onto the result picture. >>> >>> Right now I don't know if I can do DB requests in parallel, or >>> where the >>> code for database requests is buried. >>> And how about the calls inside the loop, are there side effects >>> preventing multi threading? >>> >>> >>> Do you got hints, flames or suggestions? >>> >>> >>> cu >>> Jörg >>> >>> [snip] -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [Server] performance questions
Hi Matthias, Am 22.04.2016 um 17:20 schrieb Matthias Kuhn: > Hi, > > I don't know the server code very well, so I may be wrong, but I think > that making the server multithreaded rendering (MTR) capable should be > possible. And while it is certainly often possible to increase the > performance with filtering, rules and caching there are certainly also > situations where MTR can help a server to generate quick responses (many > layers, few requests). > > As far as I can see, the main task would be porting the code from > QgsMapRenderer to QgsMapRendererJob and QgsMapSettings. A first step > into this direction can be found in a recently closed pull request [1], > in particular this commit [2]. I'll have a look. It seems to me I have to extract the code inside the loop (in QgsMapRenderer::render(), "while ( li.hasPrevious() )") into a separate method and call QtConcurrent on that. > > Concerning parallel DB connection, there's no difference between server > and desktop use. This code is already in production use and should be > fairly robust already. > > I hope I didn't base this message on wrong assumptions and hope someone > of the server developers can shed more light onto this. thanks > > Regards > Matthias > > [1] https://github.com/qgis/QGIS/pull/2989/files > [2] > https://github.com/qgis/QGIS/pull/2989/commits/4cba6393544aeb23569d6c15f19e67f4ff8bf19c > > [snip] -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] [Server] performance questions
Hi Andreas, dear list, sorry for the warm up of this old topic. There is in deed a need to enable multi threading in Qgis-Server. In our company we use around 60 different layer to lay some topics on the map. This may be an extreme example to read, but quite common in our environment, i.e. we can not lower this number significantly. A warmed up QGis-Server process which already read and cached the Configuration file qgis-xy.conf still needs around 20 seconds to calculate the complete resulting map picture. During this calculation there is only one thread working, serializing the DB-requests, calculation the layer picture and mapping the picture on the resulting picture. The process is running on a multi core server. If we can utilize the idle cores to do useful work, this can be an easy example to speed up the process (significant). I'd like to share my developing experience to do this change, but I would need further information and some support to do some useful patches. I have scanned the git-Qgis sources (commit 81744ecf90a8f6c1e2e94fdacb07e3dca6987dcc) for a suitable multithreading entry. Found QgsMapRenderer::render() because of the line "while ( li.hasPrevious() )". It seems in this loop the layers are painted in serial onto the resulting picture. If I can build an auto parallel computing with QtConcurrent in this place there seems to be the most gain. I mean to parallelize the loop with concurrent DB requests, concurrent painting in separate pictures and in serial map the single pictures onto the result picture. Right now I don't know if I can do DB requests in parallel, or where the code for database requests is buried. And how about the calls inside the loop, are there side effects preventing multi threading? Do you got hints, flames or suggestions? cu Jörg Am 23.02.2016 um 16:39 schrieb Neumann, Andreas: > Hi Régis, > > QGIS server is single-threaded. You can't simply turn on multi-threaded > rendering in QGIS server. The other issue is that each Apache FCGI > process has it's own set of cache. So if you have 5-10 parallel Apache > threads or processes each thread/process has it's own cache. If your > client hits a thread that has the cache unititialized you will have wait > longer until this cache is fillled. > > I once asked Marco whether one could > > a) have a shared cache for all Apache threads or processes There is no shared cache, because Apache spawns QGis-Processes with its own memory layout. these Processes know nothing about its neighbor, and you have a hard time to exchange data between all processes. How about a scheduler, which assigns requests to warmed up Qgis-processes? > > b) support multi-threaded rendering > > But he said it would be very complicated to implement a) and b), if not > impossible. Maybe other devs have other views. Note that was about 2 > years ago, when I last discussed this with Marco. > > - > > Nevertheless I believe there would be room for performance improvements > in QGIS server - if interested parties share their efforts (be it dev > time or financial resources). > [snip] > > Andreas > [snip] > > > > > ___ > Qgis-developer mailing list > Qgis-developer@lists.osgeo.org > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer > -- mWerk GmbH Dipl.-Ing. Jörg Habenicht Landwehrstr. 76 30519 Hannover (T) +49 511 8033 (F) +49 511 8041 (E) j...@mwerk.net Amtsgericht Hannover HRB 206522 Geschäftsführer Reiner Brachvogel Dennis Kornehl ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
