Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[David Marteau]

Ticket filled https://issues.qgis.org/issues/17866

> Le 16 janv. 2018 à 09:18, David Marteau  a écrit :
> 
> Hi Alessandro
> 
> This is a weird bug, I have also found that once the first request failed, 
> others requests succeed wathever the url or the provider is (tried with wcs 
> and wms).
> When failing, looking on what append wms server  side when we found that no 
> request is sent.
> 
> I fill a ticket on this.
> 
> 
>>> The point is: trying to instanciate  a  wmsprovider will issue a timeout 
>>> when trying to request the capabilities: the code responsible for that 
>>> creates its own QEventLoop and manage to do a blocking call (which is 
>>> expected here). 
>>> Please have a look at 
>>> https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
>>>  
>>> .
>>>   What we do not understand is why, despite the call of loop.exec() in the 
>>> c++ code, the request is not processed (my example with the  
>>> Q(gs)etworkAccessManager
>>> shows that it should processed)
>>> 
>>> Actually you cannot instanciate properly a QgsProject with a WMS layer.
>>> 
>>> To convince yourself, try to instanciate a QgsProject in python (in a 
>>> standalone python script) from a .qgs project having a single wms layer: it 
>>> will fail to instanciate the layer.
>>> 
>> 
>> 
>> Hi David, 
>> 
>> I verified this issue, I'm sorry but I don't have an explanation right away 
>> and it looks like a bug, please file a ticket.
>> 
>> 
>> -- 
>> Alessandro Pasotti
>> w3:   www.itopen.it 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[David Marteau]

Hi Alessandro

This is a weird bug, I have also found that once the first request failed, 
others requests succeed wathever the url or the provider is (tried with wcs and 
wms).
When failing, looking on what append wms server  side when we found that no 
request is sent.

I fill a ticket on this.


>> The point is: trying to instanciate  a  wmsprovider will issue a timeout 
>> when trying to request the capabilities: the code responsible for that 
>> creates its own QEventLoop and manage to do a blocking call (which is 
>> expected here). 
>> Please have a look at 
>> https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
>>  
>> .
>>   What we do not understand is why, despite the call of loop.exec() in the 
>> c++ code, the request is not processed (my example with the  
>> Q(gs)etworkAccessManager
>> shows that it should processed)
>> 
>> Actually you cannot instanciate properly a QgsProject with a WMS layer.
>> 
>> To convince yourself, try to instanciate a QgsProject in python (in a 
>> standalone python script) from a .qgs project having a single wms layer: it 
>> will fail to instanciate the layer.
>> 
> 
> 
> Hi David, 
> 
> I verified this issue, I'm sorry but I don't have an explanation right away 
> and it looks like a bug, please file a ticket.
> 
> 
> -- 
> Alessandro Pasotti
> w3:   www.itopen.it 
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[Alessandro Pasotti]

On Mon, Jan 15, 2018 at 6:27 PM, David Marteau  wrote:

> Here is a test script that illustrate the problem a bit further: if we run
> createProvider() twice, the first time
> the getCapbabilities will fail with a timeout. The second time, it will
> succeed.
>
> If you run this script, clear the Qgis network cache between two
> invocations.
>
> ===8<==
> import os
> import logging
>
> from qgis.core import QgsApplication, QgsMessageLog
> from qgis.core import QgsProject, QgsProviderRegistry
>
> logger = logging.getLogger()
> logger.setLevel(logging.DEBUG)
>
> # No Display
> os.environ['QT_QPA_PLATFORM'] = 'offscreen'
>
> qgis_application = QgsApplication([], False )
> qgis_application.setPrefixPath('/usr/local', True)
> qgis_application.initQgis()
>
> # Log any messages
> def writelogmessage(message, tag, level):
> arg = '{}: {}'.format( tag, message )
> if level == QgsMessageLog.WARNING:
> logger.warning(arg)
> elif level == QgsMessageLog.CRITICAL:
> logger.error(arg)
> else:
> logger.info(arg)
>
> messageLog = QgsApplication.messageLog()
> messageLog.messageReceived.connect( writelogmessage )
>
> wmsuri = ("contextualWMSLegend=0&crs=EPSG:4326&dpiMode=7&
> featureCount=10&format=image/jpeg"
>   "&layers=s2cloudless&styles&tileMatrixSet=s2cloudless-
> wmsc-14"
>   "&url=http://tiles.maps.eox.at/wms?"; )
>
> # GetCapabilities Fail with Request Timeout !!!
> provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )
>
> # Run again => works !!!
> provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )
>
> qgis_application.exitQgis()
> del qgis_application
> 8<=
>
> We have also tested simple QgsNetwork invocation from c++: tests shows
> that event loops have the running status, but the
> request is not processed at the  first invocation.
>
> David
>
>
> Le 15 janv. 2018 à 15:05, David Marteau  a écrit :
>
>
>
>
> Hi David,
>
> You  need an event loop to use a Q(qg)NetworkAccessManager, it's
> asynchronous.
>
>
> Hi Alessandro,
>
> Yes, we know that (see the second code in previous mail  ):  the
>  QgsNetworkAccessManager works perfeclty in python, this is not the point.
>
> The point is: trying to instanciate  a  wmsprovider will issue a timeout
> when trying to request the capabilities: the code responsible for that
> creates its own QEventLoop and manage to do a blocking call (which is
> expected here).
> Please have a look at https://github.com/qgis/QGIS/blob/master/src/
> providers/wms/qgswmscapabilities.cpp#L1964.  What we do not understand is
> why, despite the call of loop.exec() in the c++ code, the request is not
> processed (my example with the  Q(gs)etworkAccessManager
> shows that it should processed)
>
> Actually you cannot instanciate properly a QgsProject with a WMS layer.
>
> To convince yourself, try to instanciate a QgsProject in python (in a
> standalone python script) from a .qgs project having a single wms layer: it
> will fail to instanciate the layer.
>
>
Hi David,

I verified this issue, I'm sorry but I don't have an explanation right away
and it looks like a bug, please file a ticket.


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[David Marteau]

Here is a test script that illustrate the problem a bit further: if we run 
createProvider() twice, the first time 
the getCapbabilities will fail with a timeout. The second time, it will 
succeed. 

If you run this script, clear the Qgis network cache between two invocations. 

===8<==
import os
import logging

from qgis.core import QgsApplication, QgsMessageLog
from qgis.core import QgsProject, QgsProviderRegistry

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# No Display
os.environ['QT_QPA_PLATFORM'] = 'offscreen'

qgis_application = QgsApplication([], False )
qgis_application.setPrefixPath('/usr/local', True)
qgis_application.initQgis()

# Log any messages
def writelogmessage(message, tag, level):
arg = '{}: {}'.format( tag, message )
if level == QgsMessageLog.WARNING:
logger.warning(arg)
elif level == QgsMessageLog.CRITICAL:
logger.error(arg)
else:
logger.info (arg)

messageLog = QgsApplication.messageLog()
messageLog.messageReceived.connect( writelogmessage )

wmsuri = 
("contextualWMSLegend=0&crs=EPSG:4326&dpiMode=7&featureCount=10&format=image/jpeg"
  "&layers=s2cloudless&styles&tileMatrixSet=s2cloudless-wmsc-14"
  "&url=http://tiles.maps.eox.at/wms? " )

# GetCapabilities Fail with Request Timeout !!!
provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )

# Run again => works !!!
provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )

qgis_application.exitQgis()
del qgis_application
8<=

We have also tested simple QgsNetwork invocation from c++: tests shows that 
event loops have the running status, but the 
request is not processed at the  first invocation.

David


> Le 15 janv. 2018 à 15:05, David Marteau  a écrit :
> 
>> 
>> 
>> 
>> Hi David, 
>> 
>> You  need an event loop to use a Q(qg)NetworkAccessManager, it's 
>> asynchronous.
>> 
> 
> Hi Alessandro,
> 
> Yes, we know that (see the second code in previous mail  ):  the  
> QgsNetworkAccessManager works perfeclty in python, this is not the point.
> 
> The point is: trying to instanciate  a  wmsprovider will issue a timeout when 
> trying to request the capabilities: the code responsible for that creates its 
> own QEventLoop and manage to do a blocking call (which is expected here). 
> Please have a look at 
> https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
>  
> .
>   What we do not understand is why, despite the call of loop.exec() in the 
> c++ code, the request is not processed (my example with the  
> Q(gs)etworkAccessManager
> shows that it should processed)
> 
> Actually you cannot instanciate properly a QgsProject with a WMS layer.
> 
> To convince yourself, try to instanciate a QgsProject in python (in a 
> standalone python script) from a .qgs project having a single wms layer: it 
> will fail to instanciate the layer.
> 
>> 
>> 
>> -- 
>> Alessandro Pasotti
>> w3:   www.itopen.it 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[David Marteau]

> 
> 
> 
> Hi David, 
> 
> You  need an event loop to use a Q(qg)NetworkAccessManager, it's asynchronous.
> 

Hi Alessandro,

Yes, we know that (see the second code in previous mail  ):  the  
QgsNetworkAccessManager works perfeclty in python, this is not the point.

The point is: trying to instanciate  a  wmsprovider will issue a timeout when 
trying to request the capabilities: the code responsible for that creates its 
own QEventLoop and manage to do a blocking call (which is expected here). 
Please have a look at 
https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
 
.
  What we do not understand is why, despite the call of loop.exec() in the c++ 
code, the request is not processed (my example with the  
Q(gs)etworkAccessManager
shows that it should processed)

Actually you cannot instanciate properly a QgsProject with a WMS layer.

To convince yourself, try to instanciate a QgsProject in python (in a 
standalone python script) from a .qgs project having a single wms layer: it 
will fail to instanciate the layer.

> 
> 
> -- 
> Alessandro Pasotti
> w3:   www.itopen.it 
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[Alessandro Pasotti]

Forgot to mention that some time ago I wrote a python class to handle all
this boring stuff for QGIS 2:

https://github.com/boundlessgeo/lib-qgis-commons/blob/master/qgiscommons2/network/networkaccessmanager.py

On Mon, Jan 15, 2018 at 2:48 PM, Alessandro Pasotti 
wrote:

> On Mon, Jan 15, 2018 at 12:48 PM, David Marteau  wrote:
>
>> Hi,
>>
>> We hit a very nasty problem when trying to load projects with wms layer
>> defined from python: loading capabilities fail with timeout error, thus
>> preventing creating any project or layer from python.
>>
>> The problem has been verified in Debian, Ubuntu and fresh OSX build from
>> master branch.
>>
>> We have somehow been able to reduce the problem to the loading of the
>> wmsprovider. The following python code provide a minimal example for
>> reproducing the problem:
>>
>> ===8<==
>> import os
>> import logging
>>
>> from qgis.core import QgsApplication, QgsMessageLog
>> from qgis.core import QgsProject, QgsProviderRegistry
>>
>> logger = logging.getLogger()
>> logger.setLevel(logging.DEBUG)
>>
>> # No Display
>> os.environ['QT_QPA_PLATFORM'] = 'offscreen'
>>
>> qgis_application = QgsApplication([], False )
>> qgis_application.setPrefixPath('/usr/local', True)
>> qgis_application.initQgis()
>>
>> # Log any messages
>> def writelogmessage(message, tag, level):
>> arg = '{}: {}'.format( tag, message )
>> if level == QgsMessageLog.WARNING:
>> logger.warning(arg)
>> elif level == QgsMessageLog.CRITICAL:
>> logger.error(arg)
>> else:
>> logger.info(arg)
>>
>> messageLog = QgsApplication.messageLog()
>> messageLog.messageReceived.connect( writelogmessage )
>>
>>
>> wmsuri = ("contextualWMSLegend=0&crs=EPSG:4326&dpiMode=7&featureCount
>> =10&format=image/jpeg"
>>   "&layers=s2cloudless&styles&tileMatrixSet=s2cloudless-wm
>> sc-14"
>>   "&url=http://tiles.maps.eox.at/wms?"; )
>>
>> # Request Timeout
>> provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )
>>
>> qgis_application.exitQgis()
>> del qgis_application
>> 8<=
>>
>> This fail with the following qgis errors
>>
>> WMS: Download of capabilities failed: Operation canceled
>> Network: Network request https://tiles.maps.eox
>> .at/wms?SERVICE=WMS&REQUEST=GetCapabilities timed out
>>
>> Note that the following piece of code work perfectly and proceed the
>> request without problems:
>>
>> ==
>> from PyQt5.QtCore import QUrl, QEventLoop
>> from PyQt5.QtNetwork import QNetworkRequest
>>
>> url = QUrl("https://tiles.maps.eox.at/wms?SERVICE=WMS&REQUEST=GetC
>> apabilities")
>>
>> request = QNetworkRequest()
>> request.setUrl(url)
>> manager = QgsNetworkAccessManager.instance()
>> replyObject = manager.get(request)
>>
>> loop = QEventLoop()
>> def onfinish(  ):
>> print("Qt Return Code:",replyObject.attribute(Q
>> NetworkRequest.HttpStatusCodeAttribute))
>> answer = replyObject.readAll();
>> print(answer[:200],"\n")
>> loop.quit()
>>
>> replyObject.finished.connect( onfinish )
>> loop.exec()
>> ==
>>
>> So this is not fundamentally a problem with the network manager.
>>
>> At this point we are stuck because the provider do not do something
>> fundamentally different from the last piece of code,
>>
>> In qgis code the request execution stall at: https://github.com/qgis/QG
>> IS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
>>
>> The problem does not occur if we run the project from the Desktop
>> application: we think that there is a problem with the QEventLoop execution
>> but we have no clue atm to check this assumption.
>>
>> So if anybody have any idea on the subject…...
>>
>
>
> Hi David,
>
> You  need an event loop to use a Q(qg)NetworkAccessManager, it's
> asynchronous.
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[Alessandro Pasotti]

On Mon, Jan 15, 2018 at 12:48 PM, David Marteau  wrote:

> Hi,
>
> We hit a very nasty problem when trying to load projects with wms layer
> defined from python: loading capabilities fail with timeout error, thus
> preventing creating any project or layer from python.
>
> The problem has been verified in Debian, Ubuntu and fresh OSX build from
> master branch.
>
> We have somehow been able to reduce the problem to the loading of the
> wmsprovider. The following python code provide a minimal example for
> reproducing the problem:
>
> ===8<==
> import os
> import logging
>
> from qgis.core import QgsApplication, QgsMessageLog
> from qgis.core import QgsProject, QgsProviderRegistry
>
> logger = logging.getLogger()
> logger.setLevel(logging.DEBUG)
>
> # No Display
> os.environ['QT_QPA_PLATFORM'] = 'offscreen'
>
> qgis_application = QgsApplication([], False )
> qgis_application.setPrefixPath('/usr/local', True)
> qgis_application.initQgis()
>
> # Log any messages
> def writelogmessage(message, tag, level):
> arg = '{}: {}'.format( tag, message )
> if level == QgsMessageLog.WARNING:
> logger.warning(arg)
> elif level == QgsMessageLog.CRITICAL:
> logger.error(arg)
> else:
> logger.info(arg)
>
> messageLog = QgsApplication.messageLog()
> messageLog.messageReceived.connect( writelogmessage )
>
>
> wmsuri = ("contextualWMSLegend=0&crs=EPSG:4326&dpiMode=7&
> featureCount=10&format=image/jpeg"
>   "&layers=s2cloudless&styles&tileMatrixSet=s2cloudless-
> wmsc-14"
>   "&url=http://tiles.maps.eox.at/wms?"; )
>
> # Request Timeout
> provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )
>
> qgis_application.exitQgis()
> del qgis_application
> 8<=
>
> This fail with the following qgis errors
>
> WMS: Download of capabilities failed: Operation canceled
> Network: Network request https://tiles.maps.eox.at/wms?SERVICE=WMS&;
> REQUEST=GetCapabilities timed out
>
> Note that the following piece of code work perfectly and proceed the
> request without problems:
>
> ==
> from PyQt5.QtCore import QUrl, QEventLoop
> from PyQt5.QtNetwork import QNetworkRequest
>
> url = QUrl("https://tiles.maps.eox.at/wms?SERVICE=WMS&REQUEST=
> GetCapabilities")
>
> request = QNetworkRequest()
> request.setUrl(url)
> manager = QgsNetworkAccessManager.instance()
> replyObject = manager.get(request)
>
> loop = QEventLoop()
> def onfinish(  ):
> print("Qt Return Code:",replyObject.attribute(QNetworkRequest.
> HttpStatusCodeAttribute))
> answer = replyObject.readAll();
> print(answer[:200],"\n")
> loop.quit()
>
> replyObject.finished.connect( onfinish )
> loop.exec()
> ==
>
> So this is not fundamentally a problem with the network manager.
>
> At this point we are stuck because the provider do not do something
> fundamentally different from the last piece of code,
>
> In qgis code the request execution stall at: https://github.com/qgis/
> QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
>
> The problem does not occur if we run the project from the Desktop
> application: we think that there is a problem with the QEventLoop execution
> but we have no clue atm to check this assumption.
>
> So if anybody have any idea on the subject…...
>


Hi David,

You  need an event loop to use a Q(qg)NetworkAccessManager, it's
asynchronous.



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Cannot load WMS capabilities from WMS provider in QGIS3 from python

[David Marteau]

Hi,

We hit a very nasty problem when trying to load projects with wms layer defined 
from python: loading capabilities fail with timeout error, thus preventing 
creating any project or layer from python.

The problem has been verified in Debian, Ubuntu and fresh OSX build from master 
branch.

We have somehow been able to reduce the problem to the loading of the 
wmsprovider. The following python code provide a minimal example for 
reproducing the problem:

===8<==
import os
import logging

from qgis.core import QgsApplication, QgsMessageLog
from qgis.core import QgsProject, QgsProviderRegistry

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# No Display
os.environ['QT_QPA_PLATFORM'] = 'offscreen'

qgis_application = QgsApplication([], False )
qgis_application.setPrefixPath('/usr/local', True)
qgis_application.initQgis()

# Log any messages
def writelogmessage(message, tag, level):
arg = '{}: {}'.format( tag, message )
if level == QgsMessageLog.WARNING:
logger.warning(arg)
elif level == QgsMessageLog.CRITICAL:
logger.error(arg)
else:
logger.info(arg)

messageLog = QgsApplication.messageLog()
messageLog.messageReceived.connect( writelogmessage )


wmsuri = 
("contextualWMSLegend=0&crs=EPSG:4326&dpiMode=7&featureCount=10&format=image/jpeg"
  "&layers=s2cloudless&styles&tileMatrixSet=s2cloudless-wmsc-14"
  "&url=http://tiles.maps.eox.at/wms?"; )

# Request Timeout
provider = QgsProviderRegistry.instance().createProvider( "wms", wmsuri )

qgis_application.exitQgis()
del qgis_application
8<=

This fail with the following qgis errors

WMS: Download of capabilities failed: Operation canceled
Network: Network request 
https://tiles.maps.eox.at/wms?SERVICE=WMS&REQUEST=GetCapabilities 
 timed out

Note that the following piece of code work perfectly and proceed the request 
without problems:

==
from PyQt5.QtCore import QUrl, QEventLoop
from PyQt5.QtNetwork import QNetworkRequest

url = QUrl("https://tiles.maps.eox.at/wms?SERVICE=WMS&REQUEST=GetCapabilities";)

request = QNetworkRequest()
request.setUrl(url)
manager = QgsNetworkAccessManager.instance()
replyObject = manager.get(request)

loop = QEventLoop()
def onfinish(  ):
print("Qt Return 
Code:",replyObject.attribute(QNetworkRequest.HttpStatusCodeAttribute))
answer = replyObject.readAll();
print(answer[:200],"\n")
loop.quit()

replyObject.finished.connect( onfinish )
loop.exec()
==

So this is not fundamentally a problem with the network manager.

At this point we are stuck because the provider do not do something 
fundamentally different from the last piece of code, 

In qgis code the request execution stall at: 
https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmscapabilities.cpp#L1964
 
The problem does not occur if we run the project from the Desktop application: 
we think that there is a problem with the QEventLoop execution but we have no 
clue atm to check this assumption.

So if anybody have any idea on the subject…...

Regards
David



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer