Folks, I think we may be taking about different things, the wfs requests below show the problem we are seeing.
There are 2 cities in each of the sqlite databases and the only difference is on has the text 'and' and the other has the symbol '&' Layer DATA_TEST - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio and Paulo'); http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST Layer DATA_TEST2 - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio & Paulo'); http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST2 So our question is do the strings in database need to be xml safe or does mapserver check that the strings are safe before serving. Thanks Ian -----Original Message----- From: Even Rouault [mailto:even.roua...@spatialys.com] Sent: Friday, April 15, 2016 9:15 AM To: Ian Walberg <ian.walb...@airborne.aero> Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars Le vendredi 15 avril 2016 17:32:36, Ian Walberg a écrit : > If I understand this we would not get the issue if we use a WFS > version > 1.0.0 request. > > We have just tested that and it appears that changing the version does > not change the behavior. > > This is with mapserver version 7.0.1. We are probably talking about different things. In https://github.com/mapserver/mapserver/commit/105877468590e99401d234e1bbac13d7645b77d0 , I have just fixed issues in GetCapabilities response generation, but your issues might be different. I've also just tested reading from a CSV (the input datastore type shouldn't matter) file a field with & and it is properly escaped as & in the GML output of WFS 1.0, 1.1 and 2.0 Without a mapfile + data + request you issue, it is difficult to be sure we're talking about the same thing. > > Thanks > > Ian > > -----Original Message----- > From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] > On Behalf Of Even Rouault Sent: Friday, April 15, 2016 7:50 AM > To: mapserver-users@lists.osgeo.org > Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars > > Le vendredi 15 avril 2016 16:41:28, Richard Watson a écrit : > > I am using Map Server 6.4.1. I have a map project file which has > > the > > > > following title: > > WEB > > > > METADATA > > > > [ text omitted] > > "wms_title" "(*#(@#*$_)@#&\\\///" > > > > END # METADATA > > > > END # WEB > > > > When a GetCapabilties is done on the project then this is the source > > XML which is returned: > > > > <Service> > > <Name>WMS</Name> > > <Title>(*#(@#*$_)@#&\\///</Title> > > It seems to be escaping as required. > > WMS and WFS 1.0 use MapServer manual XML composition with its own XML > escaping routines. Some other services (such as WFS 1.1 & 2.0) use > libxml (sometimes not in an appropriate way). > > > On Fri, Apr 15, 2016 at 9:21 AM, Rahkonen Jukka (MML) < > > > > jukka.rahko...@maanmittauslaitos.fi> wrote: > > > Hi, > > > > > > > > > > > > I was thinking that the output of GetCapabilities is XML and & > > > must be escaped in XML. So if I write & into the abstract and > > > Mapserver works correctly it should write my test abstract “test ä > > > &” escaped perhaps as > > > > > > test ä & > > > > > > That is the output from the online tool at > > > http://www.freeformatter.com/xml-escape.html#ad-output > > > > > > > > > > > > So my conclusion was that because Mapserver does not do it right > > > here it won’t do it right elsewhere. > > > > > > > > > > > > But now I made another test by editing the Geoserver test file > > > “states.shp” in Spatialite as > > > > > > UPDATE states set STATE_NAME=’Colo&rado’ WHERE > > > STATE_NAME=’Colorado’; > > > > > > > > > > > > Output of GetFeature from WFS 1.1.0 from Mapserver 7.0.0 includes > > > > > > <ms:STATE_NAME>Colo&rado</ms:STATE_NAME> > > > > > > So my current conclusion is that my Mapserver does not escape “&” > > > correctly when it is used inside a mapfile and when I do > > > GetCapabilities request. However, when data come from Spatialite > > > database and they are consumed by GetFeature then escaping happens > > > correctly. > > > > > > > > > > > > We are still comparing different things because you have PostGIS > > > and I have Spatialite. > > > > > > > > > > > > -Jukka Rahkonen- > > > > > > > > > > > > > > > > > > Ian Walberg wrote: > > > > > > > > > > > > Jukka, > > > > > > > > > > > > I am not sure I understand the test you did. > > > > > > > > > > > > The data we are referring to is from the database field for the > > > individual features name. > > > > > > > > > > > > Regards > > > > > > > > > > > > Ian > > > > > > > > > > > > *From:* Rahkonen Jukka (MML) > > > [mailto:jukka.rahko...@maanmittauslaitos.fi > > > <jukka.rahko...@maanmittauslaitos.fi>] > > > *Sent:* Thursday, April 14, 2016 10:19 PM > > > *To:* Ian Walberg <ian.walb...@airborne.aero>; MapserverList OSGEO > > > < mapserver-users@lists.osgeo.org> > > > *Subject:* Re: WFS layer - XML/JSON safe chars > > > > > > > > > > > > Hi Ian, > > > > > > > > > > > > Thanks for the clarification. > > > > > > I made a simple test by adding an abstract into layer metadata. > > > This shows “test ä” as an abstract > > > > > > "wfs_abstract" "test ä” > > > > > > > > > > > > This one shows an empty abstract > > > > > > "wfs_abstract" "test ä &" > > > > > > > > > > > > So it looks like Mapserver doesn’t know to convert “&” into “&” > > > for XML. > > > > > > > > > > > > -Jukka Rahkonen- > > > > > > > > > > > > Ian Walberg wrote: > > > > > > > > > > > > Jukka, > > > > > > > > > > > > Our question was for the data/text read from the database and then > > > published as XML by the WFS service. > > > > > > > > > > > > So a pub with the name ‘Dog & bone’ created illegal xml as the & > > > is not converted to & > > > > > > > > > > > > Regards > > > > > > > > > > > > Ian > > > > > > > > > > > > *From:* Rahkonen Jukka (MML) > > > [mailto:jukka.rahko...@maanmittauslaitos.fi > > > <jukka.rahko...@maanmittauslaitos.fi>] > > > *Sent:* Thursday, April 14, 2016 3:10 PM > > > *To:* Ian Walberg <ian.walb...@airborne.aero>; MapserverList OSGEO > > > < mapserver-users@lists.osgeo.org> > > > *Subject:* Re: WFS layer - XML/JSON safe chars > > > > > > > > > > > > Hi, > > > > > > > > > > > > You can’t have & in feature type names or attribute names in WFS > > > because they are defined in the XML schema to be “QName”. The > > > prefix and LocalPart are of type NCName > > > https://en.wikipedia.org/wiki/QName > > > and from > > > http://stackoverflow.com/questions/1631396/what-is-an-xsncname-typ > > > e- > > > and-w > > > hen-should-it-be-used > > > > > > > > > > > > The practical restrictions of NCName are that it cannot contain > > > several symbol characters like :, @, $, %, &, /, +, ,, ;, > > > whitespace characters or different parenthesis. Furthermore an > > > NCName cannot begin with a number, dot or minus character although > > > they can appear later in an NCName. > > > > > > > > > > > > -Jukka Rahkonen- > > > > > > > > > > > > *Lähettäjä:* mapserver-users [ > > > mailto:mapserver-users-boun...@lists.osgeo.org > > > <mapserver-users-boun...@lists.osgeo.org>] *Puolesta *Ian Walberg > > > *Lähetetty:* 14. huhtikuuta 2016 19:23 > > > *Vastaanottaja:* MapserverList OSGEO > > > *Aihe:* [mapserver-users] WFS layer - XML/JSON safe chars > > > > > > > > > > > > Folks, > > > > > > > > > > > > Can mapserver produce XML or JSON safe characters for WFS served > > > data or do we need to ensure the source data has the escaped values. > > > > > > > > > > > > We got tripped up today as there were some ‘&’ in our names which > > > we had not considered. > > > > > > > > > > > > Thanks > > > > > > > > > > > > Ian > > > > > > _______________________________________________ > > > mapserver-users mailing list > > > mapserver-users@lists.osgeo.org > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users > > -- > Spatialys - Geospatial professional services http://www.spatialys.com > _______________________________________________ mapserver-users > mailing list mapserver-users@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/mapserver-users -- Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users