Re: [Talk-GB] OSM UK's first tile layer
I have submitted a ticket to the JOSM developers. The ticket contains a fully worked-out patch to upgrade the EPSG:27700 projection from the Helmert transformation to the look-up table transformation. I'm afraid this is another somewhat long and technical message. But it does contain an explanation of why the Land Registry wms is actually sort-of correct. With my modified version of JOSM, .mif files are now transformed using the look-up table. This is with the hack I described before, where no projection is defined in the file, and you choose EPSG:27700 when the opendata plugin asks you. But .shp files which declare British National Grid are still transformed with the Helmert transformation. This must be down to the opendata plugin, because JOSM itself, as modified, does not know about the Helmert transformation any more. (I have put in a ticket for that, too, but without a patch.) I now understand better, the information in the EPSG registry. If you go to the page for projection 27700, and open the panel for the OSGB36 datum, it is not entirely clear, as I described before. But if you look at the page for the OSGB36 datum (transformation 7953), it spells out that this transformation uses the OSTN15 look-up table. However, transformation 7953 isn't referenced anywhere in the page for projection 27700. The more I look at this projection stuff, the worse it gets. The Land Registry only covers England and Wales. The Scottish counterpart is Registers of Scotland. I've also had a look at their opendata. The URL is a bit hard to find so I give it here https://ros.locationcentre.co.uk/inspire/ They too have a wms, which is here http://ros.datafeed.locationcentre.co.uk/geoserver/wms They offer .shp files in British National Grid and ETRS. I've already mentioned the issues raised by .shp files. Rob was hoping to compare the BNG and ETRS files, but that's not possible because Registers of Scotland have made a mistake in preparing the ETRS files. The BNG and ETRS files are identical. In other words, the ETRS file contains BNG coordinates in metres. I am a bit surprised that I might be the first to spot this and report it. (With shapefiles, the projection is defined in an accompanying .prj file. The .prj file uses a version of 'well-known text' which does not contain EPSG numbers. This will be part of the explanation for the issue with the opendata plugin. If the .prj file is missing, the only option with the opendata plugin is EPSG:4326 (WGS84), so removing the .prj file does not provide a workaround.) The Registers of Scotland wms is misaligned, just like the Land Registry one. Rob's examples align with the two wms. *But* I always launch JOSM from the command line. And I spotted an info message on the command line saying 'reprojecting from EPSG:27700'. So, a further discovery. Some background - We are familiar with online maps from various sources. Most of them use tms protocol with Web Mercator projection (EPSG:3857). Wms is different. The server offers the client a list of projections which it can deliver, and the client chooses which one to have. Suppose JOSM is set to EPSG:3857. The Land Registry wms does not offer EPSG:3857 so JOSM chooses the first projection which it understands, from the server's list - EPSG:27700. Then JOSM reprojects it, so the wms is misaligned because *JOSM* is using Helmert. And the Land Registry was right all along! The Registers of Scotland wms does offer EPSG:3857, so that is what JOSM chooses. And the wms is misaligned because the *wms server* is using Helmert. The wms also offers EPSG:27700. So if you set JOSM to EPSG:27700; and delete and re-add the wms layer so it is redownloaded in EPSG:27700; then the wms is misaligned because *JOSM* is using Helmert. [I mean delete the layer, not delete the entry in the imagery list.] With my modified version of JOSM, both wms are correctly aligned (provided, in the case of Registers of Scotland, that you set JOSM to EPSG:27700 before adding the wms layer). The alignment of Rob's examples does not change with my modified version of JOSM, so Rob's examples no longer align with the two wms. I should add that the other projections offered by the two wms servers, in as far as I have been able to test them, are all misaligned. In other words, both servers are using Helmert. I tested with JOSM and QGIS. I've had another look at proj6 and proj7 and they do in fact use the look-up table for EPSG:27700. One piece I found online suggests they fall back to the Helmert transformation if the look-up table file is not available. This explains how come QGIS is using the look-up table transformation for EPSG:27700. I've looked further into the issue of simplification of polygons (dropping of nodes). GDAL and ogr2ogr behave the same as QGIS. ogr2ogr has a simplify option but that only increases the amount of simplification: it won't reduce it below the preset minimum. QGIS uses GDAL, so it look
Re: [Talk-GB] OSM UK's first tile layer
The test will be, if Rob is able to produce example areas processed with the OS look-up table transformation, do the misalignments go away? Sorry for a rather long and technical message. I've done some more investigation and testing. QGIS reckons EPSG:27700 is the OS look-up table transformation, while JOSM thinks it is the Helmert transformation. The ultimate authority is the EPSG registry https://epsg.org/crs_27700/OSGB-1936-British-National-Grid.html . Unfortunately that page is not entirely clear. But there is a reference to the OSTN15 grid file in a footnote, so I think EPSG:27700 is intended to be the look-up table transformation. So, apologies for saying previously that EPSG:27700 is the Helmert transformation. The work to incorporate a large number of projections into JOSM was done nearly five years ago. It is based on proj4. Both proj4 and proj5 have EPSG:27700 as the Helmert transformation (based on looking at the strings, and on the behaviour of JOSM). proj6 and proj7 have EPSG:27700 as the most basic transformation, which gives misalignments of over 100m (based on looking at the strings). By 'string', I mean the line of text that begins +proj=. Some file formats for geographic information (GIS files), can accommodate a range of projections. Most of these declare the projection near the beginning of the file. The Land Registry open data are such files and they declare EPSG:27700. If you process them with proj, or an app that uses proj, you are not going to get the right results unless you can override the declaration. The strange thing is that QGIS also uses proj. The developers of QGIS must have altered the definition of EPSG:27700 from the one built-in to proj. But I haven't discovered exactly what has been done. I set about loading some of the Land Registry open data into JOSM, with the look-up table transformation. With the opendata plugin, JOSM can read a range of GIS file formats. Unfortunately that does not include .gml, the format of the Land Registry files. The Land Registry suggest using QGIS, so I did. JOSM and QGIS have four formats in common, KML, geoJSON, Esri shapefile .shp, and MapInfo file .mif. I am a complete newbie to QGIS, but it is a nightmare! The option to disable projection handling (No CRS) does not work properly. (This would give a means of overriding the declaration in the file.) With three of the four formats for the output file, KML, .shp and .mif, QGIS simplified the polygons, weeding out nearly half of the nodes. QGIS gave no warning of this, and I could not find an option to turn off this behaviour. (There is an option to turn off this behaviour for rendering. Perhaps it would have turned it off for output too. But why were geoJSON files unaffected?) When writing a .mif file in EPSG:27700 projection, QGIS wrote the most basic transformation as the projection, without giving a warning. QGIS did this because the .mif format has limitations on the projection definitions that it can handle. Perhaps the latest version of QGIS is a bit buggy and I should have used the LTS version. I tried doing the transformation in QGIS, then loaded the output file into JOSM. All four file formats worked, and gave the same results (apart from the loss of nodes with three of the formats). So if you're using QGIS, I'd recommend doing it this way and using geoJSON. It doesn't even need the opendata plugin. If you want to do just the file format conversion in QGIS, and do the transformation in JOSM, it's more tricky. The KML and geoJSON formats are ruled out because they must by definition contain WGS 84 lats and longs. So you are stuck with the loss of nodes. The .shp format gives up to 5m misalignment because QGIS declares EPSG:27700 in the file and the opendata plugin provides no means for overriding the EPSG:27700 and using the custom projection I described previously. The .mif format works (in terms of getting no misalignment) if you do a simple hack. Open the .mif file in a text editor and change the fourth line from CoordSys Earth Projection 8, 79, "m", -2, 49, 0.9996012717, 40, -10 to CoordSys Nonearth Units "m" Bounds (0.0, 0.0) (130.0, 130.0) ensuring that none of the spaces are omitted. This means that no projection is defined in the file and the opendata plugin will then ask you which projection you want to use. You simply choose the custom projection, provided that you have previously set it up. (You may need to scroll down to see the Okay button.) The transformation in JOSM gives essentially the same results as the transformation in QGIS. For the newbie, here's how to do the transformation in QGIS. Launch QGIS. Create a new project. Then Layer > Add Layer > Add Vector Layer and open the .gml file you have downloaded from the Land Registry open data. Accept the suggested CRS of EPSG:27700 and close the Data Source Manager dialog. Then Layer > Save As > choose Format GeoJSON, filename as desired,
Re: [Talk-GB] OSM UK's first tile layer
I can confirm that the Land Registry wms parcels appear to have been converted with the Helmert 7-element transformation (no look-up table). This gives a misalignment of up to 5 metres. It's ironic that the Land Registry don't seem to know where their parcels are to better than 5m. Now we know what EPSG:27700 does. It does the above transformation. I agree with Rob that the misalignment of 5m is obvious if you look at Hugh Town (Scilly). Both if you compare with the OSM data and if you compare with the tracklogs that have been uploaded to OSM. So this transformation won't do. I think we need to go for the look-up table. I've done some testing with JOSM. The look-up table transformation is not in JOSM's list of (thousands) of projections. But this custom projection does it: +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=40 +y_0=-10 +ellps=airy +units=m +nadgrids=OSTN02_NTv2.gsb +bounds=-9,49,2,61 +no_defs I expect something very similar would work in Mapnik. When you set up this custom projection, JOSM downloads the grid file from the JOSM server, and puts it in the JOSM cache folder under a modified name. There is then a wait of several seconds while JOSM configures the custom projection. You can also get JOSM to do the latest, OSTN15 transformation. The only change needed, is to the grid file. This needs some simple hacking because it's not supported. You don't change the custom projection, but you alter the file in the cache folder. So, find the file, copy its name, and then delete it. Download the OSTN15 grid file from the OS website. As Gareth says, you need OSTN15_NTv2_OSGBtoETRS.gsb, (and not the other way round). Put the file in the cache folder and rename it to the name you just copied. You then need to quit and relaunch JOSM for this change to 'take'. The difference between OSTN02 and OSTN15 is a shift, mostly in longitude, and in a similar direction throughout GB, of 1-2cm. With the look-up table transformation, there will still be a misalignment of 0.75m relative to WGS84, but this is a lot better than 5m. If there is consensus, then the wiki needs to be updated to recommend the OSTN15 transformation. ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] OSM UK's first tile layer
The Ordnance Survey provides a transformation between OSGB36 and ETRS. It is described on this page https://www.ordnancesurvey.co.uk/gps/transformation/ and on the pages linked from there. The transformation is definitive. In other words, OSGB36 is redefined as being what you get when you apply the transformation to sets of ETRS co-ordinates. This must mean that if you compare an OS 1:1250 National Grid plan, with an older version of the plan from the era of OSTN02, features may have shifted slightly. The transformation involves a mathematical transformation, and an adjustment based on a look-up table, to make the result match the errors in the old triangulation system. The OS provides applications to do the transformation, both ways, for a range of platforms. It also provides source code, the look-up table, and details of the mathematical transformation. JOSM handles projections using proj. If you want to know what JOSM does with EPSG:27700, you need to know how it is defined in proj. The source code of JOSM includes the OSTN02 look-up table (15MB), but it can't be in the jar (also 15MB), so I don't know how that works. Rob asked about position errors from the Helmert transformation without a look-up table. Here are some examples. Larger errors Place error, m St Kilda 4.9 Scilly 4.7 Lizard Point 4.1 Butt of Lewis 3.2 King's Lynn2.7 Mallaig2.6 Flamborough Head 2.4 Colchester 2.4 Plymouth 2.4 Nottingham 2.3 Anglesey 2.1 Northampton2.0 North Foreland 1.9 Isle of Man S 1.9 Carmarthen 1.9 Smaller errors St Catherine's Pt 1.4 Carlisle 0.8 Edinburgh 0.6 Aberdeen 1.8 Thurso 1.6 Orkney 1.0 Foula (Shetland) 1.2 The errors are particularly small near Bristol, Edinburgh and Fair Isle. They exceed 2m in South Devon, Cornwall, East Anglia, Nottinghamshire, Lincolnshire, the East Riding of Yorkshire, Pembrokeshire, Anglesey and Western Scotland. +1 for referencing GB to ETRS. ___ Talk-GB mailing list Talk-GB@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk-gb
Re: [Talk-GB] Finally a RTK / NTRIP Broadcaster in London
@ Grant Slater
Thank you for spotting this and making it known.
One thing you need to know when using an RTK base station, is what is the
reference (datum). Unfortunately there is no convention for how this
information is given, and it is often not given at all. The stations listed by
Grant are on the Eurasian tectonic plate so the reference will be either ETRS
or ITRS ("WGS 84"). WGS 84 is the datum used in OSM.
I have previously connected to several of the stations on the list: DARE, HERS,
HERT and SHOE. All of these are on ETRS although HERS has a small position
error. I tried connecting to LICC and it is on ITRS. (I estimate there is a
position error, relative to ITRS, around 12cm too far north, 8cm too far west
and 31cm too high.) I think a mistake has been made in configuring the LICC
station. Incidentally, LICC is at Imperial College in South Kensington. There
is a site information page at
http://epncb.oma.be/_networkdata/siteinfo4onestation.php?station=LICC00GBR If
you click on Data Provided you can see any warnings that have been logged. It
was the warning here about a position error that prompted me to check it out.
The website I just referred to evidently expects the reference to be ETRS.
The stations on Grant's list are of a type known as Continuously Operating
Reference Stations (CORS). Stations of that type would be expected to produce
results consistent to the millimetre. The ITRS position of LICC shifts by a
millimetre every two weeks, so I hope they have an automated system, or at
least a simple system, for updating the position it is broadcasting.
If you have a consumer SatNav you probably can't tell the difference between
ETRS and ITRS. But if you are using RTK you certainly can tell the difference.
In south-east England, at the time of writing, ITRS gives a position 52.0cm
further north and 53.5cm further east than ETRS. This gives a horizontal
distance of 74.6cm. The horizontal distance increases by 2.4cm per year. The
altitude difference is less than 0.2cm.
If you record a tracklog using RTK from seven of the eight stations in the
list, the tracklog will be in ETRS. You will need to convert it to ITRS for use
in OSM. If the tracklog covers a small area, you can just apply a fixed offset
to the latitudes and longitudes. Unfortunately I don't know of any tool which
makes this simple to do.
Someone in France has organised funding for an independent network of open RTK
base stations. (The availability of free RTK base stations was even worse in
France than in the UK.) See https://centipede.fr/ They have produced detailed
instructions for setting up a base station, including a shopping list, how to
assemble the equipment, preconfigured software, and how to obtain the position
of the base station to within a couple of centimetres. It also covers setting
up a receiver for RTK. They have set up a server to broadcast all the streams
and there are already several dozen stations in operation. They will soon be
prevented from setting up a VRS, only by the cost of the software for doing
that. The documentation is all at https://jancelin.github.io/docs-centipedeRTK/
It is in French, but for those who don't speak French, I expect a well-known
online service would produce an adequate translation. There is a subscription
RTK stream service covering many countries, which professionals use. They no
longer quote prices on their web site, but when they did, the entry-level
subscription for real-time access was around £4000 per year. IIRC, that gave
the subscriber access to a maximum of five simultaneous VRS streams and
included access to a two-way satellite link, for areas where there was no
mobile phone signal.
___
Talk-GB mailing list
Talk-GB@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-gb
