Re: [talk-au] Mapping houses and addresses in Sydney

[Dion Moult]

Thanks Andrew for your reply!

1. Thanks for the link to the import guidelines. My responses to the import 
guidelines below:

 1. I am aware that big automatic updates can cause problems. I will only 
import addr:housenumber and addr:street and a single node.
 2. Yep, sending to talk-au mailing list.
 3. I think we have the appropriate license.
 >=4. Looking for more feedback on talk-au first :)

2. Yes, you are absolutely right that this is not a huge automatic import - it 
relies on a human choosing what addresses to add and a human submitting it as a 
change. All it does it automate the address lookup and make sure that the node 
is neatly positioned at the correct location.
3. It looks like you're grabbing their entire dataset. That would be the 
alternative approach, doing a data dump, then importing that dump. This can 
import a lot more addresses, but is also much more complex. Is it worth 
pursuing? What do you reckon?
4. It seems odd that they would provide an API but would prevent anything from 
using it.
5. Looks like they are doing the big data import. See 3.

Dion Moult

‐‐‐ Original Message ‐‐‐
On June 17, 2018 9:57 PM, Andrew Harvey  wrote:

> I few thoughts:
>
> 1. If you're planning an Import, and you haven't already read up on 
> https://wiki.openstreetmap.org/wiki/Import/Guidelines
>
> 2. I like the idea of doing this semi-manually in JOSM. I'm against a 
> completely automated import because we have addresses already in OSM from 
> surveys which should be retained (and avoid having duplicates dumped on top).
>
> 3. I know you've got your scripts working already, but you still might be 
> interested in how we're trying to pull out addresses via the LPI Web Services 
> as CC BY 3.0 AU data for OpenAddresses.io at 
> https://github.com/openaddresses/openaddresses/pull/3977
>
> 4. We are still bound by the Terms and Conditions of using the web services 
> at 
> http://spatialservices.finance.nsw.gov.au/mapping_and_imagery/lpi_web_services/terms_and_conditions
>  specifically "2.1.6. use any robot, spider, site search/retrieval 
> application, or other device to retrieve or index any portion of the SWS". 
> I've asked LPI about this before and was told it was okay as long as usage 
> didn't affect other users, they were thankful that I asked first though. The 
> way you're using the API I think is okay, but best to play it safe and be 
> friendly with LPI on what we're doing in light of this clause.
>
> 5. The NZ community have been importing their national LINZ data, might be 
> worth reading up on 
> https://lists.openstreetmap.org/pipermail/imports/2017-June/005014.html to 
> see if there's anything we could learn from their work.
>
> On 17 June 2018 at 19:00, Dion Moult  wrote:
>
>> I had another look at the LPI resource and would like to propose a way of 
>> importing address data from LPI, because clearly sitting there for months on 
>> end armchair mapping the LPI Base Map as a raster underlay in JOSM is 
>> incredibly inefficient. Also, given that I am proposing an import, I would 
>> really appreciate community feedback to make sure I don't do anything wrong. 
>> First, I've looked at these two pages on usage of the LPI base map:
>>
>> https://wiki.openstreetmap.org/wiki/Contributors#Australia
>> https://wiki.openstreetmap.org/wiki/Attribution/New_South_Wales_Government_Data
>>
>> It seems to me that we can use any of the data from 
>> http://spatialservices.finance.nsw.gov.au/mapping_and_imagery/lpi_web_services/access_lpi_web_services
>>  to put data into OSM as long as we add it to that contributors page and 
>> note the LPI as a source. Specifically, we want to import a node that 
>> contains addr:housenumber and addr:street for every single property in NSW.
>>
>> Unfortunately it seems as though their API doesn't have any function to give 
>> a DB dump of addresses inside a bounding box. However it does have two 
>> functions, one which allows you to query a coordinate and it'll tell you 
>> what is the closest address at that point at a screen resolution (slightly 
>> archaic, yes), and another function which allows you to submit an address 
>> and it'll tell you what the government thinks is the centroid of the 
>> property with that address. So if you whack a bunch of points in JOSM that 
>> are within the bounds of a property based off the LPI NSW Base Map as a 
>> raster layer in JOSM, you can do Edit->Copy Coordinates in JOSM, dump it in 
>> a file, and use a script I wrote which will query the LPI web services to 
>> find the address, then query it again to find the proper centroid lat/long, 
>> then create a CSV of points and addresses which you can import into JOSM. 
>> Here's the script with sample input and sample output:
>>
>> https://gist.github.com/Moult/5821c74fb792b7afa5d758aebea68e40
>>
>> ... this is the changeset I produced with this small test sample:
>>
>> https://www.openstreetmap.org/changeset/59909707
>>
>> With this method I reckon you can quit

Re: [talk-au] Mapping houses and addresses in Sydney

[Andrew Harvey]

On 18 June 2018 at 19:21, Dion Moult  wrote:

> Thanks Andrew for your reply!
>
> 1. Thanks for the link to the import guidelines. My responses to the
> import guidelines below:
>

First up I think any changesets that import addresses in this way should
have an extra changeset tag so if we need to we can identify which
changesets did the import (so more than just source=LPI NSW Base Map).
Something like import=NSW Address Points or something.

I'm not sure about separating the address with a ";" like
https://www.openstreetmap.org/way/593297556/history#map=19/-33.78072/151.06688&layers=N,
could they not be two separate points? If it's a duplex, then I'd do it as
a single building with addr:housenumber=11, then if you want two nodes
inside the building for 11A and 11B.

While I don't think there's anything wrong with 2/18 as a first pass, eg
https://www.openstreetmap.org/node/5667899003, I think it's better to use
addr:unit=2 addr:housenumber=18.

 1. I am aware that big automatic updates can cause problems. I will only
> import addr:housenumber and addr:street and a single node.
>

What are you planning on doing where the address in already in OSM? I think
in this case we should just not import that point and leave the existing
OSM addresses.


> 2. Yes, you are absolutely right that this is not a huge automatic import
> - it relies on a human choosing what addresses to add and a human
> submitting it as a change. All it does it automate the address lookup and
> make sure that the node is neatly positioned at the correct location.
>


> 3. It looks like you're grabbing their entire dataset. That would be the
> alternative approach, doing a data dump, then importing that dump. This can
> import a lot more addresses, but is also much more complex. Is it worth
> pursuing? What do you reckon?
>

Oh I'm not suggesting that. It makes sense for the OpenAddresses project to
use a complete extract, but as you might have seen in the openaddresses
ticket there's a lot of problems trying to dump the data, so your approach
of doing it bit by bit should work much better for an OSM import.


> 4. It seems odd that they would provide an API but would prevent anything
> from using it.
> 5. Looks like they are doing the big data import. See 3.
>

Not quite, they did it using the approach you've described, broken it down
into pices and manually imported everything.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Mapping houses and addresses in Sydney

[Warin]

On 18/06/18 20:30, Andrew Harvey wrote:
On 18 June 2018 at 19:21, Dion Moult > wrote:


Thanks Andrew for your reply!

1. Thanks for the link to the import guidelines. My responses to
the import guidelines below:


First up I think any changesets that import addresses in this way 
should have an extra changeset tag so if we need to we can identify 
which changesets did the import (so more than just source=LPI NSW Base 
Map). Something like import=NSW Address Points or something.


source:import=LPI API via ?? something like that?



I'm not sure about separating the address with a ";" like 
https://www.openstreetmap.org/way/593297556/history#map=19/-33.78072/151.06688&layers=N, 
could they not be two separate points? If it's a duplex, then I'd do 
it as a single building with addr:housenumber=11, then if you want two 
nodes inside the building for 11A and 11B.
I have had separate buildings for A and B - share a common wall. In some 
instances I have 11 then 11A .. but no B.


While I don't think there's anything wrong with 2/18 as a first pass, 
eg https://www.openstreetmap.org/node/5667899003, I think it's better 
to use addr:unit=2 addr:housenumber=18.


 1. I am aware that big automatic updates can cause problems. I
will only import addr:housenumber and addr:street and a single node.


What are you planning on doing where the address in already in OSM? I 
think in this case we should just not import that point and leave the 
existing OSM addresses.
Depends .. I have come across addresses that were out of sequence. 
Contacted the still active mapper (moved to Germany) and had not 
response .. after some months I have simple deleted them.
So it is worth checking that the new data is is not 'better' than the 
present OSM data.


2. Yes, you are absolutely right that this is not a huge automatic
import - it relies on a human choosing what addresses to add and a
human submitting it as a change. All it does it automate the
address lookup and make sure that the node is neatly positioned at
the correct location.

3. It looks like you're grabbing their entire dataset. That would
be the alternative approach, doing a data dump, then importing
that dump. This can import a lot more addresses, but is also much
more complex. Is it worth pursuing? What do you reckon?


Oh I'm not suggesting that. It makes sense for the OpenAddresses 
project to use a complete extract, but as you might have seen in the 
openaddresses ticket there's a lot of problems trying to dump the 
data, so your approach of doing it bit by bit should work much better 
for an OSM import.


4. It seems odd that they would provide an API but would prevent
anything from using it.
5. Looks like they are doing the big data import. See 3.


Not quite, they did it using the approach you've described, broken it 
down into pices and manually imported everything.


It might be good to do one section and let people have a look at it?
I do think you'll find it repetitive. Maybe take a break and map 
something else for a while.


Good Luck.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Mapping houses and addresses in Sydney

[osm.talk-au]

From: Warin <61sundow...@gmail.com> 
Sent: Monday, 18 June 2018 20:57
To: talk-au@openstreetmap.org
Subject: Re: [talk-au] Mapping houses and addresses in Sydney

 

On 18/06/18 20:30, Andrew Harvey wrote:

On 18 June 2018 at 19:21, Dion Moult mailto:d...@thinkmoult.com> > wrote:

Thanks Andrew for your reply!

 

1. Thanks for the link to the import guidelines. My responses to the import 
guidelines below:

 

First up I think any changesets that import addresses in this way should have 
an extra changeset tag so if we need to we can identify which changesets did 
the import (so more than just source=LPI NSW Base Map). Something like 
import=NSW Address Points or something.


source:import=LPI API via ?? something like that?




 

Following the import guidelines and using a dedicated account for such imports 
should already make it clear?

 

(sorry for replying to the wrong mailing list first)

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Mapping houses and addresses in Sydney

[Dion Moult]

On June 18, 2018 8:56 PM, Warin <61sundow...@gmail.com> wrote:

> On 18/06/18 20:30, Andrew Harvey wrote:
>
>> On 18 June 2018 at 19:21, Dion Moult  wrote:
>>
>>> Thanks Andrew for your reply!
>>>
>>> 1. Thanks for the link to the import guidelines. My responses to the import 
>>> guidelines below:
>>
>> First up I think any changesets that import addresses in this way should 
>> have an extra changeset tag so if we need to we can identify which 
>> changesets did the import (so more than just source=LPI NSW Base Map). 
>> Something like import=NSW Address Points or something.
>
> source:import=LPI API via ?? something like that?

Sure thing, I would be happy to if that is the appropriate thing to do :)

>> I'm not sure about separating the address with a ";" like 
>> https://www.openstreetmap.org/way/593297556/history#map=19/-33.78072/151.06688&layers=N,
>>  could they not be two separate points? If it's a duplex, then I'd do it as 
>> a single building with addr:housenumber=11, then if you want two nodes 
>> inside the building for 11A and 11B.
>
> I have had separate buildings for A and B - share a common wall. In some 
> instances I have 11 then 11A .. but no B.

Thanks for the advice! I've fixed it to use two nodes. However, please note 
that that particular building was not mapped as part of my import script 
proposal. That was mapped previously by me completely manually. If I had used 
the import script it would have created two nodes, one for 11A and one for 11B.

>> While I don't think there's anything wrong with 2/18 as a first pass, eg 
>> https://www.openstreetmap.org/node/5667899003, I think it's better to use 
>> addr:unit=2 addr:housenumber=18.

Thanks :) I was wondering what was a better way of doing that. Fixed :) Again 
as above this was mapped manually by me and not using the script.

>>>  1. I am aware that big automatic updates can cause problems. I will only 
>>> import addr:housenumber and addr:street and a single node.
>>
>> What are you planning on doing where the address in already in OSM? I think 
>> in this case we should just not import that point and leave the existing OSM 
>> addresses.
>
> Depends .. I have come across addresses that were out of sequence. Contacted 
> the still active mapper (moved to Germany) and had not response .. after some 
> months I have simple deleted them.
> So it is worth checking that the new data is is not 'better' than the present 
> OSM data.

With my proposal of a semi-automated approach, every single new address will 
have to be explicitly decided upon by a human mapper. A human mapper can decide 
when to import the point if the existing data look bad (based off the LPI Base 
Map raster background) and when to leave the existing OSM addresses.

>>
>>
>>> 2. Yes, you are absolutely right that this is not a huge automatic import - 
>>> it relies on a human choosing what addresses to add and a human submitting 
>>> it as a change. All it does it automate the address lookup and make sure 
>>> that the node is neatly positioned at the correct location.
>>
>>> 3. It looks like you're grabbing their entire dataset. That would be the 
>>> alternative approach, doing a data dump, then importing that dump. This can 
>>> import a lot more addresses, but is also much more complex. Is it worth 
>>> pursuing? What do you reckon?
>>
>> Oh I'm not suggesting that. It makes sense for the OpenAddresses project to 
>> use a complete extract, but as you might have seen in the openaddresses 
>> ticket there's a lot of problems trying to dump the data, so your approach 
>> of doing it bit by bit should work much better for an OSM import.

Sounds good! Sorry for the misunderstanding :)

>>
>>
>>> 4. It seems odd that they would provide an API but would prevent anything 
>>> from using it.
>>> 5. Looks like they are doing the big data import. See 3.
>>
>> Not quite, they did it using the approach you've described, broken it down 
>> into pices and manually imported everything.
>
> It might be good to do one section and let people have a look at it?
> I do think you'll find it repetitive. Maybe take a break and map something 
> else for a while.
>
> Good Luck.

Yes, an incremental approach followed by regular review sounds good.___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au