Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Tue 2023-10-24 07:04:34 +1100, Andrew Harvey wrote:

> On Mon, 23 Oct 2023, 11:12 pm Andrew Harvey, 
> wrote:



>>> Do they say that the data is supposed to be
>>> updated weekly?


>> That order was set as a weekly recurring order.
> I just got the reccuring email for my orders, so this appears fixed now and
> they will be fresh.

Great, it seems to be reflected in response headers too:

--8<---cut here---start->8---
$ wget --server-response --spider 
https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_FDBZT5.zip
 2>&1 | grep Last-Modified

Last-Modified: Mon, 23 Oct 2023 21:54:54 GMT
--8<---cut here---end--->8---

and I'd guess this time could be just a few seconds before you received
the email.

Thanks for the details about the Vicmap addresses dataset. It seems less
important, but could you also provide the details of the Vicmap Property
data, currently at (last modified on last Wednesday 18th)

https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_EUKSRV.zip

?

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Mon, 23 Oct 2023, 11:12 pm Andrew Harvey, 
wrote:

>
>
>> Do they say that the data is supposed to be
>> updated weekly?
>>
>
> That order was set as a weekly recurring order.
>

I just got the reccuring email for my orders, so this appears fixed now and
they will be fresh.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Mon, 23 Oct 2023 at 00:25, Yuchen Pei  wrote:

> This is the URL
>
> https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_FDBZT5.zip
>
> Is it from the DELWP?


Yes


> Do they say that the data is supposed to be
> updated weekly?
>

That order was set as a weekly recurring order.


> Also, could you share the order details, including name, ID, Projection,
> Buffer, File Format and Area? It would help make the whole process more
> reproducible, starting from the source (the DELWP Vicmap Address page).
> Otherwise we wouldn't know how to get an url from DELWP serving the same
> kind of files.
>

Yes good point, I've added this detail to the README
https://gitlab.com/alantgeo/vicmap2osm/-/blob/master/README.md?ref_type=heads#vicmap-source-data
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-23 00:25:21 +1100, Yuchen Pei wrote:

> On Thu 2023-10-19 21:47:28 +1100, Andrew Harvey wrote:

>>  I found the process was only using Vicmap data from last year, so I've
>> updated that. The URL it's using should be serving new data every week but
>> I'm not sure it was not happening previously.

> This is the URL
> https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_FDBZT5.zip

> Is it from the DELWP? Do they say that the data is supposed to be
> updated weekly?

> Also, could you share the order details, including name, ID, Projection,
> Buffer, File Format and Area? It would help make the whole process more
> reproducible, starting from the source (the DELWP Vicmap Address page).
> Otherwise we wouldn't know how to get an url from DELWP serving the same
> kind of files.

Here's an order I made:

Name: Vicmap Address
ID : b9e9146d-8378-5c37-b6cd-63e3a8d05d02
Projection : Geographicals on GDA2020
Buffer : No buffer
File Format : ESRI File Geodatabase
Area : Whole dataset
Download URL: 
https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_9CFKI6.zip

Judging from the URL having the same pattern I assume the URL in the
Makefile is the original one.

Though I still don't know whether the specifications of this order
matches those of yours.

The only parameters I am not sure are Name and Buffer.

> [... 35 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Thu 2023-10-19 21:47:28 +1100, Andrew Harvey wrote:

>  I found the process was only using Vicmap data from last year, so I've
> updated that. The URL it's using should be serving new data every week but
> I'm not sure it was not happening previously.

This is the URL
https://s3.ap-southeast-2.amazonaws.com/cl-isd-prd-datashare-s3-delivery/Order_FDBZT5.zip

Is it from the DELWP? Do they say that the data is supposed to be
updated weekly?

Also, could you share the order details, including name, ID, Projection,
Buffer, File Format and Area? It would help make the whole process more
reproducible, starting from the source (the DELWP Vicmap Address page).
Otherwise we wouldn't know how to get an url from DELWP serving the same
kind of files.

> So we'll need to monitor this
> if the import is delayed further.

I'll do my best to make sure the project is not stalled again.

> [... 45 lines elided]

> I'm not a fan, and besides this mailing list is for mappers so it's not the
> place for detailed code review.

I assume a mapper is a person who helps improve the osm dataset, and by
this definition the work of this import is in scope.

> I'd rather if you can submit a Merge Request on GitLab (or Pull Request on
> GitHub).

As I mentioned before git is decentralised, so no one can force anyone
else to use a specific forge :)

> [... 100 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

 I found the process was only using Vicmap data from last year, so I've
updated that. The URL it's using should be serving new data every week but
I'm not sure it was not happening previously. So we'll need to monitor this
if the import is delayed further.

On Sat, 7 Oct 2023 at 20:48, Yuchen Pei  wrote:

> On a second thought, why don't we just generate the osc file with
>
> make dist/unitFromnumber.osc
>
> and apply the osc file manually? Of course that's assuming the file is
> correct. For example, to understand the discrepancy in the number of
> nodes I mentioned above. I also noticed some minor issues with the
> script, like when the number of changes exceeds 10k, it attempts to
> split them multiple files, but they are identical rather than sequential
> parts.
>

 I took another look into this, the code wasn't completed. The osc file
generated may include ways but doesn't include all the nodes for those ways
so it can't be uploaded from JOSM (I tried and it failed). One solution
would be to have the osc file generated to include them.

Correct it attempts to split the changes to meet the changeset element
limit, but this implementation was not completed, so while two files are
generated they don't have the limit applied. It looks like JOSM may be able
to do the splitting for us though, if we use JOSM to upload rather than my
code directly.



On Mon, 16 Oct 2023 at 22:30, Yuchen Pei  wrote:

> And there's always tried and true email based code review which we can
> do say in this mailing list, which both of we are already using. The
> person (say me) who wants to send a pull/merge request can create
> patches using git format-patch[1]. I can then send the patch to this
> mailing list, and then the person who reviews (say you) can comment
> inline, and I can respond to the comments inline, just like normal
> emails. After some back and forth once you are happy with the patches
> you can apply them to your repo with git am[2].
>
> [1] https://git-scm.com/docs/git-format-patch
> [2] https://git-scm.com/docs/git-am
>
> I got a few commits so far in my fork[3], and if you are open to doing
> it in this mailing list, I'll prepare some patches and send them here.
>

I'm not a fan, and besides this mailing list is for mappers so it's not the
place for detailed code review.

I'd rather if you can submit a Merge Request on GitLab (or Pull Request on
GitHub).

On Mon, 16 Oct 2023 at 20:53, Yuchen Pei  wrote:

> On Mon 2023-10-16 15:35:13 +1100, Andrew Harvey wrote:
>
> > On Mon, 16 Oct 2023 at 15:16, Yuchen Pei  wrote:
>
> >> On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:
>
> >> > On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:
>
> >> > [... 38 lines elided]
>
> >> > mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> >> > unit_number=x && street_number=y). For example, the Makefile rule
> >> > dist/unitFromNumber.osc is generated using this script. I have
> >> > generated
> >> > the osc file[1]. However, this file contains 38k nodes, whereas
> >> > the
> >> > input MR file[2] only has 12k features. So my question is - does
> >> > anyonw
> >> > know what is the easiest way to see all the changes in this file
> >> > staged
> >> > on a map, as a sanity check? OTOH I'd assume the file format is
> >> > some
> >> > standard osm change format.
>
> >> > Yes, this is mentioned in the README
>
> >> >> You can visualise the tag changes with bin/mrCoopDiff.js and
> >> > www/mrPreview.html at URL
>
> >> What is URL?
>
>
> > Ah yeah I put URL as a placeholder I was going to replace. At the moment
> > you can only view that locally within the repository as I haven't hosted
> > it.
>
> Got it, thanks. Somehow when I open www/mrPreview.html I only see a map,
> but not the addresses in the changes in Stage 2. I took a look at the
> content of the www/mrPreview.html, and I don't see references to any of
> the geojson or osc files.
>
> > [... 8 lines elided]
>
> >> > I can do the test upload of a small area (say ~100 addresses) and
> >> > report
> >> > back.
>
> >> > Please if you insist, can you just do < 10, no need to do 100.
>
> >> Why?
>
>
> > For testing, I feel it should be a number that we can manually work with.
> > After your testing either we need to then wait for the planet export to
> > catch up with your changes, or potentially have a conflict to deal with
> in
> > JOSM (or maybe JOSM would handle it, I'm not sure).
>
> OK.
>
> >> > Once I've reviewed the above and am happy with it I'll do the upload
> >> > with the import account as planned.
>
> >> Even though you asked whether I would like to help and I said yes, the
> >> communications so far have give me the impression that you want to work
> >> on it solo
>
>
> > Not the case, I think it's great that you've taken interest in the
> import,
> > if we can work together to get it done that would be ideal.
>
> Cool thanks. What do you think people can do to help speed up the
> 

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Tue, 17 Oct 2023 at 12:13, Phil Wyatt  wrote:

> Personally I think it should be suburb and postcode (drop the country)
>

https://gitlab.com/alantgeo/vicmap2osm#inclusion-of-addrsuburb-addrpostcode-and-addrstate

It was noted that there is not a consensus within the community, therefore
I opted to omit them as part of the import to remain neutral.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Phil Wyatt]

Personally I think it should be suburb and postcode (drop the country)

 

From: Warin <61sundow...@gmail.com> 
Sent: Monday, October 16, 2023 5:53 PM
To: Graeme Fitzpatrick ; Yuchen Pei 
Cc: talk-au@openstreetmap.org
Subject: Re: [talk-au] How to efficiently improve AU address coverage?

 

 

On 16/10/23 14:06, Graeme Fitzpatrick wrote:

Do we need the country, city & post code fields?

 

No. 

 

Thanks 

 

Graeme

 

 

On Mon, 16 Oct 2023 at 12:23, Yuchen Pei mailto:y...@gnu.org> > 
wrote:

On Tue 2023-10-03 19:51:13 +1100, Warin wrote:

> On 3/10/23 14:27, Andrew Harvey wrote:

> [... 12 lines elided]

> OK, what is needed to be done for "Stage 2 - Set unit from
> housenumber"?

> Further testing of the upload script. The changes themselves are
> pretty safe. It's using a custom uploader and if something isn't
> right it could make a mess. Sure the changeset could be reverted
> in the worst case scenario but you end up with more history so
> best to avoid this. I'll see if I can find some time to progress
> this further.

> Umm 'custom uploader' .. a file compatible with JOSM should be easy
> enough to create. Then selecting a small area to upload and test would
> be a simple manual operation, as would uploading the entire change
> set. 

The osc file[1] generated in Stage 2 is compatible with JOSM.

[1] https://ypei.org/assets/tmp/unitFromNumber-1.osc

I spot checked a few nodes and they look correct too. See also the
attached screenshot.


My understanding of this Stage is to fix all the discrepancies between
streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
before Stage 3 - uploading new addresses from the latter.

I can do the test upload of a small area (say ~100 addresses) and report
back.

I will check the scripts that generate this file, to find out whether
the logic indicates the file has full coverage corresponding to the
datasets.

> [... 5 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  <https://ypei.org/assets/ypei-pubkey.txt>
___
Talk-au mailing list
Talk-au@openstreetmap.org <mailto:Talk-au@openstreetmap.org> 
https://lists.openstreetmap.org/listinfo/talk-au

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 20:32:16 +1100, Yuchen Pei wrote:

> On Mon 2023-10-16 17:01:23 +1100, Andrew Harvey wrote:

>> That's okay. I created https://github.com/alantgeo/vicmap2osm as a mirror
>> of the GitLab code, happy to collaborate via Issues and Pull Requests there.

> Thanks, but Github is bad. It requires running proprietary javascript[1]
> to create issues, pull requests and perform other basic tasks for
> collaborations, and has many other issues[2] too, like suspected
> laundering copyleft code into copilot. Since you are creating a new
> mirror, why not choose a more ethical option[3] that respects user
> freedom, like codeberg[4], notabug[5], pagure[6], or sr.ht[7]?

> [1] https://www.gnu.org/philosophy/javascript-trap.html
> [2] https://sanctum.geek.nz/why-not-github.html
> [3] https://www.gnu.org/software/repo-criteria-evaluation.html
> [4] https://codeberg.org
> [5] https://notabug.org
> [6] https://pagure.io
> [7] https://sr.ht

And there's always tried and true email based code review which we can
do say in this mailing list, which both of we are already using. The
person (say me) who wants to send a pull/merge request can create
patches using git format-patch[1]. I can then send the patch to this
mailing list, and then the person who reviews (say you) can comment
inline, and I can respond to the comments inline, just like normal
emails. After some back and forth once you are happy with the patches
you can apply them to your repo with git am[2].

[1] https://git-scm.com/docs/git-format-patch
[2] https://git-scm.com/docs/git-am

I got a few commits so far in my fork[3], and if you are open to doing
it in this mailing list, I'll prepare some patches and send them here.

[3] https://g.ypei.me/vicmap2osm.git/log/

> [... 15 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 15:35:13 +1100, Andrew Harvey wrote:

> On Mon, 16 Oct 2023 at 15:16, Yuchen Pei  wrote:

>> On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:

>> > On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:

>> > [... 38 lines elided]

>> > mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
>> > unit_number=x && street_number=y). For example, the Makefile rule
>> > dist/unitFromNumber.osc is generated using this script. I have
>> > generated
>> > the osc file[1]. However, this file contains 38k nodes, whereas
>> > the
>> > input MR file[2] only has 12k features. So my question is - does
>> > anyonw
>> > know what is the easiest way to see all the changes in this file
>> > staged
>> > on a map, as a sanity check? OTOH I'd assume the file format is
>> > some
>> > standard osm change format.

>> > Yes, this is mentioned in the README

>> >> You can visualise the tag changes with bin/mrCoopDiff.js and
>> > www/mrPreview.html at URL

>> What is URL?


> Ah yeah I put URL as a placeholder I was going to replace. At the moment
> you can only view that locally within the repository as I haven't hosted
> it.

Got it, thanks. Somehow when I open www/mrPreview.html I only see a map,
but not the addresses in the changes in Stage 2. I took a look at the
content of the www/mrPreview.html, and I don't see references to any of
the geojson or osc files.

> [... 8 lines elided]

>> > I can do the test upload of a small area (say ~100 addresses) and
>> > report
>> > back.

>> > Please if you insist, can you just do < 10, no need to do 100.

>> Why?


> For testing, I feel it should be a number that we can manually work with.
> After your testing either we need to then wait for the planet export to
> catch up with your changes, or potentially have a conflict to deal with in
> JOSM (or maybe JOSM would handle it, I'm not sure).

OK.

>> > Once I've reviewed the above and am happy with it I'll do the upload
>> > with the import account as planned.

>> Even though you asked whether I would like to help and I said yes, the
>> communications so far have give me the impression that you want to work
>> on it solo


> Not the case, I think it's great that you've taken interest in the import,
> if we can work together to get it done that would be ideal.

Cool thanks. What do you think people can do to help speed up the
project? It is still not clear to me.

>> which is less of a problem if there's a clear timeframe to
>> complete it. After all it was started in 2021 and stalled, and this is a
>> community project to improve the VIC address coverage for everyone. To
>> get it done I will continue working on it as I have been in the past few
>> weeks. If Stage 2 is done I will push for progress in Stage 3, and so
>> on.


> It's a balance, I don't want to hold things up, at the same time we don't
> want to mess up such a large import by rushing.

Understood - I agree that we gotta do it right.

> [... 2 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 17:01:23 +1100, Andrew Harvey wrote:

> That's okay. I created https://github.com/alantgeo/vicmap2osm as a mirror
> of the GitLab code, happy to collaborate via Issues and Pull Requests there.

Thanks, but Github is bad. It requires running proprietary javascript[1]
to create issues, pull requests and perform other basic tasks for
collaborations, and has many other issues[2] too, like suspected
laundering copyleft code into copilot. Since you are creating a new
mirror, why not choose a more ethical option[3] that respects user
freedom, like codeberg[4], notabug[5], pagure[6], or sr.ht[7]?

[1] https://www.gnu.org/philosophy/javascript-trap.html
[2] https://sanctum.geek.nz/why-not-github.html
[3] https://www.gnu.org/software/repo-criteria-evaluation.html
[4] https://codeberg.org
[5] https://notabug.org
[6] https://pagure.io
[7] https://sr.ht

> [... 20 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Warin]

On 16/10/23 14:06, Graeme Fitzpatrick wrote:

Do we need the country, city & post code fields?



No.



Thanks

Graeme


On Mon, 16 Oct 2023 at 12:23, Yuchen Pei  wrote:

On Tue 2023-10-03 19:51:13 +1100, Warin wrote:

> On 3/10/23 14:27, Andrew Harvey wrote:

> [... 12 lines elided]

>         OK, what is needed to be done for "Stage 2 - Set unit from
>         housenumber"?

>     Further testing of the upload script. The changes themselves are
>     pretty safe. It's using a custom uploader and if something isn't
>     right it could make a mess. Sure the changeset could be reverted
>     in the worst case scenario but you end up with more history so
>     best to avoid this. I'll see if I can find some time to progress
>     this further.

> Umm 'custom uploader' .. a file compatible with JOSM should be easy
> enough to create. Then selecting a small area to upload and test
would
> be a simple manual operation, as would uploading the entire change
> set.

The osc file[1] generated in Stage 2 is compatible with JOSM.

[1] https://ypei.org/assets/tmp/unitFromNumber-1.osc

I spot checked a few nodes and they look correct too. See also the
attached screenshot.


My understanding of this Stage is to fix all the discrepancies between
streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap
dataset,
before Stage 3 - uploading new addresses from the latter.

I can do the test upload of a small area (say ~100 addresses) and
report
back.

I will check the scripts that generate this file, to find out whether
the logic indicates the file has full coverage corresponding to the
datasets.

> [... 5 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

That's okay. I created https://github.com/alantgeo/vicmap2osm as a mirror
of the GitLab code, happy to collaborate via Issues and Pull Requests there.

On Mon, 16 Oct 2023 at 15:49, Yuchen Pei  wrote:

> On Mon 2023-10-16 15:35:13 +1100, Andrew Harvey wrote:
>
> > [... 86 lines elided]
>
> > Are you on the OSM World Discord?
>
> I do not use Discord, because it is proprietary. I am on IRC, xmpp,
> mastodon and Signal - do you use any of these? If not can you suggest
> another free software option?
>
> Best,
> Yuchen
>
> --
> Timezone: UTC+11
> PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
>   
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 15:35:13 +1100, Andrew Harvey wrote:

> [... 86 lines elided]

> Are you on the OSM World Discord?

I do not use Discord, because it is proprietary. I am on IRC, xmpp,
mastodon and Signal - do you use any of these? If not can you suggest
another free software option?

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Mon, 16 Oct 2023 at 15:16, Yuchen Pei  wrote:

> On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:
>
> > On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:
>
> > [... 38 lines elided]
>
> > mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> > unit_number=x && street_number=y). For example, the Makefile rule
> > dist/unitFromNumber.osc is generated using this script. I have
> > generated
> > the osc file[1]. However, this file contains 38k nodes, whereas
> > the
> > input MR file[2] only has 12k features. So my question is - does
> > anyonw
> > know what is the easiest way to see all the changes in this file
> > staged
> > on a map, as a sanity check? OTOH I'd assume the file format is
> > some
> > standard osm change format.
>
> > Yes, this is mentioned in the README
>
> >> You can visualise the tag changes with bin/mrCoopDiff.js and
> > www/mrPreview.html at URL
>
> What is URL?
>

Ah yeah I put URL as a placeholder I was going to replace. At the moment
you can only view that locally within the repository as I haven't hosted
it. I was working on getting it out on GitLab pages or something like that
from the CI/CD pipeline.

>
> > I did this and validated that the changes look as intended.
>
> > [... 39 lines elided]
>
> > I can do the test upload of a small area (say ~100 addresses) and
> > report
> > back.
>
> > Please if you insist, can you just do < 10, no need to do 100.
>
> Why?
>

For testing, I feel it should be a number that we can manually work with.
After your testing either we need to then wait for the planet export to
catch up with your changes, or potentially have a conflict to deal with in
JOSM (or maybe JOSM would handle it, I'm not sure).


> > Once I've reviewed the above and am happy with it I'll do the upload
> > with the import account as planned.
>
> Even though you asked whether I would like to help and I said yes, the
> communications so far have give me the impression that you want to work
> on it solo


Not the case, I think it's great that you've taken interest in the import,
if we can work together to get it done that would be ideal.


> which is less of a problem if there's a clear timeframe to
> complete it. After all it was started in 2021 and stalled, and this is a
> community project to improve the VIC address coverage for everyone. To
> get it done I will continue working on it as I have been in the past few
> weeks. If Stage 2 is done I will push for progress in Stage 3, and so
> on.
>

It's a balance, I don't want to hold things up, at the same time we don't
want to mess up such a large import by rushing.

Are you on the OSM World Discord?
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:

> On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:

> [... 38 lines elided]

> mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> unit_number=x && street_number=y). For example, the Makefile rule
> dist/unitFromNumber.osc is generated using this script. I have
> generated
> the osc file[1]. However, this file contains 38k nodes, whereas
> the
> input MR file[2] only has 12k features. So my question is - does
> anyonw
> know what is the easiest way to see all the changes in this file
> staged
> on a map, as a sanity check? OTOH I'd assume the file format is
> some
> standard osm change format.

> Yes, this is mentioned in the README

>> You can visualise the tag changes with bin/mrCoopDiff.js and
> www/mrPreview.html at URL

What is URL?

> I did this and validated that the changes look as intended.

> [... 39 lines elided]

> I can do the test upload of a small area (say ~100 addresses) and
> report
> back.

> Please if you insist, can you just do < 10, no need to do 100.

Why?

> Once I've reviewed the above and am happy with it I'll do the upload
> with the import account as planned.

Even though you asked whether I would like to help and I said yes, the
communications so far have give me the impression that you want to work
on it solo, which is less of a problem if there's a clear timeframe to
complete it. After all it was started in 2021 and stalled, and this is a
community project to improve the VIC address coverage for everyone. To
get it done I will continue working on it as I have been in the past few
weeks. If Stage 2 is done I will push for progress in Stage 3, and so
on.

> [... 17 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:

> On Tue 2023-10-03 19:51:13 +1100, Warin wrote:
>
> > [... 14 lines elided]
>
> > OK, what is needed to be done for "Stage 2 - Set unit from
> > housenumber"?
>
> > Further testing of the upload script. The changes themselves are
> > pretty safe. It's using a custom uploader and if something isn't
> > right it could make a mess. Sure the changeset could be reverted
> > in the worst case scenario but you end up with more history so
> > best to avoid this. I'll see if I can find some time to progress
> > this further.
>
> > Umm 'custom uploader' .. a file compatible with JOSM should be easy
> > enough to create. Then selecting a small area to upload and test would
> > be a simple manual operation, as would uploading the entire change
> > set.
>
> I notice two scripts in the repo with the ability to upload:
>
> ./bin/mr2osc.mjs - converts a MapRoulette geojson to ocs files, and
> upload unless --dryrun is specified
>
> ./bin/upload.sh - upload osmChange files using a python script. I have
> not looked much into this one yet, as it showcases Stage 3.
>
> mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> unit_number=x && street_number=y). For example, the Makefile rule
> dist/unitFromNumber.osc is generated using this script. I have generated
> the osc file[1]. However, this file contains 38k nodes, whereas the
> input MR file[2] only has 12k features. So my question is - does anyonw
> know what is the easiest way to see all the changes in this file staged
> on a map, as a sanity check? OTOH I'd assume the file format is some
> standard osm change format.
>

Yes, this is mentioned in the README

> You can visualise the tag changes with bin/mrCoopDiff.js and
www/mrPreview.html at URL

I did this and validated that the changes look as intended.


>
> To Andrew: what specifically are you worried about with the upload
> script, and how can we help with the testing and uploading?
>

I was worried that if the object had changed since the OSC was generated
those changes might have been lost, as well as general error handling in my
custom uploader in case there were rejections. Though reviewing again,
letting JOSM upload the OSC would work.

On Sat, 7 Oct 2023 at 20:48, Yuchen Pei  wrote:

> On a second thought, why don't we just generate the osc file with
>
> make dist/unitFromnumber.osc
>
> and apply the osc file manually? Of course that's assuming the file is
> correct. For example, to understand the discrepancy in the number of
> nodes I mentioned above. I also noticed some minor issues with the
> script, like when the number of changes exceeds 10k, it attempts to
> split them multiple files, but they are identical rather than sequential
> parts.


I'll take a look at that.



On Mon, 16 Oct 2023 at 13:23, Yuchen Pei  wrote:

> My understanding of this Stage is to fix all the discrepancies between
> streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
> before Stage 3 - uploading new addresses from the latter.
>

Correct.


>
> I can do the test upload of a small area (say ~100 addresses) and report
> back.
>

Please if you insist, can you just do < 10, no need to do 100.

Once I've reviewed the above and am happy with it I'll do the upload with
the import account as planned.



On Mon, 16 Oct 2023 at 14:10, Graeme Fitzpatrick 
wrote:

> Do we need the country, city & post code fields?
>

This is mentioned at
https://gitlab.com/alantgeo/vicmap2osm#inclusion-of-addrsuburb-addrpostcode-and-addrstate
with the conclusion that "After lengthy engagement with the local
community, we opt to omit addr:suburb, addr:postcode, addr:state tags in
the current import." However existing tags won't and shouldn't be touched
as part of this import.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Graeme Fitzpatrick]

Do we need the country, city & post code fields?

Thanks

Graeme


On Mon, 16 Oct 2023 at 12:23, Yuchen Pei  wrote:

> On Tue 2023-10-03 19:51:13 +1100, Warin wrote:
>
> > On 3/10/23 14:27, Andrew Harvey wrote:
>
> > [... 12 lines elided]
>
> > OK, what is needed to be done for "Stage 2 - Set unit from
> > housenumber"?
>
> > Further testing of the upload script. The changes themselves are
> > pretty safe. It's using a custom uploader and if something isn't
> > right it could make a mess. Sure the changeset could be reverted
> > in the worst case scenario but you end up with more history so
> > best to avoid this. I'll see if I can find some time to progress
> > this further.
>
> > Umm 'custom uploader' .. a file compatible with JOSM should be easy
> > enough to create. Then selecting a small area to upload and test would
> > be a simple manual operation, as would uploading the entire change
> > set.
>
> The osc file[1] generated in Stage 2 is compatible with JOSM.
>
> [1] https://ypei.org/assets/tmp/unitFromNumber-1.osc
>
> I spot checked a few nodes and they look correct too. See also the
> attached screenshot.
>
>
> My understanding of this Stage is to fix all the discrepancies between
> streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
> before Stage 3 - uploading new addresses from the latter.
>
> I can do the test upload of a small area (say ~100 addresses) and report
> back.
>
> I will check the scripts that generate this file, to find out whether
> the logic indicates the file has full coverage corresponding to the
> datasets.
>
> > [... 5 lines elided]
>
>
> Best,
> Yuchen
>
> --
> Timezone: UTC+11
> PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
>   
> ___
> Talk-au mailing list
> Talk-au@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-au
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Tue 2023-10-03 19:51:13 +1100, Warin wrote:

> On 3/10/23 14:27, Andrew Harvey wrote:

> [... 12 lines elided]

> OK, what is needed to be done for "Stage 2 - Set unit from
> housenumber"?

> Further testing of the upload script. The changes themselves are
> pretty safe. It's using a custom uploader and if something isn't
> right it could make a mess. Sure the changeset could be reverted
> in the worst case scenario but you end up with more history so
> best to avoid this. I'll see if I can find some time to progress
> this further.

> Umm 'custom uploader' .. a file compatible with JOSM should be easy
> enough to create. Then selecting a small area to upload and test would
> be a simple manual operation, as would uploading the entire change
> set. 

The osc file[1] generated in Stage 2 is compatible with JOSM.

[1] https://ypei.org/assets/tmp/unitFromNumber-1.osc

I spot checked a few nodes and they look correct too. See also the
attached screenshot.


My understanding of this Stage is to fix all the discrepancies between
streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
before Stage 3 - uploading new addresses from the latter.

I can do the test upload of a small area (say ~100 addresses) and report
back.

I will check the scripts that generate this file, to find out whether
the logic indicates the file has full coverage corresponding to the
datasets.

> [... 5 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Sat 2023-10-07 20:40:47 +1100, Yuchen Pei wrote:


> [... 26 lines elided]

> mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> unit_number=x && street_number=y). For example, the Makefile rule
> dist/unitFromNumber.osc is generated using this script. I have generated
> the osc file[1]. However, this file contains 38k nodes, whereas the
> input MR file[2] only has 12k features. So my question is - does anyonw
> know what is the easiest way to see all the changes in this file staged
> on a map, as a sanity check? OTOH I'd assume the file format is some
> standard osm change format.

On a second thought, why don't we just generate the osc file with

make dist/unitFromnumber.osc

and apply the osc file manually? Of course that's assuming the file is
correct. For example, to understand the discrepancy in the number of
nodes I mentioned above. I also noticed some minor issues with the
script, like when the number of changes exceeds 10k, it attempts to
split them multiple files, but they are identical rather than sequential
parts.

> [... 17 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Tue 2023-10-03 19:51:13 +1100, Warin wrote:

> [... 14 lines elided]

> OK, what is needed to be done for "Stage 2 - Set unit from
> housenumber"?

> Further testing of the upload script. The changes themselves are
> pretty safe. It's using a custom uploader and if something isn't
> right it could make a mess. Sure the changeset could be reverted
> in the worst case scenario but you end up with more history so
> best to avoid this. I'll see if I can find some time to progress
> this further.

> Umm 'custom uploader' .. a file compatible with JOSM should be easy
> enough to create. Then selecting a small area to upload and test would
> be a simple manual operation, as would uploading the entire change
> set.

I notice two scripts in the repo with the ability to upload:

./bin/mr2osc.mjs - converts a MapRoulette geojson to ocs files, and
upload unless --dryrun is specified

./bin/upload.sh - upload osmChange files using a python script. I have
not looked much into this one yet, as it showcases Stage 3.

mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
unit_number=x && street_number=y). For example, the Makefile rule
dist/unitFromNumber.osc is generated using this script. I have generated
the osc file[1]. However, this file contains 38k nodes, whereas the
input MR file[2] only has 12k features. So my question is - does anyonw
know what is the easiest way to see all the changes in this file staged
on a map, as a sanity check? OTOH I'd assume the file format is some
standard osm change format.

To Andrew: what specifically are you worried about with the upload
script, and how can we help with the testing and uploading?

[1] https://ypei.org/assets/tmp/unitFromNumber-1.osc
[2] https://ypei.org/assets/tmp/mr_explodeUnitFromNumber.geojson

> [... 5 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Warin]

On 3/10/23 14:27, Andrew Harvey wrote:



On Mon, 2 Oct 2023 at 23:48, Yuchen Pei  wrote:

On Mon 2023-10-02 21:42:01 +1100, Andrew Harvey wrote:

> [... 15 lines elided]

> It's been a while since I worked on this, but I believe it was the
> matching of existing OSM addresses to Vicmap, and that matching
> affects most of the import stages.

OK, what is needed to be done for "Stage 2 - Set unit from
housenumber"?


Further testing of the upload script. The changes themselves are 
pretty safe. It's using a custom uploader and if something isn't right 
it could make a mess. Sure the changeset could be reverted in the 
worst case scenario but you end up with more history so best to avoid 
this. I'll see if I can find some time to progress this further.



Umm 'custom uploader' .. a file compatible with JOSM should be easy 
enough to create. Then selecting a small area to upload and test would 
be a simple manual operation, as would uploading the entire change set.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Mon, 2 Oct 2023 at 23:48, Yuchen Pei  wrote:

> On Mon 2023-10-02 21:42:01 +1100, Andrew Harvey wrote:
>
> > [... 15 lines elided]
>
> > It's been a while since I worked on this, but I believe it was the
> > matching of existing OSM addresses to Vicmap, and that matching
> > affects most of the import stages.
>
> OK, what is needed to be done for "Stage 2 - Set unit from housenumber"?
>

Further testing of the upload script. The changes themselves are pretty
safe. It's using a custom uploader and if something isn't right it could
make a mess. Sure the changeset could be reverted in the worst case
scenario but you end up with more history so best to avoid this. I'll see
if I can find some time to progress this further.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Tue, 3 Oct 2023 at 00:10, Yuchen Pei  wrote:

> On Mon 2023-10-02 21:35:10 +1030, Daniel O'Connor wrote:
>
> > While OSM doesn't have layers, https://openaddresses.io/ more or less
> > acts as the address layer. The datasets there aren't all ODBL, but
> > they are generally open. It includes GNAF.
>
> Thanks, I didn't know of openaddresses, but I'm a bit confused - what
> does open mean here? I can't find the license of the GNAF data, but
> looking at its webpage[1] it does not look free. Does "open" here simply
> mean "gratis access"?
>

I recently updated a few of the Australian sources in OpenAddresses which
had stalled.

See
https://data.gov.au/data/dataset/19432f89-dc3a-4ef3-b943-5326ef1dbecc/resource/09f74802-08b1-4214-a6ea-3591b2753d30/

"The EULA terms are based on the Creative Commons Attribution 4.0
International license (CC BY 4.0). However, an important restriction
relating to the use of the open G-NAF for the sending of mail has been
added. The open G-NAF data must not be used form the generation of an
address or a compilation or address for the sending of mail unless the user
has verified that each address to be used for the sending of mail is
capable of receiving mail by reference to a secondary source of
information. Further information on this use restriction is available here."

On Tue, 3 Oct 2023 at 06:30, Simon Poole  wrote:

> Except if something has massively changed, the GNAF data isn"t actually
> open.
>

Depends on your definition of open. Not open enough for OSM, but open
enough for most use cases.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Simon Poole]

Except if something has massively changed, the GNAF data isn"t actually open.

Am 2. Oktober 2023 13:05:10 MESZ schrieb Daniel O'Connor 
:
>While OSM doesn't have layers, https://openaddresses.io/ more or less acts
>as the address layer. The datasets there aren't all ODBL, but they are
>generally open. It includes GNAF. Given how frequently addresses update, to
>keep it in sync with OSM would be pretty significant and hard to reconcile
>with surveyed, on the ground data.
>
>Getting addresses into OSM is useful for routing, etc, but there might be
>more niche datasets that can be of value.
>
>Maproulette is potentially a good way to generate *probably correct but
>needs verification *data.
>Are there questions you can frame that people can answer with remote
>mapping?
>
>IE:
>Company X has said they will put solar on all of their stores by (date),
>but there are no solar panels within 50m of (store.location)
>Company Y shut down, is this store still open?
>School area Z doesn't have any sports fields/buildings/etc - are there any
>to be mapped?

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit Kaiten Mail gesendet.___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-02 21:35:10 +1030, Daniel O'Connor wrote:

> While OSM doesn't have layers, https://openaddresses.io/ more or less
> acts as the address layer. The datasets there aren't all ODBL, but
> they are generally open. It includes GNAF.

Thanks, I didn't know of openaddresses, but I'm a bit confused - what
does open mean here? I can't find the license of the GNAF data, but
looking at its webpage[1] it does not look free. Does "open" here simply
mean "gratis access"?

[1] https://geoscape.com.au/data/g-naf/

> [... 19 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-02 21:42:01 +1100, Andrew Harvey wrote:

> [... 15 lines elided]

> It's been a while since I worked on this, but I believe it was the
> matching of existing OSM addresses to Vicmap, and that matching
> affects most of the import stages.

OK, what is needed to be done for "Stage 2 - Set unit from housenumber"?

> > Are you interested in working on it?

> Absolutely! Thankfully git is decentralised so I don't really need
> a
> gitlab account.

> It would be much easier if you can create one though as otherwise
> collaboration will be difficult. From what I can see you just need
> either a phone number OR credit card, so you should be able to create
> an account without a credit card.

Well, I just tried again, and it asked again for me credit card info. It
appears that they want all three (credit card, phone and email), and I
can't simply skip the credit card and give them my phone number / email
address. TBH I find it pretty invasive - even github only asks for email
address.

> On Mon, 2 Oct 2023 at 11:45, Yuchen Pei  wrote:

> Maybe we can go stage by stage. Stage 1 - postal_code is the first
> stage
> that has not been completed.

> I built the dist/postalCodeURLs.txt file (see attached) yesterday
> and it
> contains 2425 JOSM RemoteControl urls. Shall we go ahead and
> import
> them (or a newly generated version)?

> I spent some time today reviewing the import again, and this one was
> ready, so I've done it at
> https://www.openstreetmap.org/changeset/142031616

Cool, thanks.

> [... 18 lines elided]

> It was built to run the whole process and generate all the outputs
> from all the import stages together, at this stage I wouldn't be
> refactoring it.

Fair enough!

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Daniel O'Connor]

While OSM doesn't have layers, https://openaddresses.io/ more or less acts
as the address layer. The datasets there aren't all ODBL, but they are
generally open. It includes GNAF. Given how frequently addresses update, to
keep it in sync with OSM would be pretty significant and hard to reconcile
with surveyed, on the ground data.

Getting addresses into OSM is useful for routing, etc, but there might be
more niche datasets that can be of value.

Maproulette is potentially a good way to generate *probably correct but
needs verification *data.
Are there questions you can frame that people can answer with remote
mapping?

IE:
Company X has said they will put solar on all of their stores by (date),
but there are no solar panels within 50m of (store.location)
Company Y shut down, is this store still open?
School area Z doesn't have any sports fields/buildings/etc - are there any
to be mapped?
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Sun, 1 Oct 2023 at 23:16, Yuchen Pei  wrote:

> > The preparation and planning is well progressed in my view. There is
> > always going to be a long tail of corner cases and I was attempting to
> > handle more of these in the code and that never got finished. We
> > probably would be better to make a call to accept those corner cases
> > and go ahead with the import.
>
> Thanks for the context. Which of the 7 Stages mentioned in the README
> correspond to the corner cases?
>

It's been a while since I worked on this, but I believe it was the matching
of existing OSM addresses to Vicmap, and that matching affects most of the
import stages.


> > Are you interested in working on it?
>
> Absolutely! Thankfully git is decentralised so I don't really need a
> gitlab account.


It would be much easier if you can create one though as otherwise
collaboration will be difficult. From what I can see you just need either a
phone number OR credit card, so you should be able to create an account
without a credit card.

On Mon, 2 Oct 2023 at 11:45, Yuchen Pei  wrote:

> Maybe we can go stage by stage. Stage 1 - postal_code is the first stage
> that has not been completed.
>
> I built the dist/postalCodeURLs.txt file (see attached) yesterday and it
> contains 2425 JOSM RemoteControl urls. Shall we go ahead and import
> them (or a newly generated version)?
>

I spent some time today reviewing the import again, and this one was ready,
so I've done it at https://www.openstreetmap.org/changeset/142031616

For context, we decided not to include addr:postcode on each address object
imported, therefore to ensure we can still capture address postcodes, they
are added to the admin_level=9 boundary relations. There were a bunch
already mapped but this changeset completed those missing the tag. My prior
analysis comparing these boundaries in OSM and the Vicmap address points
with postcode found this is a reliable way to ensure we have postcode
coverage, except for some edge cases mentioned in the import documentation.


> BTW, I'd assume the whole pipeline for each stage should be generated at
> approximately the same time, using the data downloaded at approximately
> the same time. If that is the case, it would make sense to have a
> Makefile rule for each stage, that remove all files in the pipeline for
> that stage and redownload and recompute them. What do you think?
>

It was built to run the whole process and generate all the outputs from all
the import stages together, at this stage I wouldn't be refactoring it.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

I am re-sending the following message without the attachment, as the
original message is held in the moderation queue due to its size. Sorry
for any duplication if that one is approved, but I might forget if I
don't resend, and the delay could also be confusing to people because I
already sent some follow-up messages.
On Sun 2023-10-01 21:20:38 +1100, Andrew Harvey wrote:

> [... 71 lines elided]

> The preparation and planning is well progressed in my view. There is
> always going to be a long tail of corner cases and I was attempting to
> handle more of these in the code and that never got finished. We
> probably would be better to make a call to accept those corner cases
> and go ahead with the import.

Maybe we can go stage by stage. Stage 1 - postal_code is the first stage
that has not been completed.

I built the dist/postalCodeURLs.txt file (see attached) yesterday and it
contains 2425 JOSM RemoteControl urls. Shall we go ahead and import them
(or a newly generated version)?

BTW, I'd assume the whole pipeline for each stage should be generated at
approximately the same time, using the data downloaded at approximately
the same time. If that is the case, it would make sense to have a
Makefile rule for each stage, that remove all files in the pipeline for
that stage and redownload and recompute them. What do you think?

> [... 2 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  


___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Graeme Fitzpatrick]

On Sun, 1 Oct 2023 at 21:34, Warin <61sundow...@gmail.com> wrote:

> Then simply the major cities of Brisbane


Brisbane itself is already done via an import of BCC data a few years ago,
but this is only "Brisbane" itself, not the surrounding "cities"

Thanks

Graeme
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Sun 2023-10-01 21:33:45 +1100, Warin wrote:

> [... 25 lines elided]

> ---

> I have been adding addr:street to those places with addr:housenumber
> where possible. If you are adding addr:housenumber then it is not much
> more work to add the addr:street data. There are a few that are
> uncertain so I lave them alone. Australia has 98% of addr:housnumber
> with addr:street where as on a world scale only 90% of addr:housnumber
> have addr:streettoo.

I have been adding house numbers using StreetComplete, and I always add
the street name afterwards. Normally on StreetComplete I see missing
street numbers but never missing street names where street numbers are
already there. So 98% sounds about right.

> -

> So Yuchen Pei what are your thoughts/skills for adding address data to OSM?

Apart from StreetComplete, I do have a computer programming background,
and would be interested in doing it in a more automated way, like what
vicmap2osm tries to do.

> [... 6 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Sun 2023-10-01 21:20:38 +1100, Andrew Harvey wrote:

> [... 23 lines elided]


> Is there a way to efficiently improve the address coverage in
> Australia?
> What are the main roadblocks?

> I don't have the numbers, but StreetComplete would already be helping
> already to improve the coverage,

I've been using StreetComplete. I feel not many people in AU are using
it - I could get into Australia weekly top-10 by spending ~15min per
day.

Also StreetComplete does not let me add numbers when the buildings are
not already recorded. When there are millions of addresses to be mapped
I fear it could take forever.

> In terms of imports the roadblocks are data availability (only ACT,
> NSW, TAS, VIC are currently available. QLD, NT, SA, WA are not) and
> then people contributing to an important effort. For the most part the
> community would like to see more address imports so that's unlikely a
> roadblock.

> [... 34 lines elided]

> The preparation and planning is well progressed in my view. There is
> always going to be a long tail of corner cases and I was attempting to
> handle more of these in the code and that never got finished. We
> probably would be better to make a call to accept those corner cases
> and go ahead with the import.

Thanks for the context. Which of the 7 Stages mentioned in the README
correspond to the corner cases?

> Are you interested in working on it?

Absolutely! Thankfully git is decentralised so I don't really need a
gitlab account. I could send patches in this mailing list (or a more
dev-focused mailing list like say dev-au if one could be created easily)
if that works for you?

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Warin]

On 1/10/23 18:08, Yuchen Pei wrote:


On 1 October 2023 12:56:20 GMT+11:00, Phil Wyatt  wrote:

but I also see that the VIC
import is well progressed

How is it well progressed, do you have a link showing the progress? From what I 
see it seems to be stalled.



I'd take it that someone is working on the software to do the job. Note 
that the person may have a real life to be getting on with too.


Someone else was doing NSW. Sorry I tend to forget names and just work 
with the data.


Tassie .. got to read the 
https://wiki.openstreetmap.org/wiki/Australian_Data_Sources#Tasmania


and then sort out "Only datasets where the metadata indicates Land 
Tasmania is the custodian"


Most people live in NSW + Vic so having those done would be good. Then 
simply the major cities of Brisbane, Adelaide, Perth , Hobart and Darwin 
would get most  of the population.



---

I have been adding addr:street to those places with addr:housenumber 
where possible. If you are adding addr:housenumber then it is not much 
more work to add the addr:street data. There are a few that are 
uncertain so I lave them alone. Australia has 98% of addr:housnumber 
with addr:street where as on a world scale only 90% of addr:housnumber 
have addr:streettoo.



-

So Yuchen Pei what are your thoughts/skills for adding address data to OSM?


___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Sun, 1 Oct 2023 at 12:34, Yuchen Pei  wrote:

> Hello,
>
> I read in a 6-year old post[1] that the Netherlands had an address
> coverage of over 99%. This made me curious what would be the Australian
> number. G-NAF boasts 15M addresses[2], whereas according to
> metrics.improveosm.org there are less than 1.2M address points mapped in
> AU[3], so that makes it less than 8%, way lower than the Netherlands.
> But please correct me if my stats are flawed.
>
> [1]
> https://community.openstreetmap.org/t/adding-housenumbers-with-streetcomplete/81323
> [2]
> https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf
> [3]
>
> https://metrics.improveosm.org/address-points/total-metrics-per-interval?duration=weekly=country=13=km=2022-01-01=2023-09-24
>
> Is there a way to efficiently improve the address coverage in Australia?
> What are the main roadblocks?
>

I don't have the numbers, but StreetComplete would already be helping
already to improve the coverage,

In terms of imports the roadblocks are data availability (only ACT, NSW,
TAS, VIC are currently available. QLD, NT, SA, WA are not) and then people
contributing to an important effort. For the most part the community would
like to see more address imports so that's unlikely a roadblock.


> For Victoria there was an initiative to import vicmap addresses[4][5],
> I wonder how many addresses it will add, and why it apparently got stuck
> (0 edits from the importer user[6])
>

That was my work. I ran out of time to finish the pre-work and actually do
the import, and without any other activity it's been on-hold.

It also mentions

> Mon 30th May 2022 - Import executed per plan (date may be moved back

> if there is unresolved feedback or discussion from the proposal sent

> to imports)

Did the execution really happen?


No it didn't. I think that was just the planned date.



On Sun, 1 Oct 2023 at 18:15, Yuchen Pei  wrote:

> On 1 October 2023 12:56:20 GMT+11:00, Phil Wyatt 
> wrote:
> > but I also see that the VIC
> > import is well progressed
>
> How is it well progressed, do you have a link showing the progress? From
> what I see it seems to be stalled.
>

The preparation and planning is well progressed in my view. There is always
going to be a long tail of corner cases and I was attempting to handle more
of these in the code and that never got finished. We probably would be
better to make a call to accept those corner cases and go ahead with the
import.

Are you interested in working on it?
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On 1 October 2023 12:56:20 GMT+11:00, Phil Wyatt  wrote:
> but I also see that the VIC
> import is well progressed

How is it well progressed, do you have a link showing the progress? From what I 
see it seems to be stalled.

-- 
Best,
Yuchen

Sent from a mobile device. Please excuse my brevity, typos and lack of pgp.

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Phil Wyatt]

Hi Yuchen,

For me it's a lack of programming skills. In some other cases its lack of
open data for the addresses. There is also a requirement for good
documentation and community support for any imports.

https://wiki.openstreetmap.org/wiki/Import

We have open data for some states

https://wiki.openstreetmap.org/wiki/Australian_Data_Sources

If you have the skills to assist with imports then I am sure the community
will assist where possible in identifying issues that will need to be
resolved (ie like rural addresses that are often at the centroid of large
holdings).

In my home state of Tasmania we have access to the full address data as
OpenData if you are looking to start somewhere, but I also see that the VIC
import is well progressed and the code for that project may well be useful
for other state imports.

Cheers - Phil (aka tastrax)




-Original Message-
From: Yuchen Pei  
Sent: Sunday, October 1, 2023 12:31 PM
To: talk-au@openstreetmap.org
Subject: [talk-au] How to efficiently improve AU address coverage?

Hello,

I read in a 6-year old post[1] that the Netherlands had an address coverage
of over 99%. This made me curious what would be the Australian number. G-NAF
boasts 15M addresses[2], whereas according to metrics.improveosm.org there
are less than 1.2M address points mapped in AU[3], so that makes it less
than 8%, way lower than the Netherlands.
But please correct me if my stats are flawed.

[1]
https://community.openstreetmap.org/t/adding-housenumbers-with-streetcomplet
e/81323
[2]
https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf
[3]
https://metrics.improveosm.org/address-points/total-metrics-per-interval?dur
ation=weekly=country=13=km=2022-01-01=2
023-09-24

Is there a way to efficiently improve the address coverage in Australia?
What are the main roadblocks?

For Victoria there was an initiative to import vicmap addresses[4][5], I
wonder how many addresses it will add, and why it apparently got stuck
(0 edits from the importer user[6])

[4]
https://lists.openstreetmap.org/pipermail/talk-au/2021-May/014622.html
[5] https://gitlab.com/alantgeo/vicmap2osm/
[6] https://www.openstreetmap.org/user/vicmap_import

Best,
Yuchen

--
Timezone: UTC+10
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  <https://ypei.org/assets/ypei-pubkey.txt>

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au


___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Sun 2023-10-01 12:30:50 +1100, Yuchen Pei wrote:

> [... 19 lines elided]

> For Victoria there was an initiative to import vicmap addresses[4][5],
> I wonder how many addresses it will add,

Just noticed the wiki page, according to which it would increase the VIC
address coverage to >95%:
https://wiki.openstreetmap.org/wiki/Vicmap_Address

It also mentions

> Mon 30th May 2022 - Import executed per plan (date may be moved back
> if there is unresolved feedback or discussion from the proposal sent
> to imports)

Did the execution really happen?

> [... 14 lines elided]


Best,
Yuchen

--
Timezone: UTC+10
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

[talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

Hello,

I read in a 6-year old post[1] that the Netherlands had an address
coverage of over 99%. This made me curious what would be the Australian
number. G-NAF boasts 15M addresses[2], whereas according to
metrics.improveosm.org there are less than 1.2M address points mapped in
AU[3], so that makes it less than 8%, way lower than the Netherlands.
But please correct me if my stats are flawed.

[1] 
https://community.openstreetmap.org/t/adding-housenumbers-with-streetcomplete/81323
[2]
https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf
[3]
https://metrics.improveosm.org/address-points/total-metrics-per-interval?duration=weekly=country=13=km=2022-01-01=2023-09-24

Is there a way to efficiently improve the address coverage in Australia?
What are the main roadblocks?

For Victoria there was an initiative to import vicmap addresses[4][5],
I wonder how many addresses it will add, and why it apparently got stuck
(0 edits from the importer user[6])

[4]
https://lists.openstreetmap.org/pipermail/talk-au/2021-May/014622.html
[5] https://gitlab.com/alantgeo/vicmap2osm/
[6] https://www.openstreetmap.org/user/vicmap_import

Best,
Yuchen

--
Timezone: UTC+10
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au