Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 15:35:13 +1100, Andrew Harvey wrote:

> [... 86 lines elided]

> Are you on the OSM World Discord?

I do not use Discord, because it is proprietary. I am on IRC, xmpp,
mastodon and Signal - do you use any of these? If not can you suggest
another free software option?

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Mon, 16 Oct 2023 at 15:16, Yuchen Pei  wrote:

> On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:
>
> > On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:
>
> > [... 38 lines elided]
>
> > mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> > unit_number=x && street_number=y). For example, the Makefile rule
> > dist/unitFromNumber.osc is generated using this script. I have
> > generated
> > the osc file[1]. However, this file contains 38k nodes, whereas
> > the
> > input MR file[2] only has 12k features. So my question is - does
> > anyonw
> > know what is the easiest way to see all the changes in this file
> > staged
> > on a map, as a sanity check? OTOH I'd assume the file format is
> > some
> > standard osm change format.
>
> > Yes, this is mentioned in the README
>
> >> You can visualise the tag changes with bin/mrCoopDiff.js and
> > www/mrPreview.html at URL
>
> What is URL?
>

Ah yeah I put URL as a placeholder I was going to replace. At the moment
you can only view that locally within the repository as I haven't hosted
it. I was working on getting it out on GitLab pages or something like that
from the CI/CD pipeline.

>
> > I did this and validated that the changes look as intended.
>
> > [... 39 lines elided]
>
> > I can do the test upload of a small area (say ~100 addresses) and
> > report
> > back.
>
> > Please if you insist, can you just do < 10, no need to do 100.
>
> Why?
>

For testing, I feel it should be a number that we can manually work with.
After your testing either we need to then wait for the planet export to
catch up with your changes, or potentially have a conflict to deal with in
JOSM (or maybe JOSM would handle it, I'm not sure).


> > Once I've reviewed the above and am happy with it I'll do the upload
> > with the import account as planned.
>
> Even though you asked whether I would like to help and I said yes, the
> communications so far have give me the impression that you want to work
> on it solo


Not the case, I think it's great that you've taken interest in the import,
if we can work together to get it done that would be ideal.


> which is less of a problem if there's a clear timeframe to
> complete it. After all it was started in 2021 and stalled, and this is a
> community project to improve the VIC address coverage for everyone. To
> get it done I will continue working on it as I have been in the past few
> weeks. If Stage 2 is done I will push for progress in Stage 3, and so
> on.
>

It's a balance, I don't want to hold things up, at the same time we don't
want to mess up such a large import by rushing.

Are you on the OSM World Discord?
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Mon 2023-10-16 14:51:00 +1100, Andrew Harvey wrote:

> On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:

> [... 38 lines elided]

> mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> unit_number=x && street_number=y). For example, the Makefile rule
> dist/unitFromNumber.osc is generated using this script. I have
> generated
> the osc file[1]. However, this file contains 38k nodes, whereas
> the
> input MR file[2] only has 12k features. So my question is - does
> anyonw
> know what is the easiest way to see all the changes in this file
> staged
> on a map, as a sanity check? OTOH I'd assume the file format is
> some
> standard osm change format.

> Yes, this is mentioned in the README

>> You can visualise the tag changes with bin/mrCoopDiff.js and
> www/mrPreview.html at URL

What is URL?

> I did this and validated that the changes look as intended.

> [... 39 lines elided]

> I can do the test upload of a small area (say ~100 addresses) and
> report
> back.

> Please if you insist, can you just do < 10, no need to do 100.

Why?

> Once I've reviewed the above and am happy with it I'll do the upload
> with the import account as planned.

Even though you asked whether I would like to help and I said yes, the
communications so far have give me the impression that you want to work
on it solo, which is less of a problem if there's a clear timeframe to
complete it. After all it was started in 2021 and stalled, and this is a
community project to improve the VIC address coverage for everyone. To
get it done I will continue working on it as I have been in the past few
weeks. If Stage 2 is done I will push for progress in Stage 3, and so
on.

> [... 17 lines elided]

Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Andrew Harvey]

On Sat, 7 Oct 2023 at 20:40, Yuchen Pei  wrote:

> On Tue 2023-10-03 19:51:13 +1100, Warin wrote:
>
> > [... 14 lines elided]
>
> > OK, what is needed to be done for "Stage 2 - Set unit from
> > housenumber"?
>
> > Further testing of the upload script. The changes themselves are
> > pretty safe. It's using a custom uploader and if something isn't
> > right it could make a mess. Sure the changeset could be reverted
> > in the worst case scenario but you end up with more history so
> > best to avoid this. I'll see if I can find some time to progress
> > this further.
>
> > Umm 'custom uploader' .. a file compatible with JOSM should be easy
> > enough to create. Then selecting a small area to upload and test would
> > be a simple manual operation, as would uploading the entire change
> > set.
>
> I notice two scripts in the repo with the ability to upload:
>
> ./bin/mr2osc.mjs - converts a MapRoulette geojson to ocs files, and
> upload unless --dryrun is specified
>
> ./bin/upload.sh - upload osmChange files using a python script. I have
> not looked much into this one yet, as it showcases Stage 3.
>
> mr2osc.mjs is used in Stage 2 (replacing street_number=x/y with
> unit_number=x && street_number=y). For example, the Makefile rule
> dist/unitFromNumber.osc is generated using this script. I have generated
> the osc file[1]. However, this file contains 38k nodes, whereas the
> input MR file[2] only has 12k features. So my question is - does anyonw
> know what is the easiest way to see all the changes in this file staged
> on a map, as a sanity check? OTOH I'd assume the file format is some
> standard osm change format.
>

Yes, this is mentioned in the README

> You can visualise the tag changes with bin/mrCoopDiff.js and
www/mrPreview.html at URL

I did this and validated that the changes look as intended.


>
> To Andrew: what specifically are you worried about with the upload
> script, and how can we help with the testing and uploading?
>

I was worried that if the object had changed since the OSC was generated
those changes might have been lost, as well as general error handling in my
custom uploader in case there were rejections. Though reviewing again,
letting JOSM upload the OSC would work.

On Sat, 7 Oct 2023 at 20:48, Yuchen Pei  wrote:

> On a second thought, why don't we just generate the osc file with
>
> make dist/unitFromnumber.osc
>
> and apply the osc file manually? Of course that's assuming the file is
> correct. For example, to understand the discrepancy in the number of
> nodes I mentioned above. I also noticed some minor issues with the
> script, like when the number of changes exceeds 10k, it attempts to
> split them multiple files, but they are identical rather than sequential
> parts.


I'll take a look at that.



On Mon, 16 Oct 2023 at 13:23, Yuchen Pei  wrote:

> My understanding of this Stage is to fix all the discrepancies between
> streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
> before Stage 3 - uploading new addresses from the latter.
>

Correct.


>
> I can do the test upload of a small area (say ~100 addresses) and report
> back.
>

Please if you insist, can you just do < 10, no need to do 100.

Once I've reviewed the above and am happy with it I'll do the upload with
the import account as planned.



On Mon, 16 Oct 2023 at 14:10, Graeme Fitzpatrick 
wrote:

> Do we need the country, city & post code fields?
>

This is mentioned at
https://gitlab.com/alantgeo/vicmap2osm#inclusion-of-addrsuburb-addrpostcode-and-addrstate
with the conclusion that "After lengthy engagement with the local
community, we opt to omit addr:suburb, addr:postcode, addr:state tags in
the current import." However existing tags won't and shouldn't be touched
as part of this import.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Graeme Fitzpatrick]

Do we need the country, city & post code fields?

Thanks

Graeme


On Mon, 16 Oct 2023 at 12:23, Yuchen Pei  wrote:

> On Tue 2023-10-03 19:51:13 +1100, Warin wrote:
>
> > On 3/10/23 14:27, Andrew Harvey wrote:
>
> > [... 12 lines elided]
>
> > OK, what is needed to be done for "Stage 2 - Set unit from
> > housenumber"?
>
> > Further testing of the upload script. The changes themselves are
> > pretty safe. It's using a custom uploader and if something isn't
> > right it could make a mess. Sure the changeset could be reverted
> > in the worst case scenario but you end up with more history so
> > best to avoid this. I'll see if I can find some time to progress
> > this further.
>
> > Umm 'custom uploader' .. a file compatible with JOSM should be easy
> > enough to create. Then selecting a small area to upload and test would
> > be a simple manual operation, as would uploading the entire change
> > set.
>
> The osc file[1] generated in Stage 2 is compatible with JOSM.
>
> [1] https://ypei.org/assets/tmp/unitFromNumber-1.osc
>
> I spot checked a few nodes and they look correct too. See also the
> attached screenshot.
>
>
> My understanding of this Stage is to fix all the discrepancies between
> streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
> before Stage 3 - uploading new addresses from the latter.
>
> I can do the test upload of a small area (say ~100 addresses) and report
> back.
>
> I will check the scripts that generate this file, to find out whether
> the logic indicates the file has full coverage corresponding to the
> datasets.
>
> > [... 5 lines elided]
>
>
> Best,
> Yuchen
>
> --
> Timezone: UTC+11
> PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
>   
> ___
> Talk-au mailing list
> Talk-au@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-au
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] How to efficiently improve AU address coverage?

[Yuchen Pei]

On Tue 2023-10-03 19:51:13 +1100, Warin wrote:

> On 3/10/23 14:27, Andrew Harvey wrote:

> [... 12 lines elided]

> OK, what is needed to be done for "Stage 2 - Set unit from
> housenumber"?

> Further testing of the upload script. The changes themselves are
> pretty safe. It's using a custom uploader and if something isn't
> right it could make a mess. Sure the changeset could be reverted
> in the worst case scenario but you end up with more history so
> best to avoid this. I'll see if I can find some time to progress
> this further.

> Umm 'custom uploader' .. a file compatible with JOSM should be easy
> enough to create. Then selecting a small area to upload and test would
> be a simple manual operation, as would uploading the entire change
> set. 

The osc file[1] generated in Stage 2 is compatible with JOSM.

[1] https://ypei.org/assets/tmp/unitFromNumber-1.osc

I spot checked a few nodes and they look correct too. See also the
attached screenshot.


My understanding of this Stage is to fix all the discrepancies between
streetnumber=X/Y in osm and streetnumber=Y;unit=X in the vicmap dataset,
before Stage 3 - uploading new addresses from the latter.

I can do the test upload of a small area (say ~100 addresses) and report
back.

I will check the scripts that generate this file, to find out whether
the logic indicates the file has full coverage corresponding to the
datasets.

> [... 5 lines elided]


Best,
Yuchen

--
Timezone: UTC+11
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
  
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au