[talk-au] Using roads dataset from data.sa.gov.au

[Henry Haselgrove]

Hi everyone,

 

I have recently been looking at the "roads" dataset on data.sa.gov.au, and
trying to figure out if there is a good way to use it in OSM.

 

One thing I have tried so far is to process the data in a manner that
identifies roads that are missing or incorrectly tagged in OSM.  The result
is a .osm file that contains approximately 20,000 ways from data.sa.gov.au
that are either missing or named differently to what is in OSM. The data
covers not only SA but also the western 100km of VIC. I don't think that
automatically merging this data into OSM would be a good idea, unless anyone
knows of a clever way to do that. However I reckon the file could be a
valuable source for manual edits. 

 

I put a brief description of my work on the wiki:

 

https://wiki.openstreetmap.org/wiki/Import/South_Australian_Roads

 

I would be interested in other ideas you might have for how to use the
source dataset. I should mention that, in the .osm file I have distilled so
far, I tried to include only public roads, by excluding any way that does
not have a name. About half of the source dataset consists of unnamed ways,
which seem to be a combination of private access roads and driveways, and
rural tracks. Perhaps some of this data could be useful in OSM too.

 

Cheers,

 

Henry

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

Softgrow has done some work in metro Adelaide around checking roads:

http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia

I would be keen for:
- New roads (housing developments etc) in metro, outer metro areas first
- Identifying naming conflicts in metro areas next

The later I would recommend manual checking of (happy to help) to avoid
clobbering roads that simply need splitting using a process like the one on
the sa project page.
On 21/02/2015 12:20 PM, "Henry Haselgrove"  wrote:

> Hi everyone,
>
>
>
> I have recently been looking at the “roads” dataset on data.sa.gov.au,
> and trying to figure out if there is a good way to use it in OSM.
>
>
>
> One thing I have tried so far is to process the data in a manner that
> identifies roads that are missing or incorrectly tagged in OSM.  The result
> is a .osm file that contains approximately 20,000 ways from data.sa.gov.au
> that are either missing or named differently to what is in OSM. The data
> covers not only SA but also the western 100km of VIC. I don’t think that
> automatically merging this data into OSM would be a good idea, unless
> anyone knows of a clever way to do that. However I reckon the file could be
> a valuable source for manual edits.
>
>
>
> I put a brief description of my work on the wiki:
>
>
>
> https://wiki.openstreetmap.org/wiki/Import/South_Australian_Roads
>
>
>
> I would be interested in other ideas you might have for how to use the
> source dataset. I should mention that, in the .osm file I have distilled so
> far, I tried to include only public roads, by excluding any way that does
> not have a name. About half of the source dataset consists of unnamed ways,
> which seem to be a combination of private access roads and driveways, and
> rural tracks. Perhaps some of this data could be useful in OSM too.
>
>
>
> Cheers,
>
>
>
> Henry
>
>
>
> ___
> Talk-au mailing list
> Talk-au@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-au
>
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Henry Haselgrove]

 

>  From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> Softgrow has done some work in metro Adelaide around checking roads:
> http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia

Thanks. I had seen the edits Softgrow has done, but I wasn’t aware it was 
documented anywhere

> From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com] 
> Sent: Saturday, 21 February 2015 3:39 AM



> Couple of false positives:

> Knox Terrace (Metro area, near waterfall gully) - it's a bit windy and for 
> some reason not matched

 

Actually this is not a false positive. The OSM version of the name has too many 
R’s !!

 

> Some of the walking trails near mt lofty

> OBahn busway probably should be excluded.

> There's a lot of little laneways, etc that are worthwhile adding in on their 
> own.

 

> Given that importing this is probably not a great idea, what other ways could 
> you present the data - as a list with clickable links to the coordatines/OSM 
> way?

> Or even better, http://wiki.openstreetmap.org/wiki/MapRoulette/Challenges ?

 

I will try to give this some thought. I just had a brief look at MapRoulette, 
but I don’t really understand yet what the technical requirements are.

 

Just loading the .osm file as a separate layer in JOSM seems to be a reasonable 
way to use the data. Potentially, I could re-generate the file often, to 
provide an up-to-date indication of what is left to do. However there would be 
no easy way for someone to mark for others the fact that a way doesn’t really 
need changing; and that could lead to some duplicated effort.

 

Incidentally, I noticed that you used some of the data in the last 24hrs to 
update road names on KI. I decided to have a go too, adding some missing roads, 
also to KI. The procedure of copying a road from one layer to another, joining 
it to existing ways, and uploading, seems to be fairly easy once you get into 
the rhythm. I was able to add around one missing road every 30 seconds or so.

 

By the way, I have just put an updated version of “missing.osm” in my Google 
Drive folder, based on the state of OSM today (rather than 9 Feb, like the 
previous one). 

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

Neat. So map roulette would solve the 'no need to fix
On 22/02/2015 11:20 AM, "Henry Haselgrove"  wrote:

>
>
> *>  From:* Daniel O'Connor [mailto:daniel.ocon...@gmail.com
> ]
> > Softgrow has done some work in metro Adelaide around checking roads:
> > http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia
>
> Thanks. I had seen the edits Softgrow has done, but I wasn’t aware it was
> documented anywhere
>
> *> From:* Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> *> Sent:* Saturday, 21 February 2015 3:39 AM
>
> > Couple of false positives:
>
> > Knox Terrace (Metro area, near waterfall gully) - it's a bit windy and
> for some reason not matched
>
>
>
> Actually this is not a false positive. The OSM version of the name has too
> many R’s !!
>
>
>
> > Some of the walking trails near mt lofty
>
> > OBahn busway probably should be excluded.
>
> > There's a lot of little laneways, etc that are worthwhile adding in on
> their own.
>
>
>
> > Given that importing this is probably not a great idea, what other ways
> could you present the data - as a list with clickable links to the
> coordatines/OSM way?
>
> > Or even better,
> http://wiki.openstreetmap.org/wiki/MapRoulette/Challenges ?
>
>
>
> I will try to give this some thought. I just had a brief look at
> MapRoulette, but I don’t really understand yet what the technical
> requirements are.
>
>
>
> Just loading the .osm file as a separate layer in JOSM seems to be a
> reasonable way to use the data. Potentially, I could re-generate the file
> often, to provide an up-to-date indication of what is left to do. However
> there would be no easy way for someone to mark for others the fact that a
> way doesn’t really need changing; and that could lead to some duplicated
> effort.
>
>
>
> Incidentally, I noticed that you used some of the data in the last 24hrs
> to update road names on KI. I decided to have a go too, adding some missing
> roads, also to KI. The procedure of copying a road from one layer to
> another, joining it to existing ways, and uploading, seems to be fairly
> easy once you get into the rhythm. I was able to add around one missing
> road every 30 seconds or so.
>
>
>
> By the way, I have just put an updated version of “missing.osm” in my
> Google Drive folder, based on the state of OSM today (rather than 9 Feb,
> like the previous one).
>
>
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

Neat. So map roulette would solve the 'no need to fix' signal to others.

To make a project is just using curl plus json to describe it and add
tasks. Uh can't find github link right now but the api is pretty clear.

Happy to help set that up later today/this week, as well as get the process
automated (git, 1x cron, etc)

> On 22/02/2015 11:20 AM, "Henry Haselgrove"  wrote:
>>
>>
>>
>> >  From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
>> > Softgrow has done some work in metro Adelaide around checking roads:
>> > http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia
>>
>> Thanks. I had seen the edits Softgrow has done, but I wasn’t aware it
was documented anywhere
>>
>> > From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
>> > Sent: Saturday, 21 February 2015 3:39 AM
>>
>> > Couple of false positives:
>>
>> > Knox Terrace (Metro area, near waterfall gully) - it's a bit windy and
for some reason not matched
>>
>>
>>
>> Actually this is not a false positive. The OSM version of the name has
too many R’s !!
>>
>>
>>
>> > Some of the walking trails near mt lofty
>>
>> > OBahn busway probably should be excluded.
>>
>> > There's a lot of little laneways, etc that are worthwhile adding in on
their own.
>>
>>
>>
>> > Given that importing this is probably not a great idea, what other
ways could you present the data - as a list with clickable links to the
coordatines/OSM way?
>>
>> > Or even better,
http://wiki.openstreetmap.org/wiki/MapRoulette/Challenges ?
>>
>>
>>
>> I will try to give this some thought. I just had a brief look at
MapRoulette, but I don’t really understand yet what the technical
requirements are.
>>
>>
>>
>> Just loading the .osm file as a separate layer in JOSM seems to be a
reasonable way to use the data. Potentially, I could re-generate the file
often, to provide an up-to-date indication of what is left to do. However
there would be no easy way for someone to mark for others the fact that a
way doesn’t really need changing; and that could lead to some duplicated
effort.
>>
>>
>>
>> Incidentally, I noticed that you used some of the data in the last 24hrs
to update road names on KI. I decided to have a go too, adding some missing
roads, also to KI. The procedure of copying a road from one layer to
another, joining it to existing ways, and uploading, seems to be fairly
easy once you get into the rhythm. I was able to add around one missing
road every 30 seconds or so.
>>
>>
>>
>> By the way, I have just put an updated version of “missing.osm” in my
Google Drive folder, based on the state of OSM today (rather than 9 Feb,
like the previous one).
>>
>>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Alex Sims]

Hi,

Can you please make the matlab scripts available on Github so they might be 
used/improved?

I’m particularly wanting to use the output, then check the revised output, 
rinse, lather, repeat

Alex

> To make a project is just using curl plus json to describe it and add tasks. 
> Uh can't find github link right now but the api is pretty clear.
> Happy to help set that up later today/this week, as well as get the process 
> automated (git, 1x cron, etc)
> 
> > On 22/02/2015 11:20 AM, "Henry Haselgrove"  > > wrote:
> >>
> >>  
> >>
> >> >  From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com 
> >> > ]
> >> > Softgrow has done some work in metro Adelaide around checking roads:
> >> > http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia 
> >> > 
> >>
> >> Thanks. I had seen the edits Softgrow has done, but I wasn’t aware it was 
> >> documented anywhere
> 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Henry Haselgrove]

 

I have put the Matlab scripts here:

https://github.com/q-bits/osm-scripts

 

 

From: Alex Sims [mailto:a...@softgrow.com] 
Sent: Sunday, 22 February 2015 8:16 PM
To: OSM Australian Talk List
Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au

 

Hi,

 

Can you please make the matlab scripts available on Github so they might be 
used/improved?

 

I’m particularly wanting to use the output, then check the revised output, 
rinse, lather, repeat

 

Alex

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Alex Sims]

I’ve started finding errors not on OpenStreetMap but in the DPTI data. 

I’ve documented a fix process at 
https://wiki.openstreetmap.org/wiki/Import/South_Australian_Roads#Errors_in_the_source_data
 


It’s not as easy as an edit on the Map but it should work

Alex

> 
>> To make a project is just using curl plus json to describe it and add tasks. 
>> Uh can't find github link right now but the api is pretty clear.
>> Happy to help set that up later today/this week, as well as get the process 
>> automated (git, 1x cron, etc)
>> 
>> > On 22/02/2015 11:20 AM, "Henry Haselgrove" > > > wrote:
>> >>
>> >>  
>> >>
>> >> >  From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com 
>> >> > ]
>> >> > Softgrow has done some work in metro Adelaide around checking roads:
>> >> > http://wiki.openstreetmap.org/wiki/WikiProject_South_Australia 
>> >> > 
>> >>
>> >> Thanks. I had seen the edits Softgrow has done, but I wasn’t aware it was 
>> >> documented anywhere
>> 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Henry Haselgrove]

Well spotted, Alex. It is interesting that both of the errors you have found 
have been in the “type” part of the name (e.g., street/lane/terrace etc). There 
seems to be approximately 1000 roads where the difference between the OSM and 
datasa versions is the type (excluding cases that could easily be determined to 
be typos in OSM, like terace, cresent,  etc). I wonder what proportion of those 
1000 are errors in datasa? 

 

I have decided to modify my scripts so that they omit from “missing.osm” cases 
where the difference is only in the road-type portion of the name. The script 
will now put these cases into a different file, “different_road_type.osm”, 
instead. Except, the cases of obviously misspelled OSM road types will stay in 
missing.osm. 

 

I have placed my latest versions of missing.osm and different_road_type.osm in 
the Google Drive folder. I have also pushed the new versions of the scripts to 
Github.

 

Incidentally, the scripts now put the tag “upload=false” into the .osm files, 
so that if you accidentally hit “Upload” in JOSM when you are on that layer bad 
things won’t happen!

 

Cheers,

 

Henry

 

From: Alex Sims [mailto:a...@softgrow.com] 
Sent: Thursday, 26 February 2015 1:35 AM
To: OSM Australian Talk List
Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au

 

I’ve started finding errors not on OpenStreetMap but in the DPTI data. 

 

I’ve documented a fix process at 
https://wiki.openstreetmap.org/wiki/Import/South_Australian_Roads#Errors_in_the_source_data

 

It’s not as easy as an edit on the Map but it should work

 

Alex

 

 

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

So, after doing this manually for a bit; it's generally working well.

There are some where spot checking against other sources suggests the
dataset is wrong, how do you suggest we indicate these?

I've put in NOTE or FIXME on the relevant way.


The thing that is troubling me is the size of the dataset - a few hours
work barely makes a dent.

I've taken to deleting all Primary/Trunk, Secondary and Track ways from the
data set; and then cropping stuff down to the metro adelaide area; and it's
still very sizable.


I'd be *really* keen on maproulette at this point - you seem to be able to
produce updated files fairly regularly, adding a few bash scripts to turn
that into curl friendly statements seems achievable.

Going to start sending a few pull requests your way to get us started on
this.


On Sun, Mar 1, 2015 at 11:23 AM, Henry Haselgrove 
wrote:

> Well spotted, Alex. It is interesting that both of the errors you have
> found have been in the “type” part of the name (e.g., street/lane/terrace
> etc). There seems to be approximately 1000 roads where the difference
> between the OSM and datasa versions is the type (excluding cases that could
> easily be determined to be typos in OSM, like terace, cresent,  etc). I
> wonder what proportion of those 1000 are errors in datasa?
>
>
>
> I have decided to modify my scripts so that they omit from “missing.osm”
> cases where the difference is only in the road-type portion of the name.
> The script will now put these cases into a different file,
> “different_road_type.osm”, instead. Except, the cases of obviously
> misspelled OSM road types will stay in missing.osm.
>
>
>
> I have placed my latest versions of missing.osm and
> different_road_type.osm in the Google Drive folder. I have also pushed the
> new versions of the scripts to Github.
>
>
>
> Incidentally, the scripts now put the tag “upload=false” into the .osm
> files, so that if you accidentally hit “Upload” in JOSM when you are on
> that layer bad things won’t happen!
>
>
>
> Cheers,
>
>
>
> Henry
>
>
>
> *From:* Alex Sims [mailto:a...@softgrow.com]
> *Sent:* Thursday, 26 February 2015 1:35 AM
> *To:* OSM Australian Talk List
> *Subject:* Re: [talk-au] Using roads dataset from data.sa.gov.au
>
>
>
> I’ve started finding errors not on OpenStreetMap but in the DPTI data.
>
>
>
> I’ve documented a fix process at
> https://wiki.openstreetmap.org/wiki/Import/South_Australian_Roads#Errors_in_the_source_data
>
>
>
> It’s not as easy as an edit on the Map but it should work
>
>
>
> Alex
>
>
>
>
>
>
>
> ___
> Talk-au mailing list
> Talk-au@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-au
>
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Henry Haselgrove]

I agree… with a dataset this large prioritisation is important. All your 
specific suggestions for culling parts from missing.osm sound good. It would be 
easy to add an option to the scripts to exclude highway={primary, secondary, 
track}. And the suggestions you made via github look good too.

 

However, I think that some more significant changes to the script should be 
done before this data is unleased onto maproulette. The roads that are 
currently in missing.osm fall (more or less) under three categories:

-- roads that are completely absent in OSM

-- roads that appear in OSM, but have an empty name

-- roads that appear in OSM and have a non-empty name which is 
different to the datasa name, either because OSM is wrong or datasa is wrong 
(or because both are right, such as “Mount Magnificent Road” versus “Mt. 
Magnificent Road”)

 

I propose to modify the script to automatically exclude as much as possible 
from the third category. Because, it will be hard for an armchair mapper to 
decide whether OSM or datasa is wrong in those cases. I could try to do this 
over the coming week. 

 

I’m not sure I agree that the Adelaide metro area should be given particular 
priority over other areas. But I’m probably biased, since I grew up in rural SA!

 

Probably we should make a posting to the osm “imports” list before too much 
longer, to let them know what we’re thinking. 

 

From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com] 
Sent: Monday, 9 March 2015 5:32 AM
To: Henry Haselgrove
Cc: Alex Sims; OSM Australian Talk List
Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au

 

So, after doing this manually for a bit; it's generally working well.

 

There are some where spot checking against other sources suggests the dataset 
is wrong, how do you suggest we indicate these?

 

I've put in NOTE or FIXME on the relevant way.

 

 

The thing that is troubling me is the size of the dataset - a few hours work 
barely makes a dent.

 

I've taken to deleting all Primary/Trunk, Secondary and Track ways from the 
data set; and then cropping stuff down to the metro adelaide area; and it's 
still very sizable.

 

 

I'd be really keen on maproulette at this point - you seem to be able to 
produce updated files fairly regularly, adding a few bash scripts to turn that 
into curl friendly statements seems achievable.

 

Going to start sending a few pull requests your way to get us started on this.

 

 

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

https://github.com/CloCkWeRX/osm-scripts/tree/add_maproulette is what I
started last night.

Biggest missing part is a persistent identifier, because I manipulated the
samples via JOSM first (thus nuking a repeatable task id); and it's not
100% clear around if I should be creating One Big Task GeoJSON or a lot of
little ones.

I doubt my ruby script is going to scale well to the whole dataset either;
so if you wanted to produce .geojson in the format of
https://raw.githubusercontent.com/CloCkWeRX/osm-scripts/add_maproulette/maproulette/task.json

then the rest is pretty easy!

See http://dev.maproulette.org/api/challenges?return_inactive=1 for the
project stub.

On Tue, Mar 10, 2015 at 11:31 AM, Henry Haselgrove 
wrote:

> I agree… with a dataset this large prioritisation is important. All your
> specific suggestions for culling parts from missing.osm sound good. It
> would be easy to add an option to the scripts to exclude highway={primary,
> secondary, track}. And the suggestions you made via github look good too.
>
>
>
> However, I think that some more significant changes to the script should
> be done before this data is unleased onto maproulette. The roads that are
> currently in missing.osm fall (more or less) under three categories:
>
> -- roads that are completely absent in OSM
>
> -- roads that appear in OSM, but have an empty name
>
> -- roads that appear in OSM and have a non-empty name
> which is different to the datasa name, either because OSM is wrong or
> datasa is wrong (or because both are right, such as “Mount Magnificent
> Road” versus “Mt. Magnificent Road”)
>
>
>
> I propose to modify the script to automatically exclude as much as
> possible from the third category. Because, it will be hard for an armchair
> mapper to decide whether OSM or datasa is wrong in those cases. I could try
> to do this over the coming week.
>
>
>
> I’m not sure I agree that the Adelaide metro area should be given
> particular priority over other areas. But I’m probably biased, since I grew
> up in rural SA!
>
>
>
> Probably we should make a posting to the osm “imports” list before too
> much longer, to let them know what we’re thinking.
>
>
>
> *From:* Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> *Sent:* Monday, 9 March 2015 5:32 AM
> *To:* Henry Haselgrove
> *Cc:* Alex Sims; OSM Australian Talk List
> *Subject:* Re: [talk-au] Using roads dataset from data.sa.gov.au
>
>
>
> So, after doing this manually for a bit; it's generally working well.
>
>
>
> There are some where spot checking against other sources suggests the
> dataset is wrong, how do you suggest we indicate these?
>
>
>
> I've put in NOTE or FIXME on the relevant way.
>
>
>
>
>
> The thing that is troubling me is the size of the dataset - a few hours
> work barely makes a dent.
>
>
>
> I've taken to deleting all Primary/Trunk, Secondary and Track ways from
> the data set; and then cropping stuff down to the metro adelaide area; and
> it's still very sizable.
>
>
>
>
>
> I'd be *really* keen on maproulette at this point - you seem to be able
> to produce updated files fairly regularly, adding a few bash scripts to
> turn that into curl friendly statements seems achievable.
>
>
>
> Going to start sending a few pull requests your way to get us started on
> this.
>
>
>
>
>
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au

[Daniel O'Connor]

For those of your that are interested, we've made a fair number of
improvements to the process and edited quite a large number of ways.

You can see the improvement via http://qa.poole.ch/ - compared to other
states/places, South Australia is showing a very low density of errors. New
Zealand, Northern Territory and Tasmania seem comparable or slightly better.

There are approximately 5000 unnamed roads remaining, and 4000 naming
differences to check after that.
___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au. [SEC=UNCLASSIFIED]

[Simon . Costello]

Hi there,

Does any of this feedback go back to the custodians in South Australia?

I am looking at improving some of the feedback loops.

Thanks,
Simon

Simon Costello
Group Leader, National Location Information  |  EGD Management
Environmental Geoscience Division  |  GEOSCIENCE AUSTRALIA

Phone:  +61 2 6249 9716Fax:  +61 2 6249 

Email:  simon.coste...@ga.gov.au<mailto:simon.coste...@ga.gov.au>Web:  
www.ga.gov.au<http://www.ga.gov.au/>
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges



On 10 Mar 2015, at 11:01 am, Henry Haselgrove 
mailto:haselgr...@gmail.com>> wrote:

I agree… with a dataset this large prioritisation is important. All your 
specific suggestions for culling parts from missing.osm sound good. It would be 
easy to add an option to the scripts to exclude highway={primary, secondary, 
track}. And the suggestions you made via github look good too.

However, I think that some more significant changes to the script should be 
done before this data is unleased onto maproulette. The roads that are 
currently in missing.osm fall (more or less) under three categories:
-- roads that are completely absent in OSM
-- roads that appear in OSM, but have an empty name
-- roads that appear in OSM and have a non-empty name which is 
different to the datasa name, either because OSM is wrong or datasa is wrong 
(or because both are right, such as “Mount Magnificent Road” versus “Mt. 
Magnificent Road”)

I propose to modify the script to automatically exclude as much as possible 
from the third category. Because, it will be hard for an armchair mapper to 
decide whether OSM or datasa is wrong in those cases. I could try to do this 
over the coming week.

I’m not sure I agree that the Adelaide metro area should be given particular 
priority over other areas. But I’m probably biased, since I grew up in rural SA!

Probably we should make a posting to the osm “imports” list before too much 
longer, to let them know what we’re thinking.

From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
Sent: Monday, 9 March 2015 5:32 AM
To: Henry Haselgrove
Cc: Alex Sims; OSM Australian Talk List
Subject: Re: [talk-au] Using roads dataset from 
data.sa.gov.au<http://data.sa.gov.au>

So, after doing this manually for a bit; it's generally working well.

There are some where spot checking against other sources suggests the dataset 
is wrong, how do you suggest we indicate these?

I've put in NOTE or FIXME on the relevant way.


The thing that is troubling me is the size of the dataset - a few hours work 
barely makes a dent.

I've taken to deleting all Primary/Trunk, Secondary and Track ways from the 
data set; and then cropping stuff down to the metro adelaide area; and it's 
still very sizable.


I'd be really keen on maproulette at this point - you seem to be able to 
produce updated files fairly regularly, adding a few bash scripts to turn that 
into curl friendly statements seems achievable.

Going to start sending a few pull requests your way to get us started on this.



Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is 
intended only for the person or entity to which it is addressed. If you are not 
the intended recipient, then you have received this e-mail by mistake and any 
use, dissemination, forwarding, printing or copying of this e-mail and its file 
attachments is prohibited. The security of emails transmitted cannot be 
guaranteed; by forwarding or replying to this email, you acknowledge and accept 
these risks.
-

___
Talk-au mailing list
Talk-au@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-au

Re: [talk-au] Using roads dataset from data.sa.gov.au. [SEC=UNCLASSIFIED]

[Daniel O'Connor]

Hi Simon,
We are communicating back corrections via the LGAs; which in turn is fed to
DPTI and presumably onto datasets like GNAF eventually.
We are still feeling out the best way to track issues or questions. I
personally like github as a slightly more effective  way than person to
person email; as it radiates information well.

Unfortunately it's not to friendly to binary files/shapefiles; so it is
best used for feedback rather than as a publication spot (perhaps there are
things that can be done with CKAN to get the best of both worlds).

I think we as a community would be very interested in any suggestions to
make feedback more relevant/effective/streamlined. At SA's unleashed
(govhack) last year, a number of us got talking to the folks behind
data.sa.gov.au and this kind of problem - how can we show what value is
being created with open data, how can we turn it into a two way
conversation, etc.
On 13/03/2015 5:04 PM,  wrote:

> Hi there,
>
> Does any of this feedback go back to the custodians in South Australia?
>
> I am looking at improving some of the feedback loops.
>
> Thanks,
> Simon
>
> Simon Costello
> Group Leader, National Location Information  |  EGD Management
> Environmental Geoscience Division  |  GEOSCIENCE AUSTRALIA
> 
> Phone:  +61 2 6249 9716Fax:  +61 2 6249
> 
> Email:  simon.coste...@ga.gov.au<mailto:simon.coste...@ga.gov.au>
> Web:  www.ga.gov.au<http://www.ga.gov.au/>
> Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT
> GPO Box 378 Canberra ACT 2601 Australia
> Applying geoscience to Australia’s most important challenges
>
>
>
> On 10 Mar 2015, at 11:01 am, Henry Haselgrove  <mailto:haselgr...@gmail.com>> wrote:
>
> I agree… with a dataset this large prioritisation is important. All your
> specific suggestions for culling parts from missing.osm sound good. It
> would be easy to add an option to the scripts to exclude highway={primary,
> secondary, track}. And the suggestions you made via github look good too.
>
> However, I think that some more significant changes to the script should
> be done before this data is unleased onto maproulette. The roads that are
> currently in missing.osm fall (more or less) under three categories:
> -- roads that are completely absent in OSM
> -- roads that appear in OSM, but have an empty name
> -- roads that appear in OSM and have a non-empty name
> which is different to the datasa name, either because OSM is wrong or
> datasa is wrong (or because both are right, such as “Mount Magnificent
> Road” versus “Mt. Magnificent Road”)
>
> I propose to modify the script to automatically exclude as much as
> possible from the third category. Because, it will be hard for an armchair
> mapper to decide whether OSM or datasa is wrong in those cases. I could try
> to do this over the coming week.
>
> I’m not sure I agree that the Adelaide metro area should be given
> particular priority over other areas. But I’m probably biased, since I grew
> up in rural SA!
>
> Probably we should make a posting to the osm “imports” list before too
> much longer, to let them know what we’re thinking.
>
> From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> Sent: Monday, 9 March 2015 5:32 AM
> To: Henry Haselgrove
> Cc: Alex Sims; OSM Australian Talk List
> Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au<
> http://data.sa.gov.au>
>
> So, after doing this manually for a bit; it's generally working well.
>
> There are some where spot checking against other sources suggests the
> dataset is wrong, how do you suggest we indicate these?
>
> I've put in NOTE or FIXME on the relevant way.
>
>
> The thing that is troubling me is the size of the dataset - a few hours
> work barely makes a dent.
>
> I've taken to deleting all Primary/Trunk, Secondary and Track ways from
> the data set; and then cropping stuff down to the metro adelaide area; and
> it's still very sizable.
>
>
> I'd be really keen on maproulette at this point - you seem to be able to
> produce updated files fairly regularly, adding a few bash scripts to turn
> that into curl friendly statements seems achievable.
>
> Going to start sending a few pull requests your way to get us started on
> this.
>
>
>
> Geoscience Australia Disclaimer: This e-mail (and files transmitted with
> it) is intended only for the person or entity to which it is addressed. If
> you are not the intended recipient, then you have received this e-mail by
> mistake and any use, dissemination, forwarding, printing or copying of this
> e-mail and its file attachments is prohi

Re: [talk-au] Using roads dataset from data.sa.gov.au. [SEC=UNCLASSIFIED]

[Daniel O'Connor]

Nice!
On 11/05/2015 6:01 PM, "ODG:Data SA"  wrote:

> Hi All
>
>
>
> We will be releasing a new Data.SA website by 1 June which will enable
> some feedback to be provided to dataset owners.  Users of Data.SA will also
> be able to comment on a dataset, request a dataset and submit case studies
> where open data has been used to solve a problem or provide a service.
> This will help citizens engage with those who release the data.
>
>
>
> Cheers
>
> Open Data Team
>
>
>
> *From:* Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> *Sent:* Friday, 13 March 2015 5:26 PM
> *To:* simon.coste...@ga.gov.au
> *Cc:* talk-au@openstreetmap.org; Henry Haselgrove; ODG:Data SA
> *Subject:* Re: [talk-au] Using roads dataset from data.sa.gov.au.
> [SEC=UNCLASSIFIED]
>
>
>
> Hi Simon,
> We are communicating back corrections via the LGAs; which in turn is fed
> to DPTI and presumably onto datasets like GNAF eventually.
> We are still feeling out the best way to track issues or questions. I
> personally like github as a slightly more effective  way than person to
> person email; as it radiates information well.
>
> Unfortunately it's not to friendly to binary files/shapefiles; so it is
> best used for feedback rather than as a publication spot (perhaps there are
> things that can be done with CKAN to get the best of both worlds).
>
> I think we as a community would be very interested in any suggestions to
> make feedback more relevant/effective/streamlined. At SA's unleashed
> (govhack) last year, a number of us got talking to the folks behind
> data.sa.gov.au and this kind of problem - how can we show what value is
> being created with open data, how can we turn it into a two way
> conversation, etc.
>
> On 13/03/2015 5:04 PM,  wrote:
>
> Hi there,
>
> Does any of this feedback go back to the custodians in South Australia?
>
> I am looking at improving some of the feedback loops.
>
> Thanks,
> Simon
>
> Simon Costello
> Group Leader, National Location Information  |  EGD Management
> Environmental Geoscience Division  |  GEOSCIENCE AUSTRALIA
> 
> Phone:  +61 2 6249 9716>
>   Fax:  +61 2 6249 >
> Email:  simon.coste...@ga.gov.au<mailto:simon.coste...@ga.gov.au>
> Web:  www.ga.gov.au<http://www.ga.gov.au/>
> Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT
> GPO Box 378 Canberra ACT 2601 Australia
> Applying geoscience to Australia’s most important challenges
>
>
>
> On 10 Mar 2015, at 11:01 am, Henry Haselgrove  <mailto:haselgr...@gmail.com>> wrote:
>
> I agree… with a dataset this large prioritisation is important. All your
> specific suggestions for culling parts from missing.osm sound good. It
> would be easy to add an option to the scripts to exclude highway={primary,
> secondary, track}. And the suggestions you made via github look good too.
>
> However, I think that some more significant changes to the script should
> be done before this data is unleased onto maproulette. The roads that are
> currently in missing.osm fall (more or less) under three categories:
> -- roads that are completely absent in OSM
> -- roads that appear in OSM, but have an empty name
> -- roads that appear in OSM and have a non-empty name
> which is different to the datasa name, either because OSM is wrong or
> datasa is wrong (or because both are right, such as “Mount Magnificent
> Road” versus “Mt. Magnificent Road”)
>
> I propose to modify the script to automatically exclude as much as
> possible from the third category. Because, it will be hard for an armchair
> mapper to decide whether OSM or datasa is wrong in those cases. I could try
> to do this over the coming week.
>
> I’m not sure I agree that the Adelaide metro area should be given
> particular priority over other areas. But I’m probably biased, since I grew
> up in rural SA!
>
> Probably we should make a posting to the osm “imports” list before too
> much longer, to let them know what we’re thinking.
>
> From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
> Sent: Monday, 9 March 2015 5:32 AM
> To: Henry Haselgrove
> Cc: Alex Sims; OSM Australian Talk List
> Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au<
> http://data.sa.gov.au>
>
> So, after doing this manually for a bit; it's generally working well.
>
> There are some where spot checking against other sources suggests the
> dataset is wrong, how do you suggest we indicate these?
>
> I've put in NOTE or FIXME on the relevant way.
>
>
> The thing that is troubling me is the size

Re: [talk-au] Using roads dataset from data.sa.gov.au. [SEC=UNCLASSIFIED]

[ODG:Data SA]

Hi All

We will be releasing a new Data.SA website by 1 June which will enable some 
feedback to be provided to dataset owners.  Users of Data.SA will also be able 
to comment on a dataset, request a dataset and submit case studies where open 
data has been used to solve a problem or provide a service.  This will help 
citizens engage with those who release the data.

Cheers
Open Data Team

From: Daniel O'Connor [mailto:daniel.ocon...@gmail.com]
Sent: Friday, 13 March 2015 5:26 PM
To: simon.coste...@ga.gov.au
Cc: talk-au@openstreetmap.org; Henry Haselgrove; ODG:Data SA
Subject: Re: [talk-au] Using roads dataset from data.sa.gov.au. 
[SEC=UNCLASSIFIED]


Hi Simon,
We are communicating back corrections via the LGAs; which in turn is fed to 
DPTI and presumably onto datasets like GNAF eventually.
We are still feeling out the best way to track issues or questions. I 
personally like github as a slightly more effective  way than person to person 
email; as it radiates information well.

Unfortunately it's not to friendly to binary files/shapefiles; so it is best 
used for feedback rather than as a publication spot (perhaps there are things 
that can be done with CKAN to get the best of both worlds).

I think we as a community would be very interested in any suggestions to make 
feedback more relevant/effective/streamlined. At SA's unleashed (govhack) last 
year, a number of us got talking to the folks behind 
data.sa.gov.au<http://data.sa.gov.au> and this kind of problem - how can we 
show what value is being created with open data, how can we turn it into a two 
way conversation, etc.
On 13/03/2015 5:04 PM, 
mailto:simon.coste...@ga.gov.au>> wrote:
Hi there,

Does any of this feedback go back to the custodians in South Australia?

I am looking at improving some of the feedback loops.

Thanks,
Simon

Simon Costello
Group Leader, National Location Information  |  EGD Management
Environmental Geoscience Division  |  GEOSCIENCE AUSTRALIA

Phone:  +61 2 6249 9716 
   Fax:  +61 2 6249 
Email:  
simon.coste...@ga.gov.au<mailto:simon.coste...@ga.gov.au><mailto:simon.coste...@ga.gov.au<mailto:simon.coste...@ga.gov.au>>
Web:  www.ga.gov.au<http://www.ga.gov.au><http://www.ga.gov.au/>
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges



On 10 Mar 2015, at 11:01 am, Henry Haselgrove 
mailto:haselgr...@gmail.com><mailto:haselgr...@gmail.com<mailto:haselgr...@gmail.com>>>
 wrote:

I agree… with a dataset this large prioritisation is important. All your 
specific suggestions for culling parts from missing.osm sound good. It would be 
easy to add an option to the scripts to exclude highway={primary, secondary, 
track}. And the suggestions you made via github look good too.

However, I think that some more significant changes to the script should be 
done before this data is unleased onto maproulette. The roads that are 
currently in missing.osm fall (more or less) under three categories:
-- roads that are completely absent in OSM
-- roads that appear in OSM, but have an empty name
-- roads that appear in OSM and have a non-empty name which is 
different to the datasa name, either because OSM is wrong or datasa is wrong 
(or because both are right, such as “Mount Magnificent Road” versus “Mt. 
Magnificent Road”)

I propose to modify the script to automatically exclude as much as possible 
from the third category. Because, it will be hard for an armchair mapper to 
decide whether OSM or datasa is wrong in those cases. I could try to do this 
over the coming week.

I’m not sure I agree that the Adelaide metro area should be given particular 
priority over other areas. But I’m probably biased, since I grew up in rural SA!

Probably we should make a posting to the osm “imports” list before too much 
longer, to let them know what we’re thinking.

From: Daniel O'Connor 
[mailto:daniel.ocon...@gmail.com<mailto:daniel.ocon...@gmail.com>]
Sent: Monday, 9 March 2015 5:32 AM
To: Henry Haselgrove
Cc: Alex Sims; OSM Australian Talk List
Subject: Re: [talk-au] Using roads dataset from 
data.sa.gov.au<http://data.sa.gov.au><http://data.sa.gov.au>

So, after doing this manually for a bit; it's generally working well.

There are some where spot checking against other sources suggests the dataset 
is wrong, how do you suggest we indicate these?

I've put in NOTE or FIXME on the relevant way.


The thing that is troubling me is the size of the dataset - a few hours work 
barely makes a dent.

I've taken to deleting all Primary/Trunk, Secondary and Track ways from the 
data set; and then cropping stuff down to the metro adelaide area; and it's 
still very sizable.


I'd be really keen on maproulette a