Re: [Talk-ca] City of Ottawa imported buildings & addresses

[Frederik Ramm]

Hi,

On 10/18/2016 07:53 PM, Frederik Ramm wrote:
> Unfortunately the import was larger than expected and the revert drags
> on. 

And it still does! I was going a bit slower today in order not to
overload the API.

Thank you everyone for letting the revert run its course without
intervening further. Due to the fact that I hadn't spotted the full
extent of the import initially and missed out some bits, I have to use a
slightly different procedure than usual which may in some areas lead to
- temporarily - orphaned nodes where a building way has been deleted and
the nodes still stand; but the nodes are going to go away eventually and
I'll take care not to leave anything behind.

A few smaller import changesets had a lot of "normal" edits mixed in
with the import, and I manually excluded these changesets from the
revert; this means that a few imported buildings will also remain
standing but I'd rather err on the side of caution.

I saw that there have been heated discussions in some changeset comments
about "deleting all our work" - rest assured I'm only reverting a bad
import that goes back about two weeks at most. Should I accidentally
delete something that was not part of the import, we'll get it back
after the dust has settled.

I have no grudge with the people who have imported the data; I revert
something because the process was wrong, not because I dislike the
contributor.

A large number of objects has been uploaded in these past two weeks. If
I can recommend something for future imports, try to do them in smaller
chunks and take your time. Don't pack 50k nodes in a changeset and
upload several of them per day. If you spread things out over a couple
weeks or months, you'll put less strain on the API and ensure that *if*
something turns out problematic (which can happen even if following the
import rules to the letter!) you're not suddenly looking at a revert
that goes full throttle for several days.

I'll post updates about the revert as necessary.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] [Import] Ottawa Buildings & Addresses [Statistics Canada project]

[James]

I was wrong there is one sign up(i'm glad I drove there)
http://openstreetview.com/details/21585/18
See? no "ê".

On Thu, Oct 20, 2016 at 10:37 AM, James  wrote:

> What is highly questionable about two levels of Government reaching out to
> the community to help them with complying/importing authoritative data? US
> has tiger data(I'm aware it's not the greatest example), Canada has
> (CanVec). Now we are down to a municipal level and the data(especially
> addresses that they maintain a database of for emergency services, etc) is
> incorrect and doesnt match real life. As for the building footprints I have
> addressed this already:
> https://wiki.openstreetmap.org/wiki/Talk:Canada:Ontario:
> Ottawa/Import/Plan#Data_quality
>
> We seem to be turning in a circle around "building source data is not
> publicly available", it's not because of it's held up by bureaucracy which
> is non-constructive. The source data was handed to StatsCan which we use as
> our source(I linked you the geojson that has 2 attributes added to it so
> JOSM wouldn't render a bunch of tagless ways and could make sense of it, if
> you want the original data: remove both tags).
>
> I completely understand that you don't want a bunch of garbage data added
> to OSM that will take years to clean up, neither do we, hell it's where we
> live and we'd be only punishing ourselves! The thing you have to understand
> is this preparation has been going for over a year now (see:
> https://wiki.openstreetmap.org/w/index.php?title=Canada:
> Ontario:Ottawa/Import&action=history(documentation started)). Even
> though we had an incompatible license, the data was still available to play
> with and analyse
>
> I get your concerns of problems that may arise during this import, this is
> what this process is for is it not? To address concerns people may have.
>
> On Thu, Oct 20, 2016 at 9:50 AM, James  wrote:
>
>> If you are complaining that it doesnt have "ê" you are going to have to
>> talk to the city of ottawa as that street does not have it(I know it I live
>> near there). There is no street sign as it is a private development. Take a
>> look at:
>> http://maps.ottawa.ca/geoottawa/
>> or google maps or even geobase roads(canvec) there is no "ê"
>> or even: http://www.openstreetmap.org/#map=19/45.47941/-75.52464
>>
>> Maybe you can create a language board as Québec did to control French
>> display, but in Ontario! (https://www.oqlf.gouv.qc.ca/accueil.aspx)
>>
>> On Thu, Oct 20, 2016 at 9:47 AM, Christoph Hormann 
>> wrote:
>>
>>> On Thursday 20 October 2016, James wrote:
>>> > Why are you linking a french wiki page of forest?
>>> >
>>> > Un Foret
>>> > Des Forets
>>> >
>>>
>>> I think it's "des forêts".
>>>
>>> Notice the difference?
>>>
>>> --
>>> Christoph Hormann
>>> http://www.imagico.de/
>>>
>>
>>
>>
>> --
>> 外に遊びに行こう！
>>
>
>
>
> --
> 外に遊びに行こう！
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] [Import] Ottawa Buildings & Addresses [Statistics Canada project]

[James]

What is highly questionable about two levels of Government reaching out to
the community to help them with complying/importing authoritative data? US
has tiger data(I'm aware it's not the greatest example), Canada has
(CanVec). Now we are down to a municipal level and the data(especially
addresses that they maintain a database of for emergency services, etc) is
incorrect and doesnt match real life. As for the building footprints I have
addressed this already:
https://wiki.openstreetmap.org/wiki/Talk:Canada:Ontario:Ottawa/Import/Plan#Data_quality

We seem to be turning in a circle around "building source data is not
publicly available", it's not because of it's held up by bureaucracy which
is non-constructive. The source data was handed to StatsCan which we use as
our source(I linked you the geojson that has 2 attributes added to it so
JOSM wouldn't render a bunch of tagless ways and could make sense of it, if
you want the original data: remove both tags).

I completely understand that you don't want a bunch of garbage data added
to OSM that will take years to clean up, neither do we, hell it's where we
live and we'd be only punishing ourselves! The thing you have to understand
is this preparation has been going for over a year now (see:
https://wiki.openstreetmap.org/w/index.php?title=Canada:Ontario:Ottawa/Import&action=history
(documentation started)). Even though we had an incompatible license, the
data was still available to play with and analyse

I get your concerns of problems that may arise during this import, this is
what this process is for is it not? To address concerns people may have.

On Thu, Oct 20, 2016 at 9:50 AM, James  wrote:

> If you are complaining that it doesnt have "ê" you are going to have to
> talk to the city of ottawa as that street does not have it(I know it I live
> near there). There is no street sign as it is a private development. Take a
> look at:
> http://maps.ottawa.ca/geoottawa/
> or google maps or even geobase roads(canvec) there is no "ê"
> or even: http://www.openstreetmap.org/#map=19/45.47941/-75.52464
>
> Maybe you can create a language board as Québec did to control French
> display, but in Ontario! (https://www.oqlf.gouv.qc.ca/accueil.aspx)
>
> On Thu, Oct 20, 2016 at 9:47 AM, Christoph Hormann 
> wrote:
>
>> On Thursday 20 October 2016, James wrote:
>> > Why are you linking a french wiki page of forest?
>> >
>> > Un Foret
>> > Des Forets
>> >
>>
>> I think it's "des forêts".
>>
>> Notice the difference?
>>
>> --
>> Christoph Hormann
>> http://www.imagico.de/
>>
>
>
>
> --
> 外に遊びに行こう！
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] [Import] Ottawa Buildings & Addresses [Statistics Canada project]

[James]

If you are complaining that it doesnt have "ê" you are going to have to
talk to the city of ottawa as that street does not have it(I know it I live
near there). There is no street sign as it is a private development. Take a
look at:
http://maps.ottawa.ca/geoottawa/
or google maps or even geobase roads(canvec) there is no "ê"
or even: http://www.openstreetmap.org/#map=19/45.47941/-75.52464

Maybe you can create a language board as Québec did to control French
display, but in Ontario! (https://www.oqlf.gouv.qc.ca/accueil.aspx)

On Thu, Oct 20, 2016 at 9:47 AM, Christoph Hormann 
wrote:

> On Thursday 20 October 2016, James wrote:
> > Why are you linking a french wiki page of forest?
> >
> > Un Foret
> > Des Forets
> >
>
> I think it's "des forêts".
>
> Notice the difference?
>
> --
> Christoph Hormann
> http://www.imagico.de/
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] City of Ottawa imported buildings & addresses

[James]

Are you done the revert yet? I got a message from:
lesc 

"Hi,

It looks like some changes in Kanata you did removed all the lines that
represent buildings and only the corners are left as points.

Thanks, Les"

Seems like your reverting is messing up data and others are noticing. It's
been 2 days now...should it not have completed by now?


I've told him to direct his inquiries to you as you are the one in charge
of the mass revert.

On Tue, Oct 18, 2016 at 1:53 PM, Frederik Ramm  wrote:

> Hi,
>
> On 10/17/2016 11:59 PM, Frederik Ramm wrote:
> > The person responsible for cleaning up a poor revert should be the
> > person who ran it ;) it's only 30% complete and will run far into the
> > night in my time zone and I'll have to check on it after getting up. I'm
> > confident all will be fixed when you get up tomorrow morning.
>
> Unfortunately the import was larger than expected and the revert drags
> on. Meanwhile, a couple of accounts have been newly created by parties
> unknown ("addxy_imports", "ottawa_import") and these have (accidentally
> or purposefully) interfered with the revert, meaning that it will take
> even longer for me do this right.
>
> I would like to appeal to all involved parties to show some maturity.
> The import was in clear violation of established processes; it must be
> reverted, and then the community can - calmly and without any time
> pressure - decide what they want to do with the data.
>
> I haven't analysed the import in depth but I have seen a couple of
> examples where a perfectly well mapped building was wiped clean and
> replaced with one that was not at all better - this is clearly something
> we don't want to see in an import, it is a technical (or procedural)
> shortcoming that would definitely have been pointed out had there been a
> proper discussion beforehand.
>
> The requirement to talk about imports before you act is not an
> unnecessary bueraucratic hurdle; it is intended to avoid disappointment
> on all sides. An import that fears the broad daylight is probably one
> that should not be attempted at all!
>
> I should maybe have made that clearer in my initial email but I'm acting
> here as a member of the OSMF's data working group in response to a
> legitimate complaint, not as a German mapper seeking trouble. I
> sincerely ask everyone involved to keep calm and let the revert complete
> cleanly.
>
> I see that user LogicalViolinist has already found fields of endeavour
> outside of Canada for the time being.
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> Talk-ca mailing list
> Talk-ca@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] [Import] Ottawa Buildings & Addresses [Statistics Canada project]

[James]

as an added note, this is where Osmose could tell us what street needs
attention/renaming (as there is a filter to tell us what street is not
written the same as the road it is close to) which we can easily identify
from the city of Ottawa website.



On Thu, Oct 20, 2016 at 8:06 AM, James  wrote:

> Christoph, the source data for buildings has 0 tags that need to be
> translated as it is only building footprints and *0 attributes*. The
> source on an Ottawa web site/federal website is non-existant for the time
> being as: the city of ottawa litterally gave statscan a file on a flash
> drive to deal with it: *this is precisely why I put background in the
> documentation as the source for buildings will probably not be available
> from a official website for years(judging how slow things move in the
> government)* you will just have to accept that as it was handed to us
> from an official source the two tags that were added were "building" and
> "source" polygon coordinates are the same. If you really want I can strip
> all the "properties":{"building":"yes","source":"City of Ottawa"}  and it
> will be the same file. We are using this as the source. You will see that
> there are 1000s upon thousands of tiny garages/sheds and other junk we
> probably don't need in the geojson I have linked.
>
> St-Germain Crescent (Saint-Germain Crescent and yes it's written that
> way(St-))
> St-Emmanuel Terrace (Saint-Emmanuel Terrace and yes it's written that
> way(St-))
> Des Forets Avenue (What's wrong with this? There are french road names
> especially in orleans "Des Forest" means Forests in English)
> We are aware of Jeanne-D'Arc and have corrected it manually (there are a
> handful of roads we've Identified as "problematic", all in Orleans (french
> part of town they decided to put french and english together)
>
>
>
>
> *There are discrepancies between the street names in the address nodes and
> the road names in OSM by the way, mostly due to wrong names in OSM
> (probably due to bad previous imports).*
>
> O the city of ottawa feels it necessary to rename a couple
> hundred roads per year and just hasnt been updated in OSM.
> http://ottawa.ca/en/city-hall/public-consultations/
> municipal-addressing/street-name-and-address-changes-update-october
>
> On Thu, Oct 20, 2016 at 7:40 AM, Christoph Hormann 
> wrote:
>
>> On Thursday 20 October 2016, James wrote:
>> > If you really need source data it will be available when the
>> > goverment makes up their mind where to host it. If you need it we
>> > have it in GeoJSON format here:
>> > https://s3-us-west-2.amazonaws.com/addxy.com/ottawa-buildings.geojson
>>
>> That is certainly not the source data, it looks like a simple geojson
>> version of the OSM files.
>>
>> > Original file shouldn't matter anyways as it's full of errors to
>> > begin with.
>>
>> As with the addresses the point is to assess the conversion process and
>> the quality and suitability of the original data.  If you have fixed
>> errors in this data this should be documented and reviewed as part of
>> the import review.  What especially matters is the original metadata
>> and specifications, like for example what a building is - something
>> like a loading ramp/platform for example is not a building or part of a
>> building in OSM but it might be in the source data set - which then
>> would require special consideration.
>>
>> > As for addresses we have used a set of internal tools(available on
>> > github) to rename and de-retard the original shape file like:
>> > https://github.com/osmottawa/etl2osm
>> > I also have a .net application at work that can do the same
>> > processing.
>>
>> Overall this seems to work fine but it fails with some names like:
>>
>> St-Germain Crescent
>> St-Emmanuel Terrace
>> (unless this is the way it is written on the street signs of course)
>>
>> Des Forets Avenue
>> Jeanne-d'Arc Boul. N. Boulevard North
>>
>> There are discrepancies between the street names in the address nodes
>> and the road names in OSM by the way, mostly due to wrong names in OSM
>> (probably due to bad previous imports).
>>
>> --
>> Christoph Hormann
>> http://www.imagico.de/
>>
>
>
>
> --
> 外に遊びに行こう！
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] [Import] Ottawa Buildings & Addresses [Statistics Canada project]

[James]

Christoph, the source data for buildings has 0 tags that need to be
translated as it is only building footprints and *0 attributes*. The source
on an Ottawa web site/federal website is non-existant for the time being
as: the city of ottawa litterally gave statscan a file on a flash drive to
deal with it: *this is precisely why I put background in the documentation
as the source for buildings will probably not be available from a official
website for years(judging how slow things move in the government)* you will
just have to accept that as it was handed to us from an official source the
two tags that were added were "building" and "source" polygon coordinates
are the same. If you really want I can strip all the
"properties":{"building":"yes","source":"City of Ottawa"}  and it will be
the same file. We are using this as the source. You will see that there are
1000s upon thousands of tiny garages/sheds and other junk we probably don't
need in the geojson I have linked.

St-Germain Crescent (Saint-Germain Crescent and yes it's written that
way(St-))
St-Emmanuel Terrace (Saint-Emmanuel Terrace and yes it's written that
way(St-))
Des Forets Avenue (What's wrong with this? There are french road names
especially in orleans "Des Forest" means Forests in English)
We are aware of Jeanne-D'Arc and have corrected it manually (there are a
handful of roads we've Identified as "problematic", all in Orleans (french
part of town they decided to put french and english together)




*There are discrepancies between the street names in the address nodes and
the road names in OSM by the way, mostly due to wrong names in OSM
(probably due to bad previous imports).*

O the city of ottawa feels it necessary to rename a couple
hundred roads per year and just hasnt been updated in OSM.
http://ottawa.ca/en/city-hall/public-consultations/municipal-addressing/street-name-and-address-changes-update-october

On Thu, Oct 20, 2016 at 7:40 AM, Christoph Hormann 
wrote:

> On Thursday 20 October 2016, James wrote:
> > If you really need source data it will be available when the
> > goverment makes up their mind where to host it. If you need it we
> > have it in GeoJSON format here:
> > https://s3-us-west-2.amazonaws.com/addxy.com/ottawa-buildings.geojson
>
> That is certainly not the source data, it looks like a simple geojson
> version of the OSM files.
>
> > Original file shouldn't matter anyways as it's full of errors to
> > begin with.
>
> As with the addresses the point is to assess the conversion process and
> the quality and suitability of the original data.  If you have fixed
> errors in this data this should be documented and reviewed as part of
> the import review.  What especially matters is the original metadata
> and specifications, like for example what a building is - something
> like a loading ramp/platform for example is not a building or part of a
> building in OSM but it might be in the source data set - which then
> would require special consideration.
>
> > As for addresses we have used a set of internal tools(available on
> > github) to rename and de-retard the original shape file like:
> > https://github.com/osmottawa/etl2osm
> > I also have a .net application at work that can do the same
> > processing.
>
> Overall this seems to work fine but it fails with some names like:
>
> St-Germain Crescent
> St-Emmanuel Terrace
> (unless this is the way it is written on the street signs of course)
>
> Des Forets Avenue
> Jeanne-d'Arc Boul. N. Boulevard North
>
> There are discrepancies between the street names in the address nodes
> and the road names in OSM by the way, mostly due to wrong names in OSM
> (probably due to bad previous imports).
>
> --
> Christoph Hormann
> http://www.imagico.de/
>



-- 
外に遊びに行こう！
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk-ca