Re: [Talk-ca] Importing buildings in Canada

[Tim Elrick]

Thank you, Daniel, for advancing on the import! I agree with most of 
what you said below (I will reply to some details on the wiki page).


I also agree to separate the different sources as I have great 
reservations about using Microsoft's building footprints for an import 
into OSM (the outlines are simply not good enough in their topology).


Let's get this going,
Tim




On 2020-01-03 15:40, Daniel @jfd553 wrote:
Bonjour groupe, mes excuses pour ce très long courriel !-)

I have reviewed everything that has been written on the ODB import (aka
Canada Building Import) in Talk-ca and the wiki. I proposed changes to
some wiki pages (via talk tabs) to ease the discussions about this
import and the following. Now, in order to restart the import, here are
some thoughts and a proposal on how to proceed to complete the task.

*1. Issues with the ODB Data Import*

Many concerns were raised about the import. One major concern was to
obtain local communities’ buy-in in the Canadian context. Another
concern was to improve the quality of the data prior the import. The
following paragraphs intend to clear most of these concerns.

*1.1. Which data import project?*

According to the import guidelines (steps 3 & 4), a data import
explicitly refers to a single data source (ODB in our case). Discussions
about the availability and quality of Microsoft or ESRI data, while
interesting, are not relevant as they should be dealt with as other
import projects.

*1.2. What has been imported so far?*

According to what I found [1], the ODB import is completed for 21
municipalities. These imports seem to have kept OSM content’s history,
at least for the samples checked, but many problems were found. In some
case, the imports brought swimming pools in OSM because they were
included in the dataset (e.g. Moncton). In other cases, importing
buildings with accurate locations (XY) over content mapped from less
accurate imagery resulted in buildings that now overlap the street
network (e.g. Squamish). It means that all these 21 imports need to be
carefully re-examined and corrected as required.

For 12 other municipalities, the import is partial, either suspended as
requested, or because previous imports had already provided most of the
buildings (often from the same municipal provider). That said the import
will definitely improve OSM accuracy and completeness if done properly.

*2. How should ODB Data be imported?*

I will copy the following paragraphs in the “Canada Building Import”
wiki page [3] for a detailed discussion…

Since the data (ODB, OSM and imagery) differ from one municipality to
another, there can be no imports at the national or provincial level. We
have to work on a municipal basis and make sure to identify all the
problems and the corrective measures to apply when dealing with issues
like those I identified [1].

*2.1 Importing Locally*

According to the import guidelines (step 2), we must not import the data
without local buy-in. However, and contrarily to some European country,
there is usually no such thing as a local OSM community in each
municipality. However, we may find a few local mappers from time to
time. Working on a municipal basis should allow identifying these local
mappers before doing the import. I often use this tool [2] to identify
and contact local mappers. Once identified, I suggest that…

- We contact them to explain our intents by referring to appropriate
wiki pages.

- We wait a week or two to let them respond nothing, that they have
concerns, or wish to help.

- Without negative answers we could proceed to the import.

I first suggest that when a contributor wishes to import ODB for a given
municipality, he first identifies himself as responsible for the import
(we need to create an entry for each municipality somewhere in the
wiki). He can then contact local mappers, as explain above, and go ahead
with the import once everything settled. For those who already made the
import, I suggest that they review their work since many issues were
detected with some of these imports.

Since there are only a few local OSM communities in Canada, and because
Canada is large, I suggest not limiting the import of a given
municipality to the people of the concerned province or region.

*2.2 Pre-processing*

Once local mappers have agreed, some pre-processing can be done if
required.

A few months ago, I developed a tool that could be used to process the
data [4]. Concerns were raised because the application was developed
using proprietary software. So I documented the whole process and
algorithms in order to see courageous coders converting it in open
source software. In the meantime, and as long as I have access to an FME
licence, I could process the data, when necessary, prior to make it
available through the task manager.

Proposed pre-processing [4] includes:

- Reading of original ODB data,

- Removal of near collinear nodes (simplification),

- Orthogonalization of buildings (for corners having near 

Re: [Talk-ca] Importing buildings in Canada

[john whelan]

Sounds sane to me.

Thanks John

On Fri, 3 Jan 2020 at 16:05, Daniel @jfd553  wrote:

> Hi John,
>
> To comply with the import directive, I would like the discussion to
> continue on the current ODB import only. Could we talk about other sources
> of potential imports at another time? -)
>
>
>
> Thanks
>
> Daniel
>
> *From:* john whelan [mailto:jwhelan0...@gmail.com]
> *Sent:* Friday, January 03, 2020 15:57
> *To:* Daniel @jfd553
> *Cc:* talk-ca@openstreetmap.org
> *Subject:* Re: [Talk-ca] Importing buildings in Canada
>
>
>
> Sounds sane to me.
>
>
>
> Are we thinking of setting up a second import to handle Microsoft building
> outlines on a similar basis? Or extending this one?  I only ask since if we
> are doing it municipality by municipality it might be an idea to identify
> those municipalities that do not have suitable open data through stats
> canada.  I can see us with a list of municipalities and a data source and I
> think it probably should be in one place.
>
>
>
> Cheerio John
>
>
>
> On Fri, 3 Jan 2020 at 15:42, Daniel @jfd553  wrote:
>
> Bonjour groupe, mes excuses pour ce très long courriel !-)
>
> I have reviewed everything that has been written on the ODB import (aka
> Canada Building Import) in Talk-ca and the wiki. I proposed changes to some
> wiki pages (via talk tabs) to ease the discussions about this import and
> the following. Now, in order to restart the import, here are some thoughts
> and a proposal on how to proceed to complete the task.
>
>
>
> *1. Issues with the ODB Data Import*
>
> Many concerns were raised about the import. One major concern was to
> obtain local communities’ buy-in in the Canadian context. Another concern
> was to improve the quality of the data prior the import. The following
> paragraphs intend to clear most of these concerns.
>
> *1.1. Which data import project?*
>
> According to the import guidelines (steps 3 & 4), a data import explicitly
> refers to a single data source (ODB in our case). Discussions about the
> availability and quality of Microsoft or ESRI data, while interesting, are
> not relevant as they should be dealt with as other import projects.
>
> *1.2. What has been imported so far?*
>
> According to what I found [1], the ODB import is completed for 21
> municipalities. These imports seem to have kept OSM content’s history, at
> least for the samples checked, but many problems were found. In some case,
> the imports brought swimming pools in OSM because they were included in the
> dataset (e.g. Moncton). In other cases, importing buildings with accurate
> locations (XY) over content mapped from less accurate imagery resulted in
> buildings that now overlap the street network (e.g. Squamish). It means
> that all these 21 imports need to be carefully re-examined and corrected as
> required.
>
> For 12 other municipalities, the import is partial, either suspended as
> requested, or because previous imports had already provided most of the
> buildings (often from the same municipal provider). That said the import
> will definitely improve OSM accuracy and completeness if done properly.
>
> *2. How should ODB Data be imported?*
>
> I will copy the following paragraphs in the “Canada Building Import” wiki
> page [3] for a detailed discussion…
>
> Since the data (ODB, OSM and imagery) differ from one municipality to
> another, there can be no imports at the national or provincial level. We
> have to work on a municipal basis and make sure to identify all the
> problems and the corrective measures to apply when dealing with issues like
> those I identified [1].
>
> *2.1 Importing Locally*
>
> According to the import guidelines (step 2), we must not import the data
> without local buy-in. However, and contrarily to some European country,
> there is usually no such thing as a local OSM community in each
> municipality. However, we may find a few local mappers from time to time.
> Working on a municipal basis should allow identifying these local mappers
> before doing the import. I often use this tool [2] to identify and contact
> local mappers. Once identified, I suggest that…
>
> - We contact them to explain our intents by referring to appropriate wiki
> pages.
>
> - We wait a week or two to let them respond nothing, that they have
> concerns, or wish to help.
>
> - Without negative answers we could proceed to the import.
>
> I first suggest that when a contributor wishes to import ODB for a given
> municipality, he first identifies himself as responsible for the import (we
> need to create an entry for each municipality somewhere in the wiki). He
> can then contact local mappers, as explain above, and go ahead with the
> import once everything settled. For those who already made the import, I
> suggest that they review their work since many issues were detected with
> some of these imports.
>
> Since there are only a few local OSM communities in Canada, and because
> Canada is large, I suggest not limiting the import of a given municipality
> 

Re: [Talk-ca] Importing buildings in Canada

[Daniel @jfd553]

That is why I proposed changes in some wiki page (talk), to deal with many 
imports, without mixing discussions :-)

Daniel

From: Daniel @jfd553 [mailto:jfd...@hotmail.com]
Sent: Friday, January 03, 2020 16:06
To: john whelan; Daniel @jfd553
Cc: talk-ca@openstreetmap.org
Subject: RE: [Talk-ca] Importing buildings in Canada

Hi John,
To comply with the import directive, I would like the discussion to continue on 
the current ODB import only. Could we talk about other sources of potential 
imports at another time? -)

Thanks
Daniel
From: john whelan [mailto:jwhelan0...@gmail.com]
Sent: Friday, January 03, 2020 15:57
To: Daniel @jfd553
Cc: talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] Importing buildings in Canada

Sounds sane to me.

Are we thinking of setting up a second import to handle Microsoft building 
outlines on a similar basis? Or extending this one?  I only ask since if we are 
doing it municipality by municipality it might be an idea to identify those 
municipalities that do not have suitable open data through stats canada.  I can 
see us with a list of municipalities and a data source and I think it probably 
should be in one place.

Cheerio John

On Fri, 3 Jan 2020 at 15:42, Daniel @jfd553 
mailto:jfd...@hotmail.com>> wrote:
Bonjour groupe, mes excuses pour ce très long courriel !-)
I have reviewed everything that has been written on the ODB import (aka Canada 
Building Import) in Talk-ca and the wiki. I proposed changes to some wiki pages 
(via talk tabs) to ease the discussions about this import and the following. 
Now, in order to restart the import, here are some thoughts and a proposal on 
how to proceed to complete the task.

1. Issues with the ODB Data Import
Many concerns were raised about the import. One major concern was to obtain 
local communities’ buy-in in the Canadian context. Another concern was to 
improve the quality of the data prior the import. The following paragraphs 
intend to clear most of these concerns.
1.1. Which data import project?
According to the import guidelines (steps 3 & 4), a data import explicitly 
refers to a single data source (ODB in our case). Discussions about the 
availability and quality of Microsoft or ESRI data, while interesting, are not 
relevant as they should be dealt with as other import projects.
1.2. What has been imported so far?
According to what I found [1], the ODB import is completed for 21 
municipalities. These imports seem to have kept OSM content’s history, at least 
for the samples checked, but many problems were found. In some case, the 
imports brought swimming pools in OSM because they were included in the dataset 
(e.g. Moncton). In other cases, importing buildings with accurate locations 
(XY) over content mapped from less accurate imagery resulted in buildings that 
now overlap the street network (e.g. Squamish). It means that all these 21 
imports need to be carefully re-examined and corrected as required.
For 12 other municipalities, the import is partial, either suspended as 
requested, or because previous imports had already provided most of the 
buildings (often from the same municipal provider). That said the import will 
definitely improve OSM accuracy and completeness if done properly.
2. How should ODB Data be imported?
I will copy the following paragraphs in the “Canada Building Import” wiki page 
[3] for a detailed discussion…
Since the data (ODB, OSM and imagery) differ from one municipality to another, 
there can be no imports at the national or provincial level. We have to work on 
a municipal basis and make sure to identify all the problems and the corrective 
measures to apply when dealing with issues like those I identified [1].
2.1 Importing Locally
According to the import guidelines (step 2), we must not import the data 
without local buy-in. However, and contrarily to some European country, there 
is usually no such thing as a local OSM community in each municipality. 
However, we may find a few local mappers from time to time. Working on a 
municipal basis should allow identifying these local mappers before doing the 
import. I often use this tool [2] to identify and contact local mappers. Once 
identified, I suggest that…
- We contact them to explain our intents by referring to appropriate wiki pages.
- We wait a week or two to let them respond nothing, that they have concerns, 
or wish to help.
- Without negative answers we could proceed to the import.
I first suggest that when a contributor wishes to import ODB for a given 
municipality, he first identifies himself as responsible for the import (we 
need to create an entry for each municipality somewhere in the wiki). He can 
then contact local mappers, as explain above, and go ahead with the import once 
everything settled. For those who already made the import, I suggest that they 
review their work since many issues were detected with some of these imports.
Since there are only a few local OSM communities in Canada, and because Canada 
is large, 

Re: [Talk-ca] Importing buildings in Canada

[Daniel @jfd553]

Hi John,
To comply with the import directive, I would like the discussion to continue on 
the current ODB import only. Could we talk about other sources of potential 
imports at another time? -)

Thanks
Daniel
From: john whelan [mailto:jwhelan0...@gmail.com]
Sent: Friday, January 03, 2020 15:57
To: Daniel @jfd553
Cc: talk-ca@openstreetmap.org
Subject: Re: [Talk-ca] Importing buildings in Canada

Sounds sane to me.

Are we thinking of setting up a second import to handle Microsoft building 
outlines on a similar basis? Or extending this one?  I only ask since if we are 
doing it municipality by municipality it might be an idea to identify those 
municipalities that do not have suitable open data through stats canada.  I can 
see us with a list of municipalities and a data source and I think it probably 
should be in one place.

Cheerio John

On Fri, 3 Jan 2020 at 15:42, Daniel @jfd553 
mailto:jfd...@hotmail.com>> wrote:
Bonjour groupe, mes excuses pour ce très long courriel !-)
I have reviewed everything that has been written on the ODB import (aka Canada 
Building Import) in Talk-ca and the wiki. I proposed changes to some wiki pages 
(via talk tabs) to ease the discussions about this import and the following. 
Now, in order to restart the import, here are some thoughts and a proposal on 
how to proceed to complete the task.

1. Issues with the ODB Data Import
Many concerns were raised about the import. One major concern was to obtain 
local communities’ buy-in in the Canadian context. Another concern was to 
improve the quality of the data prior the import. The following paragraphs 
intend to clear most of these concerns.
1.1. Which data import project?
According to the import guidelines (steps 3 & 4), a data import explicitly 
refers to a single data source (ODB in our case). Discussions about the 
availability and quality of Microsoft or ESRI data, while interesting, are not 
relevant as they should be dealt with as other import projects.
1.2. What has been imported so far?
According to what I found [1], the ODB import is completed for 21 
municipalities. These imports seem to have kept OSM content’s history, at least 
for the samples checked, but many problems were found. In some case, the 
imports brought swimming pools in OSM because they were included in the dataset 
(e.g. Moncton). In other cases, importing buildings with accurate locations 
(XY) over content mapped from less accurate imagery resulted in buildings that 
now overlap the street network (e.g. Squamish). It means that all these 21 
imports need to be carefully re-examined and corrected as required.
For 12 other municipalities, the import is partial, either suspended as 
requested, or because previous imports had already provided most of the 
buildings (often from the same municipal provider). That said the import will 
definitely improve OSM accuracy and completeness if done properly.
2. How should ODB Data be imported?
I will copy the following paragraphs in the “Canada Building Import” wiki page 
[3] for a detailed discussion…
Since the data (ODB, OSM and imagery) differ from one municipality to another, 
there can be no imports at the national or provincial level. We have to work on 
a municipal basis and make sure to identify all the problems and the corrective 
measures to apply when dealing with issues like those I identified [1].
2.1 Importing Locally
According to the import guidelines (step 2), we must not import the data 
without local buy-in. However, and contrarily to some European country, there 
is usually no such thing as a local OSM community in each municipality. 
However, we may find a few local mappers from time to time. Working on a 
municipal basis should allow identifying these local mappers before doing the 
import. I often use this tool [2] to identify and contact local mappers. Once 
identified, I suggest that…
- We contact them to explain our intents by referring to appropriate wiki pages.
- We wait a week or two to let them respond nothing, that they have concerns, 
or wish to help.
- Without negative answers we could proceed to the import.
I first suggest that when a contributor wishes to import ODB for a given 
municipality, he first identifies himself as responsible for the import (we 
need to create an entry for each municipality somewhere in the wiki). He can 
then contact local mappers, as explain above, and go ahead with the import once 
everything settled. For those who already made the import, I suggest that they 
review their work since many issues were detected with some of these imports.
Since there are only a few local OSM communities in Canada, and because Canada 
is large, I suggest not limiting the import of a given municipality to the 
people of the concerned province or region.
2.2 Pre-processing
Once local mappers have agreed, some pre-processing can be done if required.
A few months ago, I developed a tool that could be used to process the data 
[4]. Concerns were raised because the 

Re: [Talk-ca] Importing buildings in Canada

[john whelan]

Sounds sane to me.

Are we thinking of setting up a second import to handle Microsoft building
outlines on a similar basis? Or extending this one?  I only ask since if we
are doing it municipality by municipality it might be an idea to identify
those municipalities that do not have suitable open data through stats
canada.  I can see us with a list of municipalities and a data source and I
think it probably should be in one place.

Cheerio John

On Fri, 3 Jan 2020 at 15:42, Daniel @jfd553  wrote:

> Bonjour groupe, mes excuses pour ce très long courriel !-)
>
> I have reviewed everything that has been written on the ODB import (aka
> Canada Building Import) in Talk-ca and the wiki. I proposed changes to some
> wiki pages (via talk tabs) to ease the discussions about this import and
> the following. Now, in order to restart the import, here are some thoughts
> and a proposal on how to proceed to complete the task.
>
>
>
> *1. Issues with the ODB Data Import*
>
> Many concerns were raised about the import. One major concern was to
> obtain local communities’ buy-in in the Canadian context. Another concern
> was to improve the quality of the data prior the import. The following
> paragraphs intend to clear most of these concerns.
>
> *1.1. Which data import project?*
>
> According to the import guidelines (steps 3 & 4), a data import explicitly
> refers to a single data source (ODB in our case). Discussions about the
> availability and quality of Microsoft or ESRI data, while interesting, are
> not relevant as they should be dealt with as other import projects.
>
> *1.2. What has been imported so far?*
>
> According to what I found [1], the ODB import is completed for 21
> municipalities. These imports seem to have kept OSM content’s history, at
> least for the samples checked, but many problems were found. In some case,
> the imports brought swimming pools in OSM because they were included in the
> dataset (e.g. Moncton). In other cases, importing buildings with accurate
> locations (XY) over content mapped from less accurate imagery resulted in
> buildings that now overlap the street network (e.g. Squamish). It means
> that all these 21 imports need to be carefully re-examined and corrected as
> required.
>
> For 12 other municipalities, the import is partial, either suspended as
> requested, or because previous imports had already provided most of the
> buildings (often from the same municipal provider). That said the import
> will definitely improve OSM accuracy and completeness if done properly.
>
> *2. How should ODB Data be imported?*
>
> I will copy the following paragraphs in the “Canada Building Import” wiki
> page [3] for a detailed discussion…
>
> Since the data (ODB, OSM and imagery) differ from one municipality to
> another, there can be no imports at the national or provincial level. We
> have to work on a municipal basis and make sure to identify all the
> problems and the corrective measures to apply when dealing with issues like
> those I identified [1].
>
> *2.1 Importing Locally*
>
> According to the import guidelines (step 2), we must not import the data
> without local buy-in. However, and contrarily to some European country,
> there is usually no such thing as a local OSM community in each
> municipality. However, we may find a few local mappers from time to time.
> Working on a municipal basis should allow identifying these local mappers
> before doing the import. I often use this tool [2] to identify and contact
> local mappers. Once identified, I suggest that…
>
> - We contact them to explain our intents by referring to appropriate wiki
> pages.
>
> - We wait a week or two to let them respond nothing, that they have
> concerns, or wish to help.
>
> - Without negative answers we could proceed to the import.
>
> I first suggest that when a contributor wishes to import ODB for a given
> municipality, he first identifies himself as responsible for the import (we
> need to create an entry for each municipality somewhere in the wiki). He
> can then contact local mappers, as explain above, and go ahead with the
> import once everything settled. For those who already made the import, I
> suggest that they review their work since many issues were detected with
> some of these imports.
>
> Since there are only a few local OSM communities in Canada, and because
> Canada is large, I suggest not limiting the import of a given municipality
> to the people of the concerned province or region.
>
> *2.2 Pre-processing*
>
> Once local mappers have agreed, some pre-processing can be done if
> required.
>
> A few months ago, I developed a tool that could be used to process the
> data [4]. Concerns were raised because the application was developed using
> proprietary software. So I documented the whole process and algorithms in
> order to see courageous coders converting it in open source software. In
> the meantime, and as long as I have access to an FME licence, I could
> process the data, when 

Re: [Talk-ca] Importing buildings in Canada

[Daniel @jfd553]

Bonjour groupe, mes excuses pour ce très long courriel !-)
I have reviewed everything that has been written on the ODB import (aka Canada 
Building Import) in Talk-ca and the wiki. I proposed changes to some wiki pages 
(via talk tabs) to ease the discussions about this import and the following. 
Now, in order to restart the import, here are some thoughts and a proposal on 
how to proceed to complete the task.

1. Issues with the ODB Data Import
Many concerns were raised about the import. One major concern was to obtain 
local communities' buy-in in the Canadian context. Another concern was to 
improve the quality of the data prior the import. The following paragraphs 
intend to clear most of these concerns.
1.1. Which data import project?
According to the import guidelines (steps 3 & 4), a data import explicitly 
refers to a single data source (ODB in our case). Discussions about the 
availability and quality of Microsoft or ESRI data, while interesting, are not 
relevant as they should be dealt with as other import projects.
1.2. What has been imported so far?
According to what I found [1], the ODB import is completed for 21 
municipalities. These imports seem to have kept OSM content's history, at least 
for the samples checked, but many problems were found. In some case, the 
imports brought swimming pools in OSM because they were included in the dataset 
(e.g. Moncton). In other cases, importing buildings with accurate locations 
(XY) over content mapped from less accurate imagery resulted in buildings that 
now overlap the street network (e.g. Squamish). It means that all these 21 
imports need to be carefully re-examined and corrected as required.
For 12 other municipalities, the import is partial, either suspended as 
requested, or because previous imports had already provided most of the 
buildings (often from the same municipal provider). That said the import will 
definitely improve OSM accuracy and completeness if done properly.
2. How should ODB Data be imported?
I will copy the following paragraphs in the "Canada Building Import" wiki page 
[3] for a detailed discussion...
Since the data (ODB, OSM and imagery) differ from one municipality to another, 
there can be no imports at the national or provincial level. We have to work on 
a municipal basis and make sure to identify all the problems and the corrective 
measures to apply when dealing with issues like those I identified [1].
2.1 Importing Locally
According to the import guidelines (step 2), we must not import the data 
without local buy-in. However, and contrarily to some European country, there 
is usually no such thing as a local OSM community in each municipality. 
However, we may find a few local mappers from time to time. Working on a 
municipal basis should allow identifying these local mappers before doing the 
import. I often use this tool [2] to identify and contact local mappers. Once 
identified, I suggest that...
- We contact them to explain our intents by referring to appropriate wiki pages.
- We wait a week or two to let them respond nothing, that they have concerns, 
or wish to help.
- Without negative answers we could proceed to the import.
I first suggest that when a contributor wishes to import ODB for a given 
municipality, he first identifies himself as responsible for the import (we 
need to create an entry for each municipality somewhere in the wiki). He can 
then contact local mappers, as explain above, and go ahead with the import once 
everything settled. For those who already made the import, I suggest that they 
review their work since many issues were detected with some of these imports.
Since there are only a few local OSM communities in Canada, and because Canada 
is large, I suggest not limiting the import of a given municipality to the 
people of the concerned province or region.
2.2 Pre-processing
Once local mappers have agreed, some pre-processing can be done if required.
A few months ago, I developed a tool that could be used to process the data 
[4]. Concerns were raised because the application was developed using 
proprietary software. So I documented the whole process and algorithms in order 
to see courageous coders converting it in open source software. In the 
meantime, and as long as I have access to an FME licence, I could process the 
data, when necessary, prior to make it available through the task manager.
Proposed pre-processing [4] includes:
- Reading of original ODB data,
- Removal of near collinear nodes (simplification),
- Orthogonalization of buildings (for corners having near right angles),
- Tagging of building footprints,
- Providing files in OSM format.
Proposed tagging: In addition to the tags produced by the orthogonalization 
process [4] and the source tag 
(source=Statistics Canada - 
Open Building Database), the name of the Census Subdivision provided in ODB 
data [5] is used to add the addr:city tag to each building.
The