Re: [OSM-talk] Duplicate Buildings

[Pierre Béland via talk]

My workflow in the last few days was to select/process series of way id's from 
Frederik list with closed iterations that look to come from the same changeset. 

Using the JOSM Overpass query function with instructions I provided earlier, it 
is easy to download such a block of data, validate and correct with the help of 
the validation functionality. 

But when there is both quadri-duplicates combined with Bad quality problem 
illustrated from the overpass query below, I leave it to Quality teams involved 
in the project (here #hotosm-project-11827) to document and correct. 

See http://overpass-turbo.eu/s/1sQQ

Pierre



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Marc_marc]

Hello,

Le 17.03.23 à 13:36, Harrison Scofield a écrit :

you can be overwhelmed by the shear number of "errors"


it's true, i think however the validator will only detect a douublon
only if you have touched or created a duplicate, so it seems to me 
necessary to delete this duplicate before uploading. if you have so 
many, maybe it's useful to make smaller changeset or to make from time 
to time a simulation of upload to detect the errors during your mapping


the validator proposes to correct some of them in one click
I have opened a ticket for the cases that are not correctable in one 
click at the moment

https://josm.openstreetmap.de/ticket/22820


the duplicate Ways were "glued" together which complicated detecting and 
correcting these errors.


if you open the issue "duplicate way" in the validator pannel, it allow 
to select issue one by one (2 ways)

clic on the oldest way and you have it, despite it's glued :)



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Harrison Scofield]

I sure hope I am replying correctly to your reply!  To clarify two
points:  Yes, the JOSM "validator" detected the duplicate Ways, both
pipelines, highways and buildings which is how I discovered the issue.
 Unfortunately, when you "upload", you can be overwhelmed by the shear
number of "errors".  And, a weird thing I noticed was the duplicate
Ways were "glued" together which complicated detecting and correcting
these errors.  Are the duplicate Ways you have discovered "glued"
together because many "mappers" probably wouldn't try "unglueing" when
trying to identifying/correcting such errors?  Good luck trying to
identify the cause of your "duplicates", Harrison S
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Mateusz Konieczny via talk]

Mar 16, 2023, 14:54 by sainokawara.sisyp...@gmail.com:

> By the way, it was extremely difficult to detect & delete these "extra" Ways 
> manually using JOSM.
>
Have you tried https://josm.openstreetmap.de/wiki/Help/Dialog/Validator ?

Have you disabled validator warnings about duplicated ways?
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Marc_marc]

Le 16.03.23 à 14:51, Harrison Scofield a écrit :
it was extremely difficult to detect & delete these "extra" Ways 
manually using JOSM


https://www.openstreetmap.org/#map=19/5.85176/6.82890
clic edit with josm
in josm, open the validator pannel and run it
validator output several "duplicate way"



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Harrison Scofield]

I don't know the "cause/effect" but several months ago, I noticed that
Ways were being "duplicated" multiple times when I used JOSM and they
weren't buildings!  I could not determine whether it was at fault or
when it was only for specific types of ways but I did "report" a
potential problem to JOSM support because I had never encountered such
a result using JOSM previously.  You can probably guess what happened
to my "trouble report" so I won't go there.  By the way, it was
extremely difficult to detect & delete these "extra" Ways manually
using JOSM.  I did the best job I could but do not know how successful
I was in finding & getting rid of these "extra" Ways that got created.

Thank you, Harrison S
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Pierre Béland via talk]

From the OSM-id list of Frederik, it is possible to query with Overpass. For 
the following list:

id1,id2,id3,id4
-15538065,-15538064,-15538063,1137657546

I isolate negative values and query as relation, others as way.

[out:xml][date:'2023-03-05T00:00:00Z'];
(
  way(id: 1137657546  
 );
  relation(id: 
15538065, 15538064, 15538063
 );
);
out meta;
>; out meta;

Pierre

Envoyé avec la messagerie sécurisée Proton Mail.

--- Original Message ---
Le mercredi 15 mars 2023 à 07:11, Marc_marc  a écrit :

 
> 
> in fact I was thinking about the overpass turbo query :)
> to test it in my area
> 
> > the 4 dates of edit.
> 
> 

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Marc_marc]

Le 14.03.23 à 21:35, Pierre Béland a écrit :

Hi Marc,

I created a csv file available on wetranser for 7 days. 


in fact I was thinking about the overpass turbo query :)
to test it in my area


the 4 dates of edit.


a fast lookup show a lot of similar timestamp
but also some very old object... a pitty that nobody detect
this issue (some timestamp in 2013)





___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Pierre Béland via talk]

Hi Marc,

I created a csv file available on wetranser for 7 days. It contains 
901 quadruplets. For each line, the 4 OSM id's (similar to Frederik list)  + 
the 4 dates of edit.

See https://we.tl/t-rt9KSsZXBD

The osm-id list below refers to 77 buildings (ways) with level, layer or 
building:part attributes:

310361827, 310362676, 310362677, 310362678, 618340953, 618340954, 618340955, 
618340956, 618489859, 618489861, 618489863, 618489865, 618763855, 618763856, 
618763857, 618763858, 626660536, 626660537, 626660538, 626660539, 626660540, 
626660541, 921548773, 921548775, 921550589, 921550590, 921550591, 921550592, 
921550593, 921550594, 921550595, 921550596, 921550597, 921550598, 951883751, 
951883903, 951884118, 951884269, 951884402, 951884775, 951885143, 951885226, 
951885296, 951885425, 951885509, 951885788, 969467331, 1011631197, 1011631208, 
1011631209, 1078444684, 1078444685, 1078444686, 1078444687, 1078444688, 
1078444706, 1078444707, 1078444708, 1078444709, 1078444710, 1078444711, 
1078444712, 1078444713, 1078444714, 1078444715, 1078941994, 1078941995, 
1078941996, 1078941997, 1078941999, 1078942000, 1078942001, 1078942002, 
1078942003, 1078942004, 1078942005, 1078942006


Pierre


--- Original Message ---
Le mardi 14 mars 2023 à 11:51, Marc_marc  a écrit :


> Le 14.03.23 à 13:45, Pierre Béland via talk a écrit :
> 
> > I imported the osm metadata using overpass
> 
> 
> can youu share it to avoid having to make it again :)
> 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Pierre Béland via talk]

Marc, I will revise and share data. 

First, below are the instructions to extract relations metadata with 

[out:csv(::changeset, ::timestamp,::user, ::type, ::id, building, highway, 
level, layer, 'building:part', 'building:roof')];
relation(id: 
3240506, 3240507, 3240508, 3240509, 3240510, 3240511, 3240512, 3240513, 
3240514, 3240515, 3240516, 3240517, 8325855, 8325856, 8325857, 8325858, 
8853592, 8898812, 8898813, 8898814, 9168214, 9168215, 9168216, 9168217, 
9168218, 9168219, 9173897, 9173898, 9173899, 9173900, 9173901, 9173902, 
9173903, 9173904, 9173905, 9175052, 9175053, 9175054, 9175055, 9796369, 
9796370, 9796371, 9796372, 10326411, 10326412, 10326413, 10326414, 10784846, 
10784847, 10784848, 10784849, 11305092, 11305093, 11305094, 11305095, 13965409, 
13965410, 13965411, 13965412, 14627860, 14627861, 14627862, 14627863, 15538063, 
15538064, 15538065 
   ); out meta;
Overpass. We can spot other relations with the level attribute.


Pierre

--- Original Message ---
Le mardi 14 mars 2023 à 11:51, Marc_marc  a écrit :


> Le 14.03.23 à 13:45, Pierre Béland via talk a écrit :
> 
> > I imported the osm metadata using overpass
> 
> 
> can youu share it to avoid having to make it again :)
> 
> > Simple Building duplicates from the same contributor are the easy ones to 
> > correct.
> 
> 
> yes, on the condition that you can easily detect when the contributor
> wanted to represent a simple building ?
> I imagine that a first criterion could be the absence of the level layer
> tag (which we see badly used in one of your examples) and perhaps
> building:level
> 
> > osm type nb of quaddup objects
> > way 1 838 3352
> 
> 
> maybe best to focus on that first
> 
> > See https://www.openstreetmap.org/relation/13965412 that contains 
> > way/677238859 with inner role to 6 relations that represent each level of 
> > the building.
> 
> 
> commented https://www.openstreetmap.org/changeset/118924402
> 
> > one building is represented with one way (6 nodes) and 3 relations in which 
> > this way has role=outer.
> > way 1137657546 building=cabin
> > Relation : 15538065 building=yes
> > Relation : 15538064 building=yes
> > Relation : Horsnæs Fangststation (15538063) place=locality
> 
> 
> ccommented https://www.openstreetmap.org/changeset/133075628
> 
> > These are four different relations.
> > -10326414 -10326413 -10326412 -10326411
> > They all share these 2 buildings as outer members.
> > 48002128 505561207
> 
> 
> what's strange is to have managed to do this with iD
> but that's three mistakes :
> - duplicate
> - the garage is not a house
> - the address in Switzerland represents an entrance door and by
> extention the building if this one has only one address. if the no. 7
> represents the house, the garage does have the no. 7 (but probably the
> number 7.1 which is rarely tagged in osm)
> 
> Regards,
> Marc
> 
> 
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Marc_marc]

Le 14.03.23 à 13:45, Pierre Béland via talk a écrit :

I imported the osm metadata using overpass


can youu share it to avoid having to make it again :)


Simple Building duplicates from the same contributor are the easy ones to 
correct.


yes, on the condition that you can easily detect when the contributor 
wanted to represent a simple building ?
I imagine that a first criterion could be the absence of the level layer 
tag (which we see badly used in one of your examples) and perhaps 
building:level



osm typenb ofquaddupobjects
way 1   838  3352


maybe best to focus on that first


See https://www.openstreetmap.org/relation/13965412 that contains way/677238859 
with inner role to 6 relations that represent each level of the building.


commented https://www.openstreetmap.org/changeset/118924402


one building is represented with one way (6 nodes) and 3 relations in which 
this way has role=outer.
   way 1137657546   building=cabin
   Relation : 15538065  building=yes
 Relation : 15538064building=yes
 Relation : Horsnæs Fangststation (15538063) place=locality


ccommented https://www.openstreetmap.org/changeset/133075628


These are four different relations.
-10326414   -10326413   -10326412   -10326411
They all share these  2 buildings  as outer members.
48002128505561207


what's strange is to have managed to do this with iD
but that's three mistakes :
- duplicate
- the garage is not a house
- the address in Switzerland represents an entrance door and by 
extention the building if this one has only one address. if the no. 7 
represents the house, the garage does have the no. 7 (but probably the 
number 7.1 which is rarely tagged in osm)


Regards,
Marc



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Pierre Béland via talk]

Using id's from the Frederick quadruplicate list, I imported the osm metadata 
using overpass. Note thate the negative values in the list represent relations.

The table below shows that the majority of quadruplicate cases implicate only 
one contributor. Simple Building duplicates from the same contributor are the 
easy ones to correct.  It is different for the relations or combination of 
relations/ways where it is sometime necessary to revise the tagging as a 
contributor misused duplicates to represent various levels of a building.

osm typenb ofquaddupobjects
contributors  cases
relation168   272
relation2 936
way 1   838  3352
way 2   134   536
way 3 520

See https://www.openstreetmap.org/relation/13965412 that contains way/677238859 
with inner role to 6 relations that represent each level of the building.

There are also strange schemas. For this block, one building is represented 
with one way (6 nodes) and 3 relations in which this way has role=outer.
  way 1137657546building=cabin
  Relation : 15538065   building=yes
Relation : 15538064 building=yes
Relation : Horsnæs Fangststation (15538063) place=locality

These are four different relations.
-10326414   -10326413   -10326412   -10326411
They all share these  2 buildings  as outer members.
48002128505561207


Pierre


--- Original Message ---
Le samedi 11 mars 2023 à 07:35, Frederik Ramm  a écrit :


> Hi,
> 
> I think an automatic fix of the problem is possible, however it would be
> a good idea to try and find out what the root cause of the problem is -
> bad software, bad imports, bad instructions?
> 
> To get an idea of how big the issue is, I did this on a standard
> rendering database:
> 
> create table buildings as (select way,osm_id from planet_osm_polygon
> where building is not null)
> 
> select a.osm_id, b.osm_id into duplicates from buildings a, buildings b
> where a.osm_id < b.osm_id and a.way ~= b.way and st_equals(a.way,b.way);
> 
> This took a few days - probably it could have been done more efficiently
> - and resulted in a list of about 70k buldings world-wide that are exact
> duplicates (geoetry-wise) of other buildings. The list is here:
> 
> http://www.remote.org/frederik/tmp/duplicatebuildings.csv
> 
> Some buildings are in OSM three or four times (contained i nthe above in
> the form of "a is duplicate of b, b is duplicate of c") but I've
> extracted them in extra files:
> http://www.remote.org/frederik/tmp/triplcatebuildings.csv and
> http://www.remote.org/frederik/tmp/quadruplicatebuildings.csv)
> 
> I don't have the time to analyse the situation in more detail at present
> so if anyone wants to take the above lists as a basis for deeper analysis...
> 
> Cheers
> Frederik


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Marc_marc]

Le 11.03.23 à 13:35, Frederik Ramm a écrit :
it would be a good idea to try and find out what the root cause of the 
problem is - bad software, bad imports, bad instructions?


I have already encountered 3 cases:
- as Warin says: create something in josm, save it as faile, send to 
osm, don't save after send, close and reopen the file, send to osm
- imports (cadastre) done (in France in a "legal" way, in Switzerland, 
Fribourg done with in an undocumented/incompatible source, repetitive 
issue with the same guy again)

- rare problems during a network outage while sending.

The only thing that remains incomprehensible is when this happens iD 
several days apart
example with the first one on your list: 
https://www.openstreetmap.org/way/1122282039 sent with iD 3 months ago. 
duplicate of a building sent 5 years ago with josm 
https://www.openstreetmap.org/way/553715102
the HOT validator didn't see anything, neither did the task validators, 
but that's another problem.


I'm willing to look at some examples of the quadruple list but in this 
format, it will be difficult to work on it with several people as it is 
impossible to mark elements as processed



Translated with www.DeepL.com/Translator (free version)



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[john whelan]

Many thanks for putting some numbers on this.

Warin's comment would suggest it may also be more than just buildings that
are involved.

For buildings the total number as a percentage is small unfortunately they
tend to cluster so are more of a problem than if they were more spread out.

John

On Sat, Mar 11, 2023, 07:40 Frederik Ramm  wrote:

> Hi,
>
> I think an automatic fix of the problem is possible, however it would be
> a good idea to try and find out what the root cause of the problem is -
> bad software, bad imports, bad instructions?
>
> To get an idea of how big the issue is, I did this on a standard
> rendering database:
>
> create table buildings as (select way,osm_id from planet_osm_polygon
> where building is not null)
>
> select a.osm_id, b.osm_id into duplicates from buildings a, buildings b
> where a.osm_id < b.osm_id and a.way ~= b.way and st_equals(a.way,b.way);
>
> This took a few days - probably it could have been done more efficiently
> - and resulted in a list of about 70k buldings world-wide that are exact
> duplicates (geoetry-wise) of other buildings. The list is here:
>
> http://www.remote.org/frederik/tmp/duplicatebuildings.csv
>
> Some buildings are in OSM three or four times (contained i nthe above in
> the form of "a is duplicate of b, b is duplicate of c") but I've
> extracted them in extra files:
> http://www.remote.org/frederik/tmp/triplcatebuildings.csv and
> http://www.remote.org/frederik/tmp/quadruplicatebuildings.csv)
>
> I don't have the time to analyse the situation in more detail at present
> so if anyone wants to take the above lists as a basis for deeper
> analysis...
>
> Cheers
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Warin]

On 11/3/23 23:35, Frederik Ramm wrote:

Hi,

I think an automatic fix of the problem is possible, however it would 
be a good idea to try and find out what the root cause of the problem 
is - bad software, bad imports, bad instructions?



I have duplicated an entry the following way in JOSM;

save the data locally

download the data to OSM

exit from JOSM .. without a save locally after the upload.


Later, open the saved file in JOSM, possibly make further 
changes/addition and upload the date to OSM...


--

The locally saved file data indicates that the data has not been 
uploaded to OSM (even though it has) .. thus the second upload will 
duplicates all the changes made in the file..



I did catch this some time ago (years?) and have been careful since. But 
it is a way of doing it..


I think JOSM now has a warning for this?



___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Frederik Ramm]

Hi,

I think an automatic fix of the problem is possible, however it would be 
a good idea to try and find out what the root cause of the problem is - 
bad software, bad imports, bad instructions?


To get an idea of how big the issue is, I did this on a standard 
rendering database:


create table buildings as (select way,osm_id from planet_osm_polygon 
where building is not null)


select a.osm_id, b.osm_id into duplicates from buildings a, buildings b 
where a.osm_id < b.osm_id and a.way ~= b.way and st_equals(a.way,b.way);


This took a few days - probably it could have been done more efficiently 
- and resulted in a list of about 70k buldings world-wide that are exact 
duplicates (geoetry-wise) of other buildings. The list is here:


http://www.remote.org/frederik/tmp/duplicatebuildings.csv

Some buildings are in OSM three or four times (contained i nthe above in 
the form of "a is duplicate of b, b is duplicate of c") but I've 
extracted them in extra files: 
http://www.remote.org/frederik/tmp/triplcatebuildings.csv and 
http://www.remote.org/frederik/tmp/quadruplicatebuildings.csv)


I don't have the time to analyse the situation in more detail at present 
so if anyone wants to take the above lists as a basis for deeper analysis...


Cheers
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[john whelan]

There are many and I have tools that can pick them out in a particular
area.  However to use them I inspect the building visually before deleting
one or the other.  Sometimes if the mapper has less than ten edits to their
name I may even skip adding a change set comment first.

However I do see many cases where the building or way appears to have been
uploaded twice and these I think could be handled by a bot which would save
my time and zap a substantial number.  I don't think anyone could object to
such an automated course of action whereas those that overlap really need
visual inspection to see which should be deleted.

Cheerio John

On Sun, Mar 5, 2023, 15:53 Isaac Boates  wrote:

> It might be better to check for building features which have a very large
> percentage of their area overlapping with another, rather than an exact
> duplicate, just in case there are a few kicking around that are identical
> except for one vertex
>
> Isaac
>
> On Sun, Mar 5, 2023 at 7:05 PM John Whelan  wrote:
>
>> Occasionally I come across a building that has been mapped twice.   A
>> number of them look as if they have been uploaded twice.
>>
>> Could some nice bot expert build one to run through the map and delete
>> any ways that are an exact duplicate even as far as the tags are concerned?
>>
>> The problem comes when doing population counts etc that rely on the
>> number of buildings in a given area.  A duplicated building means the wrong
>> number of vaccines etc get delivered to the area besides cluttering up the
>> map.
>>
>> Thanks John
>> --
>> Sent from Postbox 
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Duplicate Buildings

[Isaac Boates]

It might be better to check for building features which have a very large
percentage of their area overlapping with another, rather than an exact
duplicate, just in case there are a few kicking around that are identical
except for one vertex

Isaac

On Sun, Mar 5, 2023 at 7:05 PM John Whelan  wrote:

> Occasionally I come across a building that has been mapped twice.   A
> number of them look as if they have been uploaded twice.
>
> Could some nice bot expert build one to run through the map and delete any
> ways that are an exact duplicate even as far as the tags are concerned?
>
> The problem comes when doing population counts etc that rely on the number
> of buildings in a given area.  A duplicated building means the wrong number
> of vaccines etc get delivered to the area besides cluttering up the map.
>
> Thanks John
> --
> Sent from Postbox 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Duplicate Buildings

[John Whelan]

Occasionally I come across a building that has been mapped twice.   A 
number of them look as if they have been uploaded twice.


Could some nice bot expert build one to run through the map and delete 
any ways that are an exact duplicate even as far as the tags are concerned?


The problem comes when doing population counts etc that rely on the 
number of buildings in a given area.  A duplicated building means the 
wrong number of vaccines etc get delivered to the area besides 
cluttering up the map.


Thanks John
--
Sent from Postbox 
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk