Re: [OSM-talk] OSM tagging validation lib

[Simon Poole]

On 26.12.2017 14:10, Colin Smale wrote:
>
> Why bother anyway? Why not just leave it to FvGordon? 90k changesets
> fixing other people's tagging errors...
>
What wasn't clear about ".. preventing inadvertent mistakes to start
with ..". After the fact validation just doesn't really work IMHO (as
lots of the errors FvGordon is fixing show).

Simon

> http://www.openstreetmap.org/user/FvGordon/history
>
>  
>
> Actually, an analysis of all these changesets might produce some
> interesting insights into "frequently made errors".
>
> //colin
>
> On 2017-12-26 13:24, Simon Poole wrote:
>
>> We already have validation tools in many forms for post-editing
>> validation and already know how difficult they are to get "right" (see
>> OSMOSE complaining about P+R facilities for a trivial example). I don't
>> quite see why there is a need to create yet another one/system.
>>
>> On the other hand validation on upload is horrible from a UI and
>> workflow point of view (which is why JOSMs warnings get ignored so
>> often). IMHO preventing inadvertent mistakes to start with seems to be
>> far better and that can be done with consistent presets (which might
>> have to be localized) and editor support (see what Bryan has just done
>> with creating area geometries in iD).
>>
>> Simon
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org 
>> https://lists.openstreetmap.org/listinfo/talk
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Colin Smale]

Why bother anyway? Why not just leave it to FvGordon? 90k changesets
fixing other people's tagging errors... 

http://www.openstreetmap.org/user/FvGordon/history

Actually, an analysis of all these changesets might produce some
interesting insights into "frequently made errors". 

//colin 

On 2017-12-26 13:24, Simon Poole wrote:

> We already have validation tools in many forms for post-editing
> validation and already know how difficult they are to get "right" (see
> OSMOSE complaining about P+R facilities for a trivial example). I don't
> quite see why there is a need to create yet another one/system.
> 
> On the other hand validation on upload is horrible from a UI and
> workflow point of view (which is why JOSMs warnings get ignored so
> often). IMHO preventing inadvertent mistakes to start with seems to be
> far better and that can be done with consistent presets (which might
> have to be localized) and editor support (see what Bryan has just done
> with creating area geometries in iD).
> 
> Simon
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Simon Poole]

We already have validation tools in many forms for post-editing
validation and already know how difficult they are to get "right" (see
OSMOSE complaining about P+R facilities for a trivial example). I don't
quite see why there is a need to create yet another one/system.

On the other hand validation on upload is horrible from a UI and
workflow point of view (which is why JOSMs warnings get ignored so
often). IMHO preventing inadvertent mistakes to start with seems to be
far better and that can be done with consistent presets (which might
have to be localized) and editor support (see what Bryan has just done
with creating area geometries in iD).

Simon

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Frederik Ramm]

Hi,

On 24.12.2017 21:03, Colin Smale wrote:
> The real challenge here is not for the coders, but a perennial challenge
> for the OSM community. How do we get to such a consensus about tagging
> patterns, that we can actually say "this is correct" and "this is wrong
> enough to warrant correction" without upsetting a large number of
> people? 

Even upsetting a *small* number of people could be considered a problem.
If you look at this from the much-touted diversity perspective, why
should German mappers who are a majority by numbers, dictate to a small
group of mappers in Peru which tags they should be using? Why should men
who are a majority by numbers, decide which tags women should be using? Etc.

Some decisions are necessary for the project to function, and hence
taking them is important, even if the mechanisms by which they are taken
may be flawed, and suppress minorities.

But we should be careful about applying our necessarily flawed decision
making mechanisms to *too many* aspects of mapping; every time you
narrow down the envelope of "acceptable" tagging, you could be using
your superior numbers power to put someone somewhere at a disadvantage
or deprive them of a voice.

People sometimes say that every decision not taken contributes to a
vacuum that decreases the quality of mapping; but it is equally valid to
say that every decision not taken contributes to the freedom to map what
you want, how you want it.

As always, the truth is somewhere in between.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Yuri Astrakhan]

Colin, I think we should look at JOSM's validations as a good example of
what is needed. For some validations, it extended MapCSS with its own
custom quirks, and even added its own unit testing. For other types of
validations (e.g. geometries), it seems to fall back to full code.  Having
a single language to express any types of validation would simplify
maintenance and participation.

I don't think people should translate the JS reference implementation at
all. That recreates the problem we have now. Instead, all tools should use
one common library directly. The only problem is integration - Java,
Python, and C++ can all call JS functions, we just need to ensure it is
*reasonably* fast and easy to deploy.

Lastly, I am sure we will want to introduce "slow validations" - when a
call to an external service, e.g. OSM db or taginfo is required. Those may
be used by some of the tools, e.g. editors right before saving.

Thx!

On Sun, Dec 24, 2017 at 3:03 PM, Colin Smale  wrote:

> Hi Yuri,
>
> We have to decide what is most important - fidelity to the infinite number
> of tagging styles out there, or the ability to get a basic set of tagging
> grammar accepted in as many tools as possible.
>
> Any rules grammar will always have limits of course. If the rules are too
> complex to represent in a declarative way, that is in itself an indication
> of the mess we have got ourselves into. If the "unwritten rules" of OSM
> tagging are too complex to write down, then they need sorting out first!
> Having a simple base to work from might be a good first step. Automated
> chaos is still chaos.
>
> I agree that the ultimate rules engine may well end up using e.g. JS as a
> medium to express some of the subtleties of the rules. Once a JS
> implementation has been published, then people can translate the JS
> reference implementation into whatever language they need. But separating
> the basic rules from the execution engine is nothing more than
> architectural best practice, and there is no reason that the basic
> rules should not be portable across runtime environments.
>
> Can you think of any complex patterns which cannot (easily) be expressed
> in a declarative way?
>
> The real challenge here is not for the coders, but a perennial challenge
> for the OSM community. How do we get to such a consensus about tagging
> patterns, that we can actually say "this is correct" and "this is wrong
> enough to warrant correction" without upsetting a large number of people?
> As soon as a discussion is about right vs wrong, it degenerates into
> mudslinging.
>
>
> //colin
>
> On 2017-12-24 20:37, Yuri Astrakhan wrote:
>
> Declarative rules are usually not very good. Every tool must understand
> every type of rule, and must be updated when new rule types are introduced.
> Plus declarative grammar is either too limiting, or eventually starts
> looking like a scripting language itself, and we end up building an
> execution environment in every tool.
>
>
> I think a better path is executing scripts inside other languages, e.g.
>
> * a JavaScript library ran by Java, Python, C++, ...
> * a lib that gets compiled into a webassembly for browser, or connected to
> other languages via native bindings (less tried path)
>
> The lib would need an API to
> * access local data state
> * access master OSM DB for additional data
> * access other tools like taginfo
>
>
> Integrating scripting environment may be difficult, but offers far greater
> benefits of rule consistency and flexibility.
>
>
> On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale 
> wrote:
>
>> The technical differences between java and JS do not preclude generic
>> thinking. Consider tzdata[1] for example, which does something analogous
>> for time zone data.
>>
>> The "rules database" can be made portable, in the form of XML or JSON for
>> example. The logic for using these rules can be described in a portable
>> way. Then you add a set of compliance tests, and publish a reference
>> implementation to demonstrate that is is possible to implement it. After
>> that, the logic can be implemented in any language you like, checked
>> against the compliance tests and the bindings published.
>>
>> Externalising the rules database enables updates and customisations for
>> particular reasons. Depending on the specific use case and the associated
>> non-functionals, validation could possibly be offered as a cloud service
>> (not necessarily by OSM).
>>
>>
>> //colin
>>
>> [1] https://en.wikipedia.org/wiki/Tz_database
>>
>> On 2017-12-24 12:18, James wrote:
>>
>> ID is javascript, JOSM is java. So right there I already see a
>> intercompatibility issue
>>
>> On Dec 24, 2017 6:12 AM, "François Lacombe" 
>> wrote:
>>
>>> Hi
>>>
>>> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
>>> others).
>>> Subsequently to wiki proposal voting and cleanups, it's currently
>>> necessarily to open issues in each 

Re: [OSM-talk] OSM tagging validation lib

[Colin Smale]

Hi Yuri, 

We have to decide what is most important - fidelity to the infinite
number of tagging styles out there, or the ability to get a basic set of
tagging grammar accepted in as many tools as possible. 

Any rules grammar will always have limits of course. If the rules are
too complex to represent in a declarative way, that is in itself an
indication of the mess we have got ourselves into. If the "unwritten
rules" of OSM tagging are too complex to write down, then they need
sorting out first! Having a simple base to work from might be a good
first step. Automated chaos is still chaos. 

I agree that the ultimate rules engine may well end up using e.g. JS as
a medium to express some of the subtleties of the rules. Once a JS
implementation has been published, then people can translate the JS
reference implementation into whatever language they need. But
separating the basic rules from the execution engine is nothing more
than architectural best practice, and there is no reason that the basic
rules should not be portable across runtime environments. 

Can you think of any complex patterns which cannot (easily) be expressed
in a declarative way? 

The real challenge here is not for the coders, but a perennial challenge
for the OSM community. How do we get to such a consensus about tagging
patterns, that we can actually say "this is correct" and "this is wrong
enough to warrant correction" without upsetting a large number of
people? As soon as a discussion is about right vs wrong, it degenerates
into mudslinging.

//colin 

On 2017-12-24 20:37, Yuri Astrakhan wrote:

> Declarative rules are usually not very good. Every tool must understand every 
> type of rule, and must be updated when new rule types are introduced. Plus 
> declarative grammar is either too limiting, or eventually starts looking like 
> a scripting language itself, and we end up building an execution environment 
> in every tool. 
> 
> I think a better path is executing scripts inside other languages, e.g.
> 
> * a JavaScript library ran by Java, Python, C++, ... 
> * a lib that gets compiled into a webassembly for browser, or connected to 
> other languages via native bindings (less tried path) 
> The lib would need an API to * access local data state * access master OSM DB 
> for additional data * access other tools like taginfo
> 
> Integrating scripting environment may be difficult, but offers far greater 
> benefits of rule consistency and flexibility. 
> 
> On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale  wrote:
> 
> The technical differences between java and JS do not preclude generic 
> thinking. Consider tzdata[1] for example, which does something analogous for 
> time zone data. 
> 
> The "rules database" can be made portable, in the form of XML or JSON for 
> example. The logic for using these rules can be described in a portable way. 
> Then you add a set of compliance tests, and publish a reference 
> implementation to demonstrate that is is possible to implement it. After 
> that, the logic can be implemented in any language you like, checked against 
> the compliance tests and the bindings published. 
> 
> Externalising the rules database enables updates and customisations for 
> particular reasons. Depending on the specific use case and the associated 
> non-functionals, validation could possibly be offered as a cloud service (not 
> necessarily by OSM).
> 
> //colin 
> 
> [1] https://en.wikipedia.org/wiki/Tz_database [1]
> 
> On 2017-12-24 12:18, James wrote: 
> ID is javascript, JOSM is java. So right there I already see a 
> intercompatibility issue 
> 
> On Dec 24, 2017 6:12 AM, "François Lacombe"  wrote:
> 
> Hi 
> 
> Here is an idea I got regarding tagging validation in editors (iD, JOSM, 
> others). 
> Subsequently to wiki proposal voting and cleanups, it's currently necessarily 
> to open issues in each editor repository to ask for new tagging validation 
> rules.  
> 
> It can sometimes be time consuming to develop those new rules and such a work 
> is done independently by each project maintainer. While each project have its 
> own specific components, background logic is the same. 
> 
> Would a new lib called like osmtagvalidator or so in charge of doing conform 
> validation to wiki be useful? 
> It may be shared by any project involved in osm editing and preserve its 
> resources for other valuable developments. 
> 
> For me, validation doesn't prevent users to use tags they want, but only warn 
> them about possible mistakes. 
> 
> How would devs and users feel about this? 
> 
> All the best 
> 
> François 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk [2] 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk [2]

___
talk mailing list

Re: [OSM-talk] OSM tagging validation lib

[Yuri Astrakhan]

Declarative rules are usually not very good. Every tool must understand
every type of rule, and must be updated when new rule types are introduced.
Plus declarative grammar is either too limiting, or eventually starts
looking like a scripting language itself, and we end up building an
execution environment in every tool.

I think a better path is executing scripts inside other languages, e.g.
* a JavaScript library ran by Java, Python, C++, ...
* a lib that gets compiled into a webassembly for browser, or connected to
other languages via native bindings (less tried path)

The lib would need an API to
* access local data state
* access master OSM DB for additional data
* access other tools like taginfo

Integrating scripting environment may be difficult, but offers far greater
benefits of rule consistency and flexibility.


On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale  wrote:

> The technical differences between java and JS do not preclude generic
> thinking. Consider tzdata[1] for example, which does something analogous
> for time zone data.
>
> The "rules database" can be made portable, in the form of XML or JSON for
> example. The logic for using these rules can be described in a portable
> way. Then you add a set of compliance tests, and publish a reference
> implementation to demonstrate that is is possible to implement it. After
> that, the logic can be implemented in any language you like, checked
> against the compliance tests and the bindings published.
>
> Externalising the rules database enables updates and customisations for
> particular reasons. Depending on the specific use case and the associated
> non-functionals, validation could possibly be offered as a cloud service
> (not necessarily by OSM).
>
>
> //colin
>
> [1] https://en.wikipedia.org/wiki/Tz_database
>
> On 2017-12-24 12:18, James wrote:
>
> ID is javascript, JOSM is java. So right there I already see a
> intercompatibility issue
>
> On Dec 24, 2017 6:12 AM, "François Lacombe" 
> wrote:
>
>> Hi
>>
>> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
>> others).
>> Subsequently to wiki proposal voting and cleanups, it's currently
>> necessarily to open issues in each editor repository to ask for new tagging
>> validation rules.
>>
>> It can sometimes be time consuming to develop those new rules and such a
>> work is done independently by each project maintainer. While each project
>> have its own specific components, background logic is the same.
>>
>> Would a new lib called like osmtagvalidator or so in charge of doing
>> conform validation to wiki be useful?
>> It may be shared by any project involved in osm editing and preserve its
>> resources for other valuable developments.
>>
>> For me, validation doesn't prevent users to use tags they want, but only
>> warn them about possible mistakes.
>>
>> How would devs and users feel about this?
>>
>> All the best
>>
>> François
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[john whelan]

The problem with HOT inspired maperthons is half the participants will not
return.  So restricting their choices to the most common is helpful.

Typically I see highway=primary between two small settlements,
highway=living_steet by mappers mapping from imagery in African countries
that do not have any in legal existence.

Whilst there is a mechanism of validation to correct errors it's only very
recently that it has been possible to restrict validation to mappers with
some experience.  Even today with TM3 I can see tiles have been validated
by mappers with very little or even no experience and it is not possible to
identify the tiles easily to revalidate them and to be honest not that much
HOT mapping gets validated.

Even with mappers who use JOSM and do not map through HOT there are
problems.  One cheerful mapper managed to upload over 400 untagged ways
with JOSM and they would have been warned when uploading them.

It isn't possible to make things idiot proof new improved idiots arrive all
the time but by suggesting the most common tags to new mappers might make
it easier for them to select the correct values.

Cheerio John

On 24 Dec 2017 1:51 pm, "Andy Townsend"  wrote:

> On 24/12/17 18:41, john whelan wrote:
>
>> True but it's to do with data quality and avoiding the need to validate.
>>
>
> I don't think that different presets will "avoid the need to validate" in
> any scenario (HOT or otherwise) - the only thing that will do that is
> education and training - and having those educated and trained people come
> back again to do more mapping.
>
> Best Regards,
>
> Andy
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Andy Townsend]

On 24/12/17 18:41, john whelan wrote:

True but it's to do with data quality and avoiding the need to validate.


I don't think that different presets will "avoid the need to validate" 
in any scenario (HOT or otherwise) - the only thing that will do that is 
education and training - and having those educated and trained people 
come back again to do more mapping.


Best Regards,

Andy


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[john whelan]

True but it's to do with data quality and avoiding the need to validate.

Thank you for the comment.

Cheerio John

On 24 Dec 2017 1:37 pm, "Bryan Housel"  wrote:

> What you are describing has nothing to do with validation. This could be
> done now without any new development just by running a copy of iD with
> different presets.
>
> Sent from my iPhone
>
> On Dec 24, 2017, at 12:08 PM, john whelan  wrote:
>
> But what I suspect could be done is when iD in invoked from a HOT project
> it could restrict the options on things like highways to those in the wiki
> https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
> which would save a lot of living_streets etc. being added in parts of
> Africa where there aren't any.  Also reduce the numbers of highway=footway
> rather than path.
>
> We might even get a few more building=yes tags rather than the range we
> get at the moment.
>
> Thanks John
>
> On 24 December 2017 at 10:45, Bryan Housel  wrote:
>
>> Have you looked at https://github.com/osmlab/osmlint ?
>> Of all the current validation efforts, that seems like the most
>> promising.
>>
>> I’d definitely echo what other people are saying about avoiding the osm
>> wiki if possible.
>>
>> It works on vector tiles though, so to stuff it into an editor like iD,
>> we would need to write some kind of pipeline that does:
>> “current view of stuff in editor” -> "vector tile" -> "osmlint engine" ->
>> “results (geojson)” -> “back to the editor for user to see"
>>
>> It might work?
>>
>> Also… This problem of “validating OSM” is really unbounded.  You should
>> know that before you start working on it!  I’m not one to tell people not
>> to work on something but.. It’s really hard!  Tags are just made up all the
>> time by people.
>>
>> Can a `highway=residential` connect to a `power=line`?  - no!
>> Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
>> Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!
>>
>> Bryan
>>
>>
>>
>>
>> On Dec 24, 2017, at 4:47 AM, François Lacombe 
>> wrote:
>>
>> Hi
>>
>> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
>> others).
>> Subsequently to wiki proposal voting and cleanups, it's currently
>> necessarily to open issues in each editor repository to ask for new tagging
>> validation rules.
>>
>> It can sometimes be time consuming to develop those new rules and such a
>> work is done independently by each project maintainer. While each project
>> have its own specific components, background logic is the same.
>>
>> Would a new lib called like osmtagvalidator or so in charge of doing
>> conform validation to wiki be useful?
>> It may be shared by any project involved in osm editing and preserve its
>> resources for other valuable developments.
>>
>> For me, validation doesn't prevent users to use tags they want, but only
>> warn them about possible mistakes.
>>
>> How would devs and users feel about this?
>>
>> All the best
>>
>> François
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>>
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
>>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Bryan Housel]

What you are describing has nothing to do with validation. This could be done 
now without any new development just by running a copy of iD with different 
presets. 

Sent from my iPhone

> On Dec 24, 2017, at 12:08 PM, john whelan  wrote:
> 
> But what I suspect could be done is when iD in invoked from a HOT project it 
> could restrict the options on things like highways to those in the wiki  
> https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
> which would save a lot of living_streets etc. being added in parts of Africa 
> where there aren't any.  Also reduce the numbers of highway=footway rather 
> than path.
> 
> We might even get a few more building=yes tags rather than the range we get 
> at the moment.
> 
> Thanks John
> 
>> On 24 December 2017 at 10:45, Bryan Housel  wrote:
>> Have you looked at https://github.com/osmlab/osmlint ?
>> Of all the current validation efforts, that seems like the most promising.  
>> 
>> I’d definitely echo what other people are saying about avoiding the osm wiki 
>> if possible. 
>> 
>> It works on vector tiles though, so to stuff it into an editor like iD, we 
>> would need to write some kind of pipeline that does:
>> “current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> 
>> “results (geojson)” -> “back to the editor for user to see"
>> 
>> It might work?
>> 
>> Also… This problem of “validating OSM” is really unbounded.  You should know 
>> that before you start working on it!  I’m not one to tell people not to work 
>> on something but.. It’s really hard!  Tags are just made up all the time by 
>> people.
>> 
>> Can a `highway=residential` connect to a `power=line`?  - no!
>> Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
>> Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!
>> 
>> Bryan
>> 
>> 
>> 
>> 
>>> On Dec 24, 2017, at 4:47 AM, François Lacombe  
>>> wrote:
>>> 
>>> Hi
>>> 
>>> Here is an idea I got regarding tagging validation in editors (iD, JOSM, 
>>> others).
>>> Subsequently to wiki proposal voting and cleanups, it's currently 
>>> necessarily to open issues in each editor repository to ask for new tagging 
>>> validation rules. 
>>> 
>>> It can sometimes be time consuming to develop those new rules and such a 
>>> work is done independently by each project maintainer. While each project 
>>> have its own specific components, background logic is the same.
>>> 
>>> Would a new lib called like osmtagvalidator or so in charge of doing 
>>> conform validation to wiki be useful?
>>> It may be shared by any project involved in osm editing and preserve its 
>>> resources for other valuable developments.
>>> 
>>> For me, validation doesn't prevent users to use tags they want, but only 
>>> warn them about possible mistakes.
>>> 
>>> How would devs and users feel about this?
>>> 
>>> All the best
>>> 
>>> François
>>> ___
>>> talk mailing list
>>> talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk
>> 
>> 
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>> 
> 
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[François Lacombe]

Hi Bryan

Le 24 déc. 2017 4:45 PM, "Bryan Housel"  a écrit :

Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.


I didn't know OSMLint and OSM QA tiles before
Very promising indeed for parallel processing
Issue I see it's relations aren't available unfortunately


I’d definitely echo what other people are saying about avoiding the osm
wiki if possible.

Can you elaborate please ?
I just don't know elsewhere anyone can find comprehensive and consistent
information about tags despite wiki is not always perfect
Wiki got good functionalities to log contributions and revert vandalism too.


It works on vector tiles though, so to stuff it into an editor like iD, we
would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" ->
“results (geojson)” -> “back to the editor for user to see"

It might work?

It can clearly work :)
Nevertheless it's one usecase out of plenty
Validation systems can be used to do data audit too
That's why focusing on rules formatting is more versatile than writing
implementation unlinke what i was originally suggesting



Also… This problem of “validating OSM” is really unbounded.  You should
know that before you start working on it!  I’m not one to tell people not
to work on something but.. It’s really hard!  Tags are just made up all the
time by people.


I agree and it's a different problem
Focusing on rules formalism doesn't assume what rules should be.
Even if tags are made by people, some definitions can be commonly accepted
and they can be refined after some discussion. Validation can follow the
same peocess also.


Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

These are rules, not the description we should build to make them
understandable by software

All the best

François
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[john whelan]

But what I suspect could be done is when iD in invoked from a HOT project
it could restrict the options on things like highways to those in the wiki
https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
which would save a lot of living_streets etc. being added in parts of
Africa where there aren't any.  Also reduce the numbers of highway=footway
rather than path.

We might even get a few more building=yes tags rather than the range we get
at the moment.

Thanks John

On 24 December 2017 at 10:45, Bryan Housel  wrote:

> Have you looked at https://github.com/osmlab/osmlint ?
> Of all the current validation efforts, that seems like the most promising.
>
>
> I’d definitely echo what other people are saying about avoiding the osm
> wiki if possible.
>
> It works on vector tiles though, so to stuff it into an editor like iD, we
> would need to write some kind of pipeline that does:
> “current view of stuff in editor” -> "vector tile" -> "osmlint engine" ->
> “results (geojson)” -> “back to the editor for user to see"
>
> It might work?
>
> Also… This problem of “validating OSM” is really unbounded.  You should
> know that before you start working on it!  I’m not one to tell people not
> to work on something but.. It’s really hard!  Tags are just made up all the
> time by people.
>
> Can a `highway=residential` connect to a `power=line`?  - no!
> Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
> Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!
>
> Bryan
>
>
>
>
> On Dec 24, 2017, at 4:47 AM, François Lacombe 
> wrote:
>
> Hi
>
> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
> others).
> Subsequently to wiki proposal voting and cleanups, it's currently
> necessarily to open issues in each editor repository to ask for new tagging
> validation rules.
>
> It can sometimes be time consuming to develop those new rules and such a
> work is done independently by each project maintainer. While each project
> have its own specific components, background logic is the same.
>
> Would a new lib called like osmtagvalidator or so in charge of doing
> conform validation to wiki be useful?
> It may be shared by any project involved in osm editing and preserve its
> resources for other valuable developments.
>
> For me, validation doesn't prevent users to use tags they want, but only
> warn them about possible mistakes.
>
> How would devs and users feel about this?
>
> All the best
>
> François
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Bryan Housel]

Have you looked at https://github.com/osmlab/osmlint 
 ?
Of all the current validation efforts, that seems like the most promising.  

I’d definitely echo what other people are saying about avoiding the osm wiki if 
possible. 

It works on vector tiles though, so to stuff it into an editor like iD, we 
would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> 
“results (geojson)” -> “back to the editor for user to see"

It might work?

Also… This problem of “validating OSM” is really unbounded.  You should know 
that before you start working on it!  I’m not one to tell people not to work on 
something but.. It’s really hard!  Tags are just made up all the time by people.

Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

Bryan




> On Dec 24, 2017, at 4:47 AM, François Lacombe  
> wrote:
> 
> Hi
> 
> Here is an idea I got regarding tagging validation in editors (iD, JOSM, 
> others).
> Subsequently to wiki proposal voting and cleanups, it's currently necessarily 
> to open issues in each editor repository to ask for new tagging validation 
> rules. 
> 
> It can sometimes be time consuming to develop those new rules and such a work 
> is done independently by each project maintainer. While each project have its 
> own specific components, background logic is the same.
> 
> Would a new lib called like osmtagvalidator or so in charge of doing conform 
> validation to wiki be useful?
> It may be shared by any project involved in osm editing and preserve its 
> resources for other valuable developments.
> 
> For me, validation doesn't prevent users to use tags they want, but only warn 
> them about possible mistakes.
> 
> How would devs and users feel about this?
> 
> All the best
> 
> François
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[François Lacombe]

Thanks for answers :)

I said a lib, but indeed the diversity of projects would be too problematic
for implementation and rollouts.
JSON, xml, text are fine
https://twitter.com/VincentPrivat/status/944923315831033856

As Simon said, taginfo is a great tool to extract data from wiki.
Even if it's not perfect, the wiki is the only reference I know and where
data consumers go to get information. Then it's relevent to improve it
It would be hard to put all practicies and rules in wiki keys scorecard
templates (from where taginfo get most of its input). Should we complete it
in taginfo itself ?

All the best

François

Le 24 déc. 2017 1:47 PM, "Simon Poole"  a écrit :

> On the one hand lots of the in principle useful information in the wiki is
> not really easily extractable and on the other hand it is prone to
> manipulation in more than one way (current fad is to add big warnings about
> tagging errors what are not).
>
> IMHO addressing the first issue would likely be more helpful and perhaps
> allow the generation of at least rudimentary presets directly from the wiki
> (potentially with support from taginfo).
>
> Simon
>
>
>
> On 24.12.2017 12:54, Mateusz Konieczny wrote:
>
> > conform validation to wiki
>
> Sometimes wiki is wrong and should be changed.
>
> Note also that authors of different tools have different opinions how and
> what should be reported as errors.
>
>
> On 24 Dec 2017 11:12 a.m., "François Lacombe" 
> wrote:
>
> Hi
>
> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
> others).
> Subsequently to wiki proposal voting and cleanups, it's currently
> necessarily to open issues in each editor repository to ask for new tagging
> validation rules.
>
> It can sometimes be time consuming to develop those new rules and such a
> work is done independently by each project maintainer. While each project
> have its own specific components, background logic is the same.
>
> Would a new lib called like osmtagvalidator or so in charge of doing
> conform validation to wiki be useful?
> It may be shared by any project involved in osm editing and preserve its
> resources for other valuable developments.
>
> For me, validation doesn't prevent users to use tags they want, but only
> warn them about possible mistakes.
>
> How would devs and users feel about this?
>
> All the best
>
> François
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
>
>
> ___
> talk mailing 
> listtalk@openstreetmap.orghttps://lists.openstreetmap.org/listinfo/talk
>
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Simon Poole]

On the one hand lots of the in principle useful information in the wiki
is not really easily extractable and on the other hand it is prone to
manipulation in more than one way (current fad is to add big warnings
about tagging errors what are not).

IMHO addressing the first issue would likely be more helpful and perhaps
allow the generation of at least rudimentary presets directly from the
wiki (potentially with support from taginfo).

Simon



On 24.12.2017 12:54, Mateusz Konieczny wrote:
> > conform validation to wiki
>
> Sometimes wiki is wrong and should be changed. 
>
> Note also that authors of different tools have different opinions how
> and what should be reported as errors.
>
>
> On 24 Dec 2017 11:12 a.m., "François Lacombe"
> > wrote:
>
> Hi
>
> Here is an idea I got regarding tagging validation in editors (iD,
> JOSM, others).
> Subsequently to wiki proposal voting and cleanups, it's currently
> necessarily to open issues in each editor repository to ask for
> new tagging validation rules. 
>
> It can sometimes be time consuming to develop those new rules and
> such a work is done independently by each project maintainer.
> While each project have its own specific components, background
> logic is the same.
>
> Would a new lib called like osmtagvalidator or so in charge of
> doing conform validation to wiki be useful?
> It may be shared by any project involved in osm editing and
> preserve its resources for other valuable developments.
>
> For me, validation doesn't prevent users to use tags they want,
> but only warn them about possible mistakes.
>
> How would devs and users feel about this?
>
> All the best
>
> François
>
> ___
> talk mailing list
> talk@openstreetmap.org 
> https://lists.openstreetmap.org/listinfo/talk
> 
>
>
>
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Colin Smale]

The technical differences between java and JS do not preclude generic
thinking. Consider tzdata[1] for example, which does something analogous
for time zone data. 

The "rules database" can be made portable, in the form of XML or JSON
for example. The logic for using these rules can be described in a
portable way. Then you add a set of compliance tests, and publish a
reference implementation to demonstrate that is is possible to implement
it. After that, the logic can be implemented in any language you like,
checked against the compliance tests and the bindings published. 

Externalising the rules database enables updates and customisations for
particular reasons. Depending on the specific use case and the
associated non-functionals, validation could possibly be offered as a
cloud service (not necessarily by OSM).

//colin 

[1] https://en.wikipedia.org/wiki/Tz_database 

On 2017-12-24 12:18, James wrote:

> ID is javascript, JOSM is java. So right there I already see a 
> intercompatibility issue 
> 
> On Dec 24, 2017 6:12 AM, "François Lacombe"  wrote:
> 
>> Hi 
>> 
>> Here is an idea I got regarding tagging validation in editors (iD, JOSM, 
>> others). 
>> Subsequently to wiki proposal voting and cleanups, it's currently 
>> necessarily to open issues in each editor repository to ask for new tagging 
>> validation rules.  
>> 
>> It can sometimes be time consuming to develop those new rules and such a 
>> work is done independently by each project maintainer. While each project 
>> have its own specific components, background logic is the same. 
>> 
>> Would a new lib called like osmtagvalidator or so in charge of doing conform 
>> validation to wiki be useful? 
>> It may be shared by any project involved in osm editing and preserve its 
>> resources for other valuable developments. 
>> 
>> For me, validation doesn't prevent users to use tags they want, but only 
>> warn them about possible mistakes. 
>> 
>> How would devs and users feel about this? 
>> 
>> All the best 
>> 
>> François 
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk [1]
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
 

Links:
--
[1] https://lists.openstreetmap.org/listinfo/talk___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Andy Townsend]

On 24/12/17 11:49, Darafei "Komяpa" Praliaskouski wrote:



It would be cool to have such library running on osm2pgsql import, 
fixing and complaining/skipping all the mistaked tag usage it can 
detect. This can possibly resurrect some objects that are lost because 
of typos or old tagging schemes.




That already exists - the lua tag transformation process can do exactly 
that.  Here's an example:


https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua#L2056

Obviously whether a particular map style wants to render typing / 
tagging errors is a different matter entirely.


Best Regards,

Andy




___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Mateusz Konieczny]

> conform validation to wiki

Sometimes wiki is wrong and should be changed.

Note also that authors of different tools have different opinions how and
what should be reported as errors.


On 24 Dec 2017 11:12 a.m., "François Lacombe" 
wrote:

Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM,
others).
Subsequently to wiki proposal voting and cleanups, it's currently
necessarily to open issues in each editor repository to ask for new tagging
validation rules.

It can sometimes be time consuming to develop those new rules and such a
work is done independently by each project maintainer. While each project
have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing
conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its
resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only
warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[Komяpa]

Hi,

What language would that library be in, if level0 and iD are in javascript,
JOSM and Vespucci are in Java, and many random console tools are in python
and perl?

It would be cool to have such library running on osm2pgsql import, fixing
and complaining/skipping all the mistaked tag usage it can detect. This can
possibly resurrect some objects that are lost because of typos or old
tagging schemes.

вс, 24 дек. 2017 г. в 14:12, François Lacombe :

> Hi
>
> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
> others).
> Subsequently to wiki proposal voting and cleanups, it's currently
> necessarily to open issues in each editor repository to ask for new tagging
> validation rules.
>
> It can sometimes be time consuming to develop those new rules and such a
> work is done independently by each project maintainer. While each project
> have its own specific components, background logic is the same.
>
> Would a new lib called like osmtagvalidator or so in charge of doing
> conform validation to wiki be useful?
> It may be shared by any project involved in osm editing and preserve its
> resources for other valuable developments.
>
> For me, validation doesn't prevent users to use tags they want, but only
> warn them about possible mistakes.
>
> How would devs and users feel about this?
>
> All the best
>
> François
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM tagging validation lib

[James]

ID is javascript, JOSM is java. So right there I already see a
intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" 
wrote:

> Hi
>
> Here is an idea I got regarding tagging validation in editors (iD, JOSM,
> others).
> Subsequently to wiki proposal voting and cleanups, it's currently
> necessarily to open issues in each editor repository to ask for new tagging
> validation rules.
>
> It can sometimes be time consuming to develop those new rules and such a
> work is done independently by each project maintainer. While each project
> have its own specific components, background logic is the same.
>
> Would a new lib called like osmtagvalidator or so in charge of doing
> conform validation to wiki be useful?
> It may be shared by any project involved in osm editing and preserve its
> resources for other valuable developments.
>
> For me, validation doesn't prevent users to use tags they want, but only
> warn them about possible mistakes.
>
> How would devs and users feel about this?
>
> All the best
>
> François
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk