Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Brett Henderson]

Rogier Wolff wrote:
> On Fri, Aug 15, 2008 at 07:59:07AM -0700, Karl Newman wrote:
>   
>>> Making some parameter negative to mean "this object is incomplete"
>>> sounds like a hack to me. It's something that you might do in an
>>> internal database, or in an in-memory datastructure, but it should not
>>> be in an API. Not ever. It's a cludge.
>>>   
>
>   
>> I was thinking more of the general case--"this object should not be
>> uploaded because it's been manipulated in some way that invalidates
>> the data".  Setting the version of an entity as negative would
>> easily prevent that, using the (proposed) server api. This "hack" is
>> intended for offline files, not the main API.
>> 
>
> Of course it's not the "main API", but if you use this method in one
> instance, it will be seen as an example. People will copy it. And
> before you know it, data will be passed along from app to app with this
> advice taken to heart. And then you have it in an API. 
>
> You're saying the main API will ignore such uploads. Great. Now the
> fact that these are ignored has become part of the API!
>   

Apologies for returning email to the list if that wasn't the intent of 
your email.  I just assumed it was accidental, it happens a lot around 
here :-)  Feel free to voice opinions, I'll always listen, and will 
often disagree :-)

To be honest, I don't know what the best solution is here.  I tend to 
agree that using features for something they were never intended is a 
bad idea and often has unintended consequences down the track.  In this 
case it may be the lesser of several evils though.

We have these options:
1. Do nothing - This is always my default position, and the best unless 
you're absolutely sure you have a better answer.
2. Change to negative identifier - Doesn't solve the problem of 
preventing uploads, and is somewhat painful to implement, especially if 
you take relations into account.
3. Change version identifier - Prevents uploads which was the original 
problem.  Not the original intent of this attribute.  Consumers need to 
know that version became negative due to mangling of the entity.
4. Add a tag - Only works if all consumers of the data explicitly look 
for the tag.
5. Invent a new entity attribute indicating mangling - Perhaps the 
"cleanest" approach.  But more complicated than all others and requires 
an API change.  Would have to be well thought out, perhaps could even be 
an "origin" tag making it more generic.  Just throwing the idea out 
there, haven't thought it through.

Of the above I tend to lean towards updating the version.

Cheers,
Brett


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Brett Henderson]

Rogier Wolff wrote:
>
> So, then, the recommendation should be that when extracting data, this
> same procedure is followed, and if not, those ways should be marked
> incomplete in a way that as much as possible doesn't interfere with
> current tools. Making a node-id negative, which in other tools is used
> a as a cheap "flag", therefore becomes a non-option. It also runs the
> risk of those negative-ids becoming entered into a database, as
> separate entities.
>   
If possible I'd always provide complete ways, but it isn't simple using 
a stream approach such as that used in osmosis.  A database is much more 
suited to this type of application and is something the osmosis pgsql 
schema tasks aim to solve.

If stream processing is used (ie. current --bounding-box task) I don't 
know whether chopping ways to fit the available node set or including 
full ways and breaking referential integrity is the best way forward.  
Both are easy to support and could be switched with a task option, the 
main question is which behaviour should be the default.
> So a "key=value" pair like "incomplete=true" or "status=chopped" would
> be my favorite. 
>   
I agree changing ids to negative is a bad idea.  It does nothing to 
prevent uploads although is arguably better than modifying existing 
ones.  My real issue with changing to negative is maintaining the 
referential integrity, it has to be changed everywhere it is referenced.

I'd recommend against using a tag for this.  The tag will only help if 
everybody always looks for it.  The chances of everybody agreeing on a 
single tag and more importantly updating all tools to utilise it is 
fairly remote.  I guess you could modify the api to look for the tag and 
reject it but ideally the api should remain tag agnostic, something 
osmosis has also managed to do up to this point.  I still think setting 
a version id to 0 would be the best approach, it will forcefully stop a 
modified entity from being uploaded.  Although I guess a tag could also 
be added if desired.


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Rogier Wolff]

On Fri, Aug 15, 2008 at 07:59:07AM -0700, Karl Newman wrote:
> > Making some parameter negative to mean "this object is incomplete"
> > sounds like a hack to me. It's something that you might do in an
> > internal database, or in an in-memory datastructure, but it should not
> > be in an API. Not ever. It's a cludge.

> I was thinking more of the general case--"this object should not be
> uploaded because it's been manipulated in some way that invalidates
> the data".  Setting the version of an entity as negative would
> easily prevent that, using the (proposed) server api. This "hack" is
> intended for offline files, not the main API.

Of course it's not the "main API", but if you use this method in one
instance, it will be seen as an example. People will copy it. And
before you know it, data will be passed along from app to app with this
advice taken to heart. And then you have it in an API. 

You're saying the main API will ignore such uploads. Great. Now the
fact that these are ignored has become part of the API!

Roger. 

-- 
** [EMAIL PROTECTED] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233**
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. - Adapted from lxrbot FAQ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Karl Newman]

On Fri, Aug 15, 2008 at 7:28 AM, Rogier Wolff <[EMAIL PROTECTED]>wrote:

> On Fri, Aug 15, 2008 at 07:04:11AM -0700, Karl Newman wrote:
> > The suggestion was to mark the version of the entity as negative. The
> > version attribute does not yet exist but is proposed for the upcoming 0.6
> > API. Making it negative would prevent it from being uploaded because the
> > server would reject it. The intent for the version attribute is to avoid
> > simultaneous editing of the same object.
>
> I've been on this list for a few days now, so I might be
> rambling. Feel free to ignore my advice.
>
> Making some parameter negative to mean "this object is incomplete"
> sounds like a hack to me. It's something that you might do in an
> internal database, or in an in-memory datastructure, but it should not
> be in an API. Not ever. It's a cludge.
>

I was thinking more of the general case--"this object should not be uploaded
because it's been manipulated in some way that invalidates the data".
Setting the version of an entity as negative would easily prevent that,
using the (proposed) server api. This "hack" is intended for offline files,
not the main API.


> If you add cludges to your datastructures and APIs you'll end up with
> a big mess that in the future will become hard to maintain. The
> current structure with XML APIs and transfer methods sounds like a
> well thought-out system. Things can be done more efficient, but for
> the sake of readability everything is in ascii.
>

Well, UTF-8, but okay...


>
> That's nice. Cludges don't belong here.
>
> I had sent the mail that this discussion came out of as private
> EMail. That was intentional: I don't consider myself to be
> knowledgable enough about the subject matter to give accurate public
> advice in this matter. This is evidenced by the fact that I don't have
> the terminology down enough to have noticed the difference between
> version numbers and node IDs... So, again, feel free to tell me that I
> don't know enough yet to comment, and I'll shut up. :-)
>
>Roger.
>

Hmm...  According to my mail client (gmail), you sent it to me, Brett H and
the dev list. If you do want to have a private conversation, please indicate
it in the message (i.e., "Please don't respond to the list") otherwise I'll
bring conversations back on to the list for any relevant discussions.
Anyway, I won't tell you to shut up; it's good to have discussion--it's why
the lists exist. I see your point, but I don't think it's a terrible hack. I
thought it was sort of an elegant solution, really.

Karl
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Rogier Wolff]

On Fri, Aug 15, 2008 at 07:04:11AM -0700, Karl Newman wrote:
> The suggestion was to mark the version of the entity as negative. The
> version attribute does not yet exist but is proposed for the upcoming 0.6
> API. Making it negative would prevent it from being uploaded because the
> server would reject it. The intent for the version attribute is to avoid
> simultaneous editing of the same object.

I've been on this list for a few days now, so I might be
rambling. Feel free to ignore my advice. 

Making some parameter negative to mean "this object is incomplete"
sounds like a hack to me. It's something that you might do in an
internal database, or in an in-memory datastructure, but it should not
be in an API. Not ever. It's a cludge.

If you add cludges to your datastructures and APIs you'll end up with
a big mess that in the future will become hard to maintain. The
current structure with XML APIs and transfer methods sounds like a
well thought-out system. Things can be done more efficient, but for
the sake of readability everything is in ascii.

That's nice. Cludges don't belong here. 

I had sent the mail that this discussion came out of as private
EMail. That was intentional: I don't consider myself to be
knowledgable enough about the subject matter to give accurate public
advice in this matter. This is evidenced by the fact that I don't have
the terminology down enough to have noticed the difference between
version numbers and node IDs... So, again, feel free to tell me that I
don't know enough yet to comment, and I'll shut up. :-)

Roger. 

-- 
** [EMAIL PROTECTED] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233**
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. - Adapted from lxrbot FAQ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Karl Newman]

On Fri, Aug 15, 2008 at 6:52 AM, Rogier Wolff <[EMAIL PROTECTED]>wrote:

> On Fri, Aug 15, 2008 at 06:28:04AM -0700, Karl Newman wrote:
> > > Those need to be marked in a way that won't interfere with current use
> > > of those ways, as well as inform interested parties/programs that a
> > > more complete way is available on the server.
> > >
> > >
> > > My personal choice would be to move towards the server delivering all
> > > ways that have at least one point inside the area. If you request two
> > > neighbouring areas some ways will be reported double
> > >
> > >Roger.
> > >
> >
> > The main api server does return complete ways that have at least one
> point
> > inside the area. This discussion was about the behavior of other tools
> > (namely, Osmosis) that manipulate the OSM data offline.
>
> So, then, the recommendation should be that when extracting data, this
> same procedure is followed, and if not, those ways should be marked
> incomplete in a way that as much as possible doesn't interfere with
> current tools. Making a node-id negative, which in other tools is used
> a as a cheap "flag", therefore becomes a non-option. It also runs the
> risk of those negative-ids becoming entered into a database, as
> separate entities.
>
> So a "key=value" pair like "incomplete=true" or "status=chopped" would
> be my favorite.
>
>Roger.
>

The suggestion was to mark the version of the entity as negative. The
version attribute does not yet exist but is proposed for the upcoming 0.6
API. Making it negative would prevent it from being uploaded because the
server would reject it. The intent for the version attribute is to avoid
simultaneous editing of the same object.

Karl
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Rogier Wolff]

On Fri, Aug 15, 2008 at 06:28:04AM -0700, Karl Newman wrote:
> > Those need to be marked in a way that won't interfere with current use
> > of those ways, as well as inform interested parties/programs that a
> > more complete way is available on the server.
> >
> >
> > My personal choice would be to move towards the server delivering all
> > ways that have at least one point inside the area. If you request two
> > neighbouring areas some ways will be reported double
> >
> >Roger.
> >
> 
> The main api server does return complete ways that have at least one point
> inside the area. This discussion was about the behavior of other tools
> (namely, Osmosis) that manipulate the OSM data offline.

So, then, the recommendation should be that when extracting data, this
same procedure is followed, and if not, those ways should be marked
incomplete in a way that as much as possible doesn't interfere with
current tools. Making a node-id negative, which in other tools is used
a as a cheap "flag", therefore becomes a non-option. It also runs the
risk of those negative-ids becoming entered into a database, as
separate entities. 

So a "key=value" pair like "incomplete=true" or "status=chopped" would
be my favorite. 

Roger. 

-- 
** [EMAIL PROTECTED] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233**
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. - Adapted from lxrbot FAQ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Karl Newman]

> I'm not sure if I understand things fully, but I thought that e.g. for
> offline editing you can request the map data about an area from the
> mapserver. Say 45.5N to 45.7N, 4.3E to 4.4E.
>
> Now while "exporting" from the database, the question is: What do you
> do with ways that are only partially inside that area? Either you
> don't export them alltogether. But in for example an offline editor
> this can look very odd: some ways near the edge will be missing
> because they are partially outside the selected area.
>
> You can also include nodes outside the area, and deliver complete ways.
>
> This also looks odd, as at the boundary some ways will continue
> outside, and again the temptation will exist to add ways outside the
> area, although they are already present in the full database.
>
> So apparently a choice has been made to chop off ways outside the
> selected area.
>
> Those need to be marked in a way that won't interfere with current use
> of those ways, as well as inform interested parties/programs that a
> more complete way is available on the server.
>
>
> My personal choice would be to move towards the server delivering all
> ways that have at least one point inside the area. If you request two
> neighbouring areas some ways will be reported double
>
>Roger.
>

The main api server does return complete ways that have at least one point
inside the area. This discussion was about the behavior of other tools
(namely, Osmosis) that manipulate the OSM data offline.

Karl
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] [Fwd: Re: Chopped of ways. New flag for OSM XML?]

[Brett Henderson]

Forwarding to list ...

 Original Message 
Subject:Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
Date:   Fri, 15 Aug 2008 06:21:14 +0200
From:   Rogier Wolff <[EMAIL PROTECTED]>
Organisation:   BitWizard.nl
To: Brett Henderson <[EMAIL PROTECTED]>
References: <[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]> <[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]> 
<[EMAIL PROTECTED]>



On Fri, Aug 15, 2008 at 12:28:14AM +1000, Brett Henderson wrote:
> Karl Newman wrote:
> > On Thu, Aug 14, 2008 at 12:47 AM, Brett Henderson <[EMAIL PROTECTED] 
> > > wrote:
> >
> > Jochen Topf wrote:
> >
> > But people *want* to use the data they get from some kind of
> > extraction
> > process to base decisions upon. And they should be able to.
> > Not all the
> > data is invalidated by an extraction, only some of it. It
> > makes sense
> > to mark those parts invalid that are, so that an automated
> > process can
> > decide what it uses and what not.
> >
> > Jochen
> >  
> >
> > Sure.  If I stop mangling ways then I can't think of a particular
> > case where this would be useful for osmosis processing ... but
> > that's not to say one doesn't exist.
> >
> > I'll keep quiet and let those more impacted by the topic resume
> > normal discussions ;-)
> >
> > How about Dave Stubb's new tag transform plugin? And the other example 
> > I mentioned, tile cutting, where ways are intentionally mangled to end 
> > exactly on the tile border?
> I didn't explain myself very well.  In the case of these transform tasks 
> you could simply mark the whole file as unsuitable for upload, I 
> couldn't see a case where you'd need to know which entities have been 
> mangled and which haven't.  But again, perhaps I just haven't considered 
> a case where it might be useful.
> 
> If you do need per-entity flagging then 80n's suggestion of fudging the 
> version id (eg. setting to 0) is worth thinking about.

I'm not sure if I understand things fully, but I thought that e.g. for
offline editing you can request the map data about an area from the
mapserver. Say 45.5N to 45.7N, 4.3E to 4.4E. 

Now while "exporting" from the database, the question is: What do you
do with ways that are only partially inside that area? Either you
don't export them alltogether. But in for example an offline editor
this can look very odd: some ways near the edge will be missing
because they are partially outside the selected area.

You can also include nodes outside the area, and deliver complete ways. 

This also looks odd, as at the boundary some ways will continue
outside, and again the temptation will exist to add ways outside the
area, although they are already present in the full database.

So apparently a choice has been made to chop off ways outside the
selected area. 

Those need to be marked in a way that won't interfere with current use
of those ways, as well as inform interested parties/programs that a
more complete way is available on the server.


My personal choice would be to move towards the server delivering all
ways that have at least one point inside the area. If you request two
neighbouring areas some ways will be reported double 

Roger. 

-- 
** [EMAIL PROTECTED] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233**
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. - Adapted from lxrbot FAQ


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev