Re: [OSM-talk] xybot edit area size
On Fri, 09 Apr 2010 16:14:15 +0100, Martin Fossdal Guttesen wrote: The best way to get rid of bot changes is to ensure that as fews as possible errors get into osm Currently even if my area is error free I see errors from across the globe :( -- pratite me na twitteru - www.twitter.com/valentt blog: http://kernelreloaded.blog385.com linux, anime, spirituality, windsurf, wireless, ronjenje, pametne kuće registered as user #367004 with the Linux Counter, http://counter.li.org. ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On 10 Apr 2010, at 03:33, Martin Fossdal Guttesen wrote: Sorry i dont know Flash or ActionScript and after thinking about it. it would be better to handle it on the server and i dont know rails ruby or what it is called but i have looked at the source and i think i have found the spot where the change has to be made in way.rb line 189 def add_tag_keyval(k, v) @tags = Hash.new unless @tags # duplicate tags are now forbidden, so we can't allow values # in the hash to be overwritten. raise OSM::APIDuplicateTagsError.new(way, self.id, k) if @tags.include? k @tags[k] = v end as i said before i dont know ruby, but a little googling i found strip so i think that it should be @tags[k.strip] = v.strip there should also be a check if the trimmed v and k are empty and raise errors accordingly Actually you'll find that keys and values of zero length are explicitly allowed take a look at the *_tag model validations and the relevant unit tests. Shaun ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Wed, 07 Apr 2010 07:57:37 +, Valent Turkovic wrote: It would be ideal if it would use small areas of only few square km around each edit, idea beeing so that xybot edits don't show in history tab when looking at some area that wasn't edited. Example of area size that xybot uses: http://dl.dropbox.com/u/184632/xybot.png Ideally I would like to see bot edits in area that I'm looking at but not ones 5000km away ;( Anybody? -- pratite me na twitteru - www.twitter.com/valentt blog: http://kernelreloaded.blog385.com linux, anime, spirituality, windsurf, wireless, ronjenje, pametne kuće registered as user #367004 with the Linux Counter, http://counter.li.org. ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Isn't there some option to osmosis that bot operators could easily use to convert their huge .osc files into something that's split by either 10km^2 or 1000 changes, whatever comes first? ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Hi, Ævar Arnfjörð Bjarmason wrote: Isn't there some option to osmosis that bot operators could easily use to convert their huge .osc files into something that's split by either 10km^2 or 1000 changes, whatever comes first? Neither option exists, and the first one you mention is not even possible since osc files do not necessarily have spatial information for ways and relations. Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
The best way to get rid of bot changes is to ensure that as fews as possible errors get into osm no errors = no need to correct some of the corrections made by xybot are possible to avoid by making the editors a little smarter foreksample Removing surrounding whitespace and empty tags i checked one node and it was a trailing whitespace made in Potlatch 1.3e it should be a nobrainer to automaticly remove surrounding whitespace in potlatch when a user enters one -- From: Frederik Ramm frede...@remote.org Sent: Friday, April 09, 2010 3:15 PM To: Ævar Arnfjörð Bjarmason ava...@gmail.com Cc: talk@openstreetmap.org Subject: Re: [OSM-talk] xybot edit area size Hi, Ævar Arnfjörð Bjarmason wrote: Isn't there some option to osmosis that bot operators could easily use to convert their huge .osc files into something that's split by either 10km^2 or 1000 changes, whatever comes first? Neither option exists, and the first one you mention is not even possible since osc files do not necessarily have spatial information for ways and relations. Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
The best way to get rid of bot changes is to ensure that as fews as possible errors get into osm no errors = no need to correct some of the corrections made by xybot are possible to avoid by making the editors a little smarter foreksample Removing surrounding whitespace and empty tags i checked one node and it was a trailing whitespace made in Potlatch 1.3e it should be a nobrainer to automaticly remove surrounding whitespace in potlatch when a user enters one -- From: Frederik Ramm frede...@remote.org Sent: Friday, April 09, 2010 3:15 PM To: Ævar Arnfjörð Bjarmason ava...@gmail.com Cc: talk@openstreetmap.org Subject: Re: [OSM-talk] xybot edit area size Hi, Ævar Arnfjörð Bjarmason wrote: Isn't there some option to osmosis that bot operators could easily use to convert their huge .osc files into something that's split by either 10km^2 or 1000 changes, whatever comes first? Neither option exists, and the first one you mention is not even possible since osc files do not necessarily have spatial information for ways and relations. Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Sorry i dont know Flash or ActionScript and after thinking about it. it would be better to handle it on the server and i dont know rails ruby or what it is called but i have looked at the source and i think i have found the spot where the change has to be made in way.rb line 189 def add_tag_keyval(k, v) @tags = Hash.new unless @tags # duplicate tags are now forbidden, so we can't allow values # in the hash to be overwritten. raise OSM::APIDuplicateTagsError.new(way, self.id, k) if @tags.include? k @tags[k] = v end as i said before i dont know ruby, but a little googling i found strip so i think that it should be @tags[k.strip] = v.strip there should also be a check if the trimmed v and k are empty and raise errors accordingly and the same changes should also be made in node.rb line 255 cheers Martin (LiFo) -- From: Richard Fairhurst rich...@systemed.net Sent: Friday, April 09, 2010 4:54 PM To: talk@openstreetmap.org Subject: Re: [OSM-talk] xybot edit area size Martin Fossdal Guttesen wrote: it should be a nobrainer to automaticly remove surrounding whitespace in potlatch when a user enters one I agree. Since it's so easy, I look forward to your patch. :) cheers Richard -- View this message in context: http://n2.nabble.com/xybot-edit-area-size-tp4863523p4877907.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Ban botsone!!! I don't really like how they run on the whole world, and not even on a country level. Why are they just editing one node here and there, it could make more programmic sense to download an appropriate-sized area, work through it (doing what ever checks these bots deem important) and then upload the 'corrections' for that area. This certainly makes it easier if people implement a bots=no on areas to flag that bots should not mess with their turf. On 7 April 2010 13:15, andrzej zaborowski balr...@gmail.com wrote: On 7 April 2010 21:50, Valent Turkovic valent.turko...@gmail.com wrote: On Wed, 07 Apr 2010 10:41:22 +0200, Frank Sautter wrote: maybe this service could also be implemented on the openstreetmap.org website, as the ito osmmapper is frequently offline (as it is at the moment) I agree we shouldn't depend on 3rd party service for such important feature like watching for malicious activity. If other bots implement bot=yes tag then it would be trivial to filter them, right? I'm using bot=yes on some bigger, bot-like, edits often. But still ideal situation is that bots also use much smaller areas for edits so that you see if some bot has changed something in your area of interest, right? That would be good but on the other hand lots of very similar changes should be grouped together in one changeset for convenience, if our tools can't cope with it then let's fix the tools. Otherwise you might get 10k changesets in on run, each changing a single node, just for the sake of not generating too big bboxes. The ideal situation would be having infinite processing power etc. :) But with the existing limitations, I think the show bot edits check boxes is a fair approximation. Frederik said it wouldn't be a complete solution to the problem of easy monitoring changes in your area, I think, because sometimes the bot edit is just the one you're interested in. But it is a nice feature if we don't claim that it's a solution to this problem. It would be a nice feature on its own. On a different note, Frederik, you said you wouldn't put the history tab on the website and didn't think it was a good thing. Is that precisely because of the bboxes problem (i.e. that the edit needs not intersect the current view, just its bbox), or is there another reason you said that? Cheers ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Gregory o...@livingwithdragons.com http://www.livingwithdragons.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Valent Turkovic schrieb: But still ideal situation is that bots also use much smaller areas for edits so that you see if some bot has changed something in your area of interest, right? most bots (and also xybot) do not have any clue about the spatial aspects of a way or relation. normally a osm-xml file is piped through a bot (e.g. perl script). if it stumbles upon something fishy (e.g. a multiple consecutive nodes in a way) it will correct this problem on this entity. it does not know where in the world this problem is. it doesn't have a big database in the background which it could query where this entity is located and store it until the pipe has reached EOF and then make some handy spatial chunks of all its changes. the next question would be: what is the right size of a changeset? 10m², 100m², 1km², 10km², 100km², 1000km² the only solution to really fix the spamming of the history tab the way the history tab is implemented right now would be to put each change in its own changeset. but that's what we had in api 0.5 and would lead changesets ad absurdum and i believe a single bot-run should result in a single changeset. in my eyes there are three solutions for the problem of history spamming: * evaluate the bot=yes flag in changesets in the history tab. in consequence all bots should set this tag. drawback will be, that there could be malicious bot changes! * add a bot-flag to bot-accounts and filter those changeset in the history. same drawback as above. * replace the history tab with something like on ito-world osmmapper (including the rss-feed of changes) frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Thu, 08 Apr 2010 12:45:44 +0200, Frank Sautter openstreet...@sautter.com wrote: Valent Turkovic schrieb: But still ideal situation is that bots also use much smaller areas for edits so that you see if some bot has changed something in your area of interest, right? most bots (and also xybot) do not have any clue about the spatial aspects of a way or relation. normally a osm-xml file is piped through a bot (e.g. perl script). if it stumbles upon something fishy (e.g. a multiple consecutive nodes in a way) it will correct this problem on this entity. it does not know where in the world this problem is. it doesn't have a big database in the background which it could query where this entity is located and store it until the pipe has reached EOF and then make some handy spatial chunks of all its changes. Do correct me if I'm wrong, but don't the nodes have lat/lon with them and is the location therefore known? Maarten ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Gregory schrieb: Ban botsone!!! one of the keys on your keyboard seems to be stuck ;-) I don't really like how they run on the whole world, and not even on a country level. Why are they just editing one node here and there, it could make more programmic sense to download an appropriate-sized area, work through it (doing what ever checks these bots deem important) and then upload the 'corrections' for that area. again: what should be the size of those appropriate-sized areas? This certainly makes it easier if people implement a bots=no on areas to flag that bots should not mess with their turf. aahh i see, some areas contain only YOUR precious data. is your middle name Gollum? alone xybot is fixing several thousand errors each day, not including all the other bots that are active, but they still leave many errors behind that could not be fixed by a peace of software... i'm curious how many errors will show up in http://keepright.ipax.at/ or http://tools.geofabrik.de/osmi/ in YOUR nobot aera and how many this would be if they would not be constantly fixed by all those bots. frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Maarten Deen schrieb: Do correct me if I'm wrong, but don't the nodes have lat/lon with them and is the location therefore known? just have a look: http://www.openstreetmap.org/api/0.6/way/101/ frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Thu, 08 Apr 2010 13:24:50 +0200, Frank Sautter openstreet...@sautter.com wrote: Maarten Deen schrieb: Do correct me if I'm wrong, but don't the nodes have lat/lon with them and is the location therefore known? just have a look: http://www.openstreetmap.org/api/0.6/way/101/ Oh, right. They are in the osm file but not in the element that is being processed at that time. It would make it a tad more complicated to get that data at that point. But what would be against starting with a osm file with a smaller bbox as input? Or uploading single items in seperate changesets? Maarten ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Hi, On 8 April 2010 12:45, Frank Sautter openstreet...@sautter.com wrote: normally a osm-xml file is piped through a bot (e.g. perl script). if it stumbles upon something fishy (e.g. a multiple consecutive nodes in a way) it will correct this problem on this entity. it does not know where in the world this problem is. it doesn't have a big database in the background which it could query where this entity is located and store it until the pipe has reached EOF and then make some handy spatial chunks of all its changes. the next question would be: what is the right size of a changeset? 10m², 100m², 1km², 10km², 100km², 1000km² the only solution to really fix the spamming of the history tab the way the history tab is implemented right now would be to put each change in its own changeset. but that's what we had in api 0.5 and would lead changesets ad absurdum and i believe a single bot-run should result in a single changeset. It still wouldn't be ideal because you might be editing a long diagonal way and the changeset would be visible in history anywhere in the rectangle defined by the two ends. Really the only solution is if the tab used something like ST_Intersect on the geometries instead of on the bboxes and this is more intensive computationally. Cheers ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
Maarten Deen schrieb: Oh, right. They are in the osm file but not in the element that is being processed at that time. It would make it a tad more complicated to get that data at that point. yep... But what would be against starting with a osm file with a smaller bbox as input? sources for the worldwide edits are the diff from http://planet.openstreetmap.org/hour-replicate/000/003/ in some cases xybot is using country excerpts from http://download.geofabrik.de/osm/europe/ but they are still very large and it takes geofabrik around 7-9 hours to produce them each day. Or uploading single items in seperate changesets? as i wrote before, this would lead changesets ad absurdum frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] xybot edit area size
Hi, I'm not sure if this list is appropriate for this discussion and please point me in the right direction if it is not. Is it possible to change xybot bot script not to use so large areas for edits? It would be ideal if it would use small areas of only few square km around each edit, idea beeing so that xybot edits don't show in history tab when looking at some area that wasn't edited. Cheers! -- pratite me na twitteru - www.twitter.com/valentt blog: http://kernelreloaded.blog385.com linux, anime, spirituality, windsurf, wireless, ronjenje, pametne kuće registered as user #367004 with the Linux Counter, http://counter.li.org. ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Wed, 07 Apr 2010 07:45:58 +, Valent Turkovic wrote: It would be ideal if it would use small areas of only few square km around each edit, idea beeing so that xybot edits don't show in history tab when looking at some area that wasn't edited. Example of area size that xybot uses: http://dl.dropbox.com/u/184632/xybot.png Ideally I would like to see bot edits in area that I'm looking at but not ones 5000km away ;( -- pratite me na twitteru - www.twitter.com/valentt blog: http://kernelreloaded.blog385.com linux, anime, spirituality, windsurf, wireless, ronjenje, pametne kuće registered as user #367004 with the Linux Counter, http://counter.li.org. ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Wed, Apr 7, 2010 at 9:57 AM, Valent Turkovic valent.turko...@gmail.com wrote: On Wed, 07 Apr 2010 07:45:58 +, Valent Turkovic wrote: It would be ideal if it would use small areas of only few square km around each edit, idea beeing so that xybot edits don't show in history tab when looking at some area that wasn't edited. Ideally I would like to see bot edits in area that I'm looking at but not ones 5000km away ;( What about the edit history only showing edits to elementsinside the view box, not just edits that contain view box, Sounds like a feature request for trac. http://trac.openstreetmap.org/ticket/2867 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
jamesmikedup...@googlemail.com schrieb: It would be ideal if it would use small areas of only few square km around each edit, idea beeing so that xybot edits don't show in history tab when looking at some area that wasn't edited. xybot and its siblings have implemented a bot=yes tag in each changeset they create. http://www.openstreetmap.org/browse/changeset/4346988 this could be a way to add a checkbox on the history tab don't show bot edits on the other hand... i think it was an error to add the history tab. the itoworld osmmapper website and it's rss-feeds are a much better way to monitor malicous edits in a given area. maybe this service could also be implemented on the openstreetmap.org website, as the ito osmmapper is frequently offline (as it is at the moment) frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On Wed, 07 Apr 2010 10:41:22 +0200, Frank Sautter wrote: maybe this service could also be implemented on the openstreetmap.org website, as the ito osmmapper is frequently offline (as it is at the moment) I agree we shouldn't depend on 3rd party service for such important feature like watching for malicious activity. If other bots implement bot=yes tag then it would be trivial to filter them, right? But still ideal situation is that bots also use much smaller areas for edits so that you see if some bot has changed something in your area of interest, right? -- pratite me na twitteru - www.twitter.com/valentt blog: http://kernelreloaded.blog385.com linux, anime, spirituality, windsurf, wireless, ronjenje, pametne kuće registered as user #367004 with the Linux Counter, http://counter.li.org. ICQ: 2125241, Skype: valent.turkovic, MSN: valent.turko...@hotmail.com ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] xybot edit area size
On 7 April 2010 21:50, Valent Turkovic valent.turko...@gmail.com wrote: On Wed, 07 Apr 2010 10:41:22 +0200, Frank Sautter wrote: maybe this service could also be implemented on the openstreetmap.org website, as the ito osmmapper is frequently offline (as it is at the moment) I agree we shouldn't depend on 3rd party service for such important feature like watching for malicious activity. If other bots implement bot=yes tag then it would be trivial to filter them, right? I'm using bot=yes on some bigger, bot-like, edits often. But still ideal situation is that bots also use much smaller areas for edits so that you see if some bot has changed something in your area of interest, right? That would be good but on the other hand lots of very similar changes should be grouped together in one changeset for convenience, if our tools can't cope with it then let's fix the tools. Otherwise you might get 10k changesets in on run, each changing a single node, just for the sake of not generating too big bboxes. The ideal situation would be having infinite processing power etc. :) But with the existing limitations, I think the show bot edits check boxes is a fair approximation. Frederik said it wouldn't be a complete solution to the problem of easy monitoring changes in your area, I think, because sometimes the bot edit is just the one you're interested in. But it is a nice feature if we don't claim that it's a solution to this problem. It would be a nice feature on its own. On a different note, Frederik, you said you wouldn't put the history tab on the website and didn't think it was a good thing. Is that precisely because of the bboxes problem (i.e. that the edit needs not intersect the current view, just its bbox), or is there another reason you said that? Cheers ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
