Re: [OSM-dev] Tiles cache and HTTPS

[Stefan Baebler]

In that case you would ideally find a way for the caching proxy server to
reach the original tiles over plain http (make sure to strip any user's
personal data from the requests!) and save some CPU cycles on both ends,
improving the latency slightly.

You might also want to consider serving larger tiles. 256x256 is really
small for today's standards and most major map frameworks (well, at least
leaflet, openlayers and mapbox) already support loading 512x512 tiles,
reducing the number of requests, improving the load times and server disk
usage. Yes, those tiles will take some more time to render, but most likely
less than four times more ;-)

Some rendering/caching systems internally already use even larger tiles to
combat these problems
https://wiki.openstreetmap.org/wiki/Meta_tiles

Br,
Štefan

V ned., 3. nov. 2019 09:24 je oseba Yves  napisala:

> Ah, I should've told that I own the opensnowmap render server.
> My concern is to keep the ~0.5 TB of tiles available on both a proxy cache
> and the render server in case things goes bad on the later, even if only
> old tiles.
> Also, as heavy tiles user are more and more common, I'd like to propose
> them a quick solution that looks better than 'no warranty, I may have to
> cut your access'.
> Yves
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tiles cache and HTTPS

[Stefan Baebler]

Https might not be strictly needed between tile renderxcxer and the cache,
but if this is the only way that you can get to the tiles (eg from the
pucxcxccxczxcxzcxxxblic opensnowmap tile server) you can technically still
use that.

If possible try to use http 2 or at least keep-alive connections.

Also check their tile usage policy.

Br,
Štefan
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Shared borders of (multi) polygons represented as duplicated ways

[Stefan Baebler]

Paul, thanks for fast response! I perfectly understand you, it's normal for
opensource :)

Imports from shapefiles are nothing new and have been going on with various
tools for many years. I wanted to discuss with commuity whether this
situation is anomaly or something tolerable as at least two major OSM tools
(JOSM and ogr2osm)  have this same problem (JOSM is only fixed in
development version), and probably had it for years. Is it dev@ -only issue
or should it be discussed more broadly on talk@ ?

>From the bugfix perspective it should probably be one more step of
merging (eg MergeWays()) after two prior merges (mergePoints()
and mergeWayPoints()).

greets,
Stefan
.

On Fri, Nov 21, 2014 at 6:21 PM, Paul Norman  wrote:

> On 11/21/2014 8:58 AM, Stefan Baebler wrote:
>
>> Is ogr2osm being actively maintained?
>>
> Yes - but I don't have time to add new features like #28. I'd welcome a PR
> doing so, but have no ETA of when I would get to it.
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Shared borders of (multi) polygons represented as duplicated ways

[Stefan Baebler]

Hi!

While preparing a dataset for a community import (
https://wiki.openstreetmap.org/wiki/Slovenia_Landcover_Import_-_RABA-KGZ )
we ran into 2 kinds of related problems that originate from Shapefile's
different representation of polygons:

a) duplicated ways: when no tags are present on two overlaping ways, but
ways are members of different relations JOSM reports this as an error. This
is common mostly with nested multipolygon relations, where one relation's
outer way is the other's inner member.

b) Ways on same position: this is similar to previous, but some tags
differ, making the match not exact. JOSM reports this as warning. This is
more common, because it doesn't require that many levels of nesting, A
simple meadow in a multipolygon forest is enough.

The illustraded description is in the issue of ogr2osm (my first choice for
preparing the dataset):
https://github.com/pnorman/ogr2osm/issues/28 (not fixed yet).

Vincent promptly fixed the JOSM's OpenData plugin for importing shapefiles
and adjusted validation logic: https://josm.openstreetmap.de/ticket/10743 ,
so we might go this way (shp -> JOSM instead of ogr2osm -> JOSM or any
other editor), but it will require adjusting the data preparation scripts,
and more importantly, relying on the end users (community importers) for
using the latest (development) builds of JOSM with latest OpenData plugin
to fix the data during the import.

While editing existing OSM data I noticed that this is quite common problem
in OSM database, even if not coming in via imports (manual edits).

My questions are:
Are duplicated ways considered erronous and to be avoided at all costs or
are they tolerable?
Are any tools (osm inspector and similar) detecting these problems?
Are there any plans to fix such topological errors (if considered as such)
in existing data by bots (automated edits)?
Is ogr2osm being used for other imports with similar problems?
Is ogr2osm being actively maintained?

thanks,
Stefan
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Any OpenStreetMap viewer for Android?

[Stefan Baebler]

There a page in wiki:
http://wiki.openstreetmap.org/wiki/Android
that lists also other OSM software for Android.

Stefan

On Mon, Sep 28, 2009 at 11:21 PM, Stefan Keller  wrote:
> Many thanks for all the hints so far.
> Regarding Navit I did not found any sources on the SVN and it's far
> from being easy to install.
> Vespucci declares itself explicitely as NOT being "a map-view or even
> a routing-application". And it failed to work on my G1.
> Remains actually osm-android (http://code.google.com/p/osm-android/)
> and eventually AndNav2 of which we'll have a closer look.
>
> Yours, S.
>
> 2009/9/28 Eric Marsden :
>>> "sk" == Stefan Keller  writes:
>>
>>  sk> Am I right that there's no OpenStreetMap viewer software available for
>>  sk> Android-based phones - at least no open source except AndNav2?
>>
>>  There is an experimental port of Navit, which does on-device vector
>>  rendering and routing with OSM data. It works reasonably well in my
>>  limited testing so far.
>>
>>    http://wiki.navit-project.org/index.php/Navit_on_Android
>>
>>
>>  There is also an experimental on-device editor called Vespucci
>>
>>     http://www.cyrket.com/package/de.blau.android
>>     http://code.google.com/p/osmeditor4android/wiki/Overview
>>
>>   which writes directly to the API.
>>
>>   With the multiple excellent logging applications that are available,
>>   Android is a nice mobile platform for OSM!
>>
>> --
>> Eric Marsden
>>
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/dev
>>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] cruisecontrol needs intervention

[stefan . baebler]

build log at
http://cruise.openstreetmap.org/builds/osmapi-postgres
currently says:
**Running the tests** Run "rake db:migrate" to update your database then try 
again.
(in /srv/.cruise/projects/osmapi-postgres/work) 
You have 3 pending migrations: 
   40 CreateOauthTables 
   41 AddFineOAuthPermissions 
   42 AddForeignKeysToOauthTables 


not sure who can do this and why this isn't also automated (as it is always 
building & testing the latest HEAD version anyway and DB needs to be prepared 
before testing)

Stefan

signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Variable scale maps

[Stefan Baebler]

I  believe Kelly was interested in static maps (eg paper maps) that
cannot be re-rendered every second.
Instead of providing map insets with higher detail it would be
interesting to have a "fisheye" style map, which would cover large
areas, but still provide high details in the centre or some other
area(s) of interest.

Google has some examples to illustrate the idea:
http://images.google.si/images?q=fisheye+map

"3d map" view on navigation devices is similarly showing detailed map
of immediate surroundings, but showing low scale map only in the
direction of travel.

Maybe areas of interest could be determined by node density instead of
having to define them manually, which brings me to an idea of
elastically joining nodes with springs of equal length and see how
they settle. This might be computationally very demanding and yielding
strange results if an unconnected straight way traverses highly and
densely connected area (eg underground under a city centre, straight
administrative borders...). Some average node density would probably
be simpler and giving smoother maps.

Stefan

On Tue, Jul 14, 2009 at 4:43 AM, Marcus Wolschon wrote:
> On Tue, Jul 14, 2009 at 4:30 AM, Kelly Jones 
> wrote:
>> Most maps of small areas have a constant scale: eg: 100 pixels = 1 mile.
>>
>> Has anyone created maps w/ variable scales?
>
> Yes, all the time.
> That's interactive rendering in the client.
> Every satnav does that.
>
> Marcus
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OAuth

[Stefan Baebler]

> please DO NOT revert my work without talking to me first. OSM doesn't
> stop just because you're on holiday.
+1
Do not get me wrong. Tom undoubtedly puts in a lot of effort and does
great work. However, if the huge community relies on a single person
to do all code checks before deploying it, and this cannot be (even
temporarily, during planned vacations) delegated to anyone else
(skilled and trustworthy enough), we have a continuity problem that
needs to be addressed. Tom deserves better holidays.

Stefan
/disclaimer: I did a similar thingy a week ago, having a slight clash
with Tom, so my view probably isn't perfectly unbiased. But that gave
me some time to think about it.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [osmosis-dev] Licence ...

[Stefan Baebler]

I consider my contributions so minor that i hereby give you all the
power to relicense them as you please.

Stefan
/trying to shorten possibly never-ending license debate :)

Brett Henderson wrote:
> I hate to bring this topic up because I don't know if I have the 
> necessary patience for it :-)
> 
> Osmosis is currently GPL3.  At one point I was going to use GPL2 or 
> later but the use of the Apache bzip2 implementation precludes that.
> 
> The number of authors is fairly low at the moment so there is still a 
> chance to change the licence.
> 
> At this point I'm wishing I'd just released it as public domain in the 
> first place.  I don't intend to make money from it, I don't care what 
> people do with it, and the recent discussions on legal-talk have 
> convinced me that any gains that a copyleft licence might achieve are 
> dwarfed by the problems that licence compatibility causes.  The use of 
> bzip2 libraries has already forced my hand in one direction, and it's 
> likely to only get worse.  Osmosis is a piece of plumbing that is most 
> useful if it can be used anywhere.
> 
> My motivation for creating osmosis in the first place was simply that my 
> day job no longer had a technical component and I wanted a hobby 
> software project to tinker with.  It quickly grew into a bigger time 
> sink than I originally planned but my reasons for involvement are still 
> that it is a hobby.  I get more satisfaction out of seeing people using 
> something I've built than I do out of any recognition for being the 
> author.  At this point my time is becoming more limited so I see my role 
> becoming less central but I'd like to make sure that everything 
> including the licence status is robust before I drop anything.
> 
> So, what are people's thoughts?  There are approximately half a dozen 
> contributors so far who I can mail separately if required (or name them 
> on this list, not sure what the etiquette is here).  With one exception 
> (initial 0.6 support) most patches have been fairly self contained and 
> could be replaceable if required.  If there's no major arguments I'll 
> send the existing authors an email over the next few days asking 
> permission to release all software as public domain.  If that goes 
> smoothly then the existing licence text can be removed and replaced with 
> a public domain dedication statement of some kind.
> 
> Brett
> 
> 
> ___
> osmosis-dev mailing list
> osmosis-...@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/osmosis-dev
> 




signature.asc
Description: OpenPGP digital signature
___
osmosis-dev mailing list
osmosis-...@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [OSM-dev] trace file extension in links to data

[Stefan Baebler]

My previous patch probably conflicts with
http://trac.openstreetmap.org/ticket/534 and
http://trac.openstreetmap.org/changeset/6562

which allows compressed (eg .gpx.gz) data usually served by
http://www.openstreetmap.org/trace/137465/data
to be served as uncompressed gpx xml if requested as
http://www.openstreetmap.org/trace/137465/data.xml

Why .xml extension instead of simply .gpx?
Can we link that in the trace detail page or is there a good reason for
not doing so?

Stefan

Stefan Baebler wrote:
> Hi!
> 
> It is a nice practice to indicate the file type in the url of the
> download link to let users know what they are getting before actually
> pushing it down their throat.
> 
> Filename on the trace detail page often says "somecity.gpx" (as at the
> time of upload), download link is simply "data" and the resulting file
> is named "123456.gpx.gz" (gzipped later on server?), which makes it a
> bit confusing.
> 
> Example:
> http://www.openstreetmap.org/user/Rigless/traces/308275
> has a download link to
> http://www.openstreetmap.org/trace/308275/data
> my aim is to change that to
> http://www.openstreetmap.org/trace/308275/data.gpx
> 
> and similarly at
> http://www.openstreetmap.org/user/StefanB/traces/137465
> has the link:
> http://www.openstreetmap.org/trace/137465/data
> that should be:
> http://www.openstreetmap.org/trace/137465/data.gpx.gz
> 
> I'm attaching an *UNTESTED* patch for above changes in the rails port.
> 
> Going a step further would be to match also the base filename, not just
> extensions:
> Eg instead:
> http://www.openstreetmap.org/trace/308275/data
> link would be
> http://www.openstreetmap.org/trace/308275/308275.gpx
> or even just
> http://www.openstreetmap.org/trace/308275.gpx
> 
> 
> I couldn't find traces with other extensions (.tar.gz, .zip, .gpx.bz2
> ...) for additional test cases, but it should be easy by peeking into
> database or into folder full of traces. Or having some faith that if it
> works for two types it should work for all of them.
> 
> Related bug: http://trac.openstreetmap.org/ticket/498 - there the
> extension ".gpx" was probably hardcoded into the link.
> 
> Can someone please apply the patch into their test rails setup and test
> it before committing to svn and pushing it to the main site please?
> 
> thank you!
> Stefan
> 
> 
> 




signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] trace file extension in links to data

[Stefan Baebler]

Hi!

It is a nice practice to indicate the file type in the url of the
download link to let users know what they are getting before actually
pushing it down their throat.

Filename on the trace detail page often says "somecity.gpx" (as at the
time of upload), download link is simply "data" and the resulting file
is named "123456.gpx.gz" (gzipped later on server?), which makes it a
bit confusing.

Example:
http://www.openstreetmap.org/user/Rigless/traces/308275
has a download link to
http://www.openstreetmap.org/trace/308275/data
my aim is to change that to
http://www.openstreetmap.org/trace/308275/data.gpx

and similarly at
http://www.openstreetmap.org/user/StefanB/traces/137465
has the link:
http://www.openstreetmap.org/trace/137465/data
that should be:
http://www.openstreetmap.org/trace/137465/data.gpx.gz

I'm attaching an *UNTESTED* patch for above changes in the rails port.

Going a step further would be to match also the base filename, not just
extensions:
Eg instead:
http://www.openstreetmap.org/trace/308275/data
link would be
http://www.openstreetmap.org/trace/308275/308275.gpx
or even just
http://www.openstreetmap.org/trace/308275.gpx


I couldn't find traces with other extensions (.tar.gz, .zip, .gpx.bz2
...) for additional test cases, but it should be easy by peeking into
database or into folder full of traces. Or having some faith that if it
works for two types it should work for all of them.

Related bug: http://trac.openstreetmap.org/ticket/498 - there the
extension ".gpx" was probably hardcoded into the link.

Can someone please apply the patch into their test rails setup and test
it before committing to svn and pushing it to the main site please?

thank you!
Stefan


Index: app/views/trace/view.rhtml
===
--- app/views/trace/view.rhtml  (revision 13565)
+++ app/views/trace/view.rhtml  (working copy)
@@ -9,7 +9,7 @@
 
   
 Filename:
-<%= @trace.name %> (<%= link_to 'download', :controller => 'trace', 
:action => 'data', :id => @trace.id %>)
+<%= @trace.name %> (<%= link_to 'download', :controller => 'trace', 
:action => 'data', :id => @trace.id, :format => @trace.extension_name %>)

   
 Uploaded at:
Index: config/routes.rb
===
--- config/routes.rb(revision 13565)
+++ config/routes.rb(working copy)
@@ -108,7 +108,7 @@
   map.connect '/traces/mine/tag/:tag/page/:page', :controller => 'trace', 
:action => 'mine'
   map.connect '/trace/create', :controller => 'trace', :action => 'create'
   map.connect '/trace/:id/data', :controller => 'trace', :action => 'data'
-  map.connect '/trace/:id/data.:format', :controller => 'trace', :action => 
'data'
+  map.connect '/trace/:id/data:format', :controller => 'trace', :action => 
'data'
   map.connect '/trace/:id/edit', :controller => 'trace', :action => 'edit'
   map.connect '/trace/:id/delete', :controller => 'trace', :action => 'delete'
   map.connect '/trace/:id/make_public', :controller => 'trace', :action => 
'make_public'


signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Altitude data

[Stefan Baebler]

Some nodes have altitiude (or alt) tag if the altitude is well known
(mountain peaks etc) as vertical precision of GPS is so inaccurate.

You can convert srtm data into OSM vector contours with
http://wiki.openstreetmap.org/wiki/Srtm2Osm

A general page about visualizing relief (including other techniques,
such as shading etc):
http://wiki.openstreetmap.org/wiki/Relief_maps

hope it helps,
Stefan

Philip Fitzsimons wrote:
> Hi,
> 
> In terms of altitude information, am I correct in thinking that integrating 
> the SRTM data (http://www2.jpl.nasa.gov/srtm/) is the
> accepted approach to creating altitude based maps?
> 
> I have seen key="altitude" in some of the OSM data, but not comprehensively. 
> Is there a SRTM enhanced version of the OSM data? or an
> XML version of the SRTM data?
> 
> I'm using it to create vector based maps 
> (http://blog.figmentengine.com/search/label/OpenStreetMap) in silverlight and 
> just wanted
> to know what was the "accepted" way to incorporate altitude information.
> 
> Thanks in advance
> 
> Philip
> (Figment Engine)
> 
> 
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
> 


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [josm-dev] JOSM language plugins and stable version?

[Stefan Baebler]

On Tue, Nov 4, 2008 at 8:28 AM, Dirk Stöcker <[EMAIL PROTECTED]> wrote:
> On Sun, 2 Nov 2008, Florian Schmitt wrote:
>
>> it seems that the lang-* plugins aren't available any more. This means that 
>> new
>> users who want to use the "last stable version" 1010 aren't able to change 
>> the
>> language of JOSM. I think as long as a "stable" version is recommended, the
>> localisation plugins should be available, even if they're not necessary for 
>> the
>> latest build. Would it be possible to provide the lang-Plugins again?
>
> I updated the "stable" to 1065.

but this "stable" is also missing translations until build script is
adjusted so that the jar includes translations.

Stefan
___
josm-dev mailing list
[EMAIL PROTECTED]
http://lists.openstreetmap.org/listinfo/josm-dev

Re: [OSM-dev] Spatial vs. multi-column indexes for points

[Stefan Baebler]

Instead of indexing by precise coordinates you could index by virtual
tiles as it is done in OSM's main DB since a year ago with nice
performance boost:

http://wiki.openstreetmap.org/index.php/QuadTiles

good luck,
Stefan


On Thu, Sep 11, 2008 at 1:49 PM, Andreas Kalsch <[EMAIL PROTECTED]> wrote:
> Hey,
>
> last week I made some experiments with huge datasets of lat/lon points. I use 
> MySQL 5.0, which partially support GIS extensions, including R-trees. But it 
> is still not able to make queries based on the GIS features, so I have to use 
> the normal way - multi-column indexes on lat/lon columns. It works well but 
> probably there is a way to make it even quicker ;)
>
> Has anybody used GIS successfully in MySQL or PGSQL and can tell me how the 
> performance compares between the two techniques?
>
> Thanks,
>
> Andi
> --
> Pt! Schon das coole Video vom GMX MultiMessenger gesehen?
> Der Eine für Alle: http://www.gmx.net/de/go/messenger03
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Potlatch causing DB inconsistency?

[Stefan Baebler]

Tnx for the confirmation.
I guess 0.6 API aims to fix that with atomic uploads, but Potlatch
will remain using its own "API", right? Is it now (version 0.10)
preventing this somehow?

so the events went something like:
- user adjusts the way somehow [common operation]
- potlatch decides to delete old way and create a new one for some
reason [stupid (breaking history), but legal]
- potlatch deletes old way and all the nodes it can (those that aren't
used by other ways where other rivers join the riverbank) [stupid, but
ok]
- potlatch creates a new way, consisting of old (now mostly deleted)
nodes [illegal]

Not really sure this has anything to do with missing transactions,
more likely some bad logic. Foreign keys in the db on current_ways ->
current_nodes could prevent that, but AFAIK they aren't used for
performance reasons.

But why is potlatch today STILL showing the riverbank as it would be
there, but in fact it isn't?
If this accident would be visible to the user then he could do
something about it (re-draw it in worst case) immediately.

Stefan

-- Forwarded message --
From: Richard Fairhurst <[EMAIL PROTECTED]>
Date: Tue, Jul 22, 2008 at 9:52 AM
Subject: Re: [OSM-dev] Potlatch causing DB inconsistency?
To: dev@openstreetmap.org


Stefan Baebler wrote:

> Any ideas what's going on?

I think it's reasonably well attested that, without transactions, this
sort of thing is going to happen once in a while. As posted the other
day, Potlatch's database access code has been rewritten almost beyond
recognition for 0.10 (see http://tinyurl.com/5996mn) so there's not a
whole lot of point filing bugs against 0.9c; let me know if you see
anything like this happen with 0.10, of course.

cheers
Richard


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

[OSM-dev] Potlatch causing DB inconsistency?

[Stefan Baebler]

Hello, Huston! :)

http://www.openstreetmap.org/?lat=46.04702&lon=14.52213&zoom=15&layers=0B0FTF
shows a large watery triangle and same can be seen in JOSM (nad navit,
where i originally came across this new lake :).
When i asked the local user mihau, who did the last change he pointed me
to Potlatch where the riverbank looks perfect.

There is something fishy with the way 25397686 as returned by API:
http://www.openstreetmap.org/api/0.5/way/25397686/history (latest
version has many node here)
http://www.openstreetmap.org/api/0.5/way/25397686 (only 7 nodes!)
One of the missing nodes seems to be (unintentionally) deleted:
http://www.openstreetmap.org/api/0.5/node/175826598/history
but potlatch seems to show them, since they are included in the way.

Yep, deleted nodes are included in the visible way. History api call
shows all the nodes in a way, but regular api call includes just visible
nodes (forming a triangle). Potlatch seems to show also deleted nodes.
Potlatch deleted some nodes which it shouldn't and is showing them as
not deleted.

>From my local planet extracts i found the riverbank which existed before
potlatch decided to create a new one:
http://www.openstreetmap.org/api/0.5/way/16956741/history

User mihau was doing just minor adjustments to the riverbank (certainly
not deleting it or its nodes) when all this happened.

Any ideas what's going on?
Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] Launchpad OSM Team

[Stefan Baebler]

Excellent, I'm in!

Can it be linked somehow to the actual OSM projects, such as:
https://launchpad.net/josm
?

Stefan


On Sun, Jun 29, 2008 at 6:40 PM, Milo van der Linden
<[EMAIL PROTECTED]> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hello Wolfgang,
>
> Excellent move!
>
> I signed up with the team.
>
> I hope to be able to get in touch with bvh soon to see if we can move a
> stable version of Merkaartor to launchpad.
>
> kind regards,
>
> Milo van der Linden
>
> Wolfgang Silbermayr wrote:
>> Hello!
>> I founded an openstreetmap team on [0]. The goal of the team is to make
>> it easy to use ubuntu for any use of openstreetmap. Therefore, it would
>> be the main work to create and maintain packages of programs like JOSM,
>> pyroute, navit, tangogps and so on.
>> If you would like to help out, please register for the team on [0]. If
>> you have any ideas what I could improve concerning the team, please let
>> me know too here on the dev-list.
>>
>> Greetings, Wolfgang.
>>
>> ---
>>
>> [0] http://launchpad.net/~openstreetmap
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>>
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFIZ7rwFHMFRcTlnzoRAtpdAJ479qHNuCA35FRlzNqgQso9UNVFRQCgg4x8
> bE4Ey8Mfnm+6B/R7NT1MVwY=
> =jU2i
> -END PGP SIGNATURE-
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] WMS

[Stefan Baebler]

Iván Sánchez Ortega wrote:
> El Miércoles, 25 de Junio de 2008, 80n escribió:
>>> So, I'd like to know if you think it would be a good idea to do load
>>> balancing between several OSMXAPI servers.
>> This is probably a good idea.  Do you have any servers in mind?
> 
> Well, for what I've read in the Wiki about OSMXAPI, I only know about 
> Informationfreeway. Is there a complete list of OSMXAPI servers?

AFAIK you just wrote the complete list :)

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] Question about .bz2 file parsing

[Stefan Baebler]

My command line for extracting a bounding box can serve as example:

java -jar "Osmosis\osmosis.jar" --rx "slovenia.osm.bz2" 
enableDateParsing="yes" --lp interval="600" --bb left="14.4" 
right="14.6" top="46.15" bottom="46" --wx "ljubljana.osm.bz2"

--rx = read xml (your input.osm.bz2)
--lp = log progress
--bb = bounding box limits (adjust coordinates)
--wx = write xml (adjust result filename)

it can take several hours to parse whole planet, so you might want to 
start with one of the larger extracts (continent or country).

Stefan

Fire Girl wrote:
> Wow, that is excellent.  I had this right under my nose and didn't 
> know it!
>
> I installed Java 5 on my Windows XP 64 bit machine, and then the Java 
> 5 version of Osmosis.  It took me a moment to re-orient myself but 
> I think I made a little progress but have a slight knowledge gap 
> how to parse commands with it.
>
> If I put the Java 5 version of Osmosis in a directory, and then run 
> this command from a prompt,
>
> java -jar osmosis.jar
>
> I get back:
>
> INFO: Osmosis Version 0.24.1-java5
> INFO: Preparing pipeline
> INFO: Launching pipeline execution
> INFO: Pipeline executing, waiting for completion.
> INFO: Pipeline complete.
>
> ... but I am really unsure where to go from here, and how to actually 
> run Osmosis commands as presented at that Wiki page? :-)
>
> Could you please point me in the right direction, if possible? :-)
>
>
> - Original Message -
> From: "Stefan Baebler"
> To: "Fire Girl"
> Subject: Re: [OSM-dev] Question about .bz2 file parsing
> Date: Sat, 14 Jun 2008 07:35:32 +0200
>
>
> Try Osmosis. It's written in Java and works nicely also on windows.
> It can read osm.bz2 and can extract bounding boxes.
>
> http://wiki.openstreetmap.org/index.php/Osmosis
>
> Stefan
>
> Fire Girl wrote:
> >
> >
> > Greetings
> >
> > But ok, are there any Windows tools for parsing a decompressed
> > OSM file for a Country into a smaller OSM file based on a
> > bounding box input? :)
> >
> > - Original Message -
> > From: "Iván Sánchez Ortega"
> > To: dev@openstreetmap.org
> > Subject: Re: [OSM-dev] Question about .bz2 file parsing
> > Date: Sat, 14 Jun 2008 05:35:15 +0200
> >
> >
> > El Sábado, 14 de Junio de 2008, Fire Girl escribió:
> > > Hello, can someone be so kind to point the way to any Windows
> based
> > > tools, that will parse a .bz2 file directly,
> >
> > None! You have to decompress the file first!!!
> >
> > Use 7-zip/Winzip/winrar, or whatever tool you like the most. A
> > graphical tool
> > may clog on files > 4GB, so please also try with a command-line
> > bunzip2 tool.
> >
> >
> > Cheers,
> > --
> > --
> > Iván Sánchez Ortega
> >
> > Now listening to: Sugababes - Three (2003) - [10] Too Lost in You
> > (3:56)
> > (96.00%)
> > << signature.asc >>
> >
> >
> > -- See Exclusive Videos: * 10th Annual Young Hollywood Awards*
> >
> >
> 
> >
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
> >
>
>
> -- 
> See Exclusive Videos: * 10th Annual Young Hollywood Awards* 
> <http://link.brightcove.com/services/player/bcpid1403465002?bclid=1527697154&bctid=1531204364>
> 
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>   


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] Question about .bz2 file parsing

[Stefan Baebler]

Try Osmosis. It's written in Java and works nicely also on windows. It 
can read osm.bz2 and can extract bounding boxes.

http://wiki.openstreetmap.org/index.php/Osmosis

Stefan

Fire Girl wrote:
>
>
> Greetings
>
> But ok, are there any Windows tools for parsing a decompressed OSM 
> file for a Country into a smaller OSM file based on a bounding box 
> input? :)
>
> - Original Message -
> From: "Iván Sánchez Ortega"
> To: dev@openstreetmap.org
> Subject: Re: [OSM-dev] Question about .bz2 file parsing
> Date: Sat, 14 Jun 2008 05:35:15 +0200
>
>
> El Sábado, 14 de Junio de 2008, Fire Girl escribió:
> > Hello, can someone be so kind to point the way to any Windows based
> > tools, that will parse a .bz2 file directly,
>
> None! You have to decompress the file first!!!
>
> Use 7-zip/Winzip/winrar, or whatever tool you like the most. A
> graphical tool
> may clog on files > 4GB, so please also try with a command-line
> bunzip2 tool.
>
>
> Cheers,
> --
> --
> Iván Sánchez Ortega
>
> Now listening to: Sugababes - Three (2003) - [10] Too Lost in You
> (3:56)
> (96.00%)
> << signature.asc >>
>
>
> -- 
> See Exclusive Videos: * 10th Annual Young Hollywood Awards* 
> 
> 
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>   


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] 0.6 - older trackpoints /was: Re: 0.6

[Stefan Baebler]

Dirk-Lüder Kreie wrote:
>>>  > I propose to introduce the ability to limit the age of trackpoints
>>>  > downloaded into the client by introducing an extra parameter "age",
>>>  > eg:
>>>  >
>>>  > GET 
>>> /api/0.6/trackpoints?bbox=left,bottom,right,top&age=ageInDays&page=pageNumber
>>>
>>>  +1! Although it might be easier on the database if the age is
>>>  replaced by a cutoffdate instead?
>>>   
> Can we fix this to whole days, and make it use the calendar day, instead
> of the last 24 hour period? Otherwise you might get to know which track
> was created exactly when by repeatedly downloading an area and watching
> the differences
As said the time would be the time when the trace was imported into the 
db, so this wouldn't give out personal moving patterns, but time of  
upload patterns (delayed by the time gpx spends in the queue), which 
isn't such an issue i believe. Think of a situation when someone uploads 
7 month old traces and users choose to see only the last 6 months worth 
of traces. Such traces would still be visible for 6 months after being 
uploaded to allow some time for them to be traced.

But it makes sense that such precision (with decimals) is not needed.
Whole days is good enough, heck, the age unit could also be weeks or 
even months to further lower the precision to provide some more 
obfuscation for traces that might not be public (linked to user and 
downloadable as raw gpx), but i don't think this level of additional 
privacy would be necesarry. Midnight (GMT?) seems ok for a cutoff step 
(in case of weeks it would need to be a fixed day of week and in case of 
month a fixed day in month, which could be strange to get used to)

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] 0.6 - typo-cleaning

[Stefan Baebler]

>  | The api change would be a good time to clean out most of the typos.
>  | with the node_tags, way_tags and relation_tags table (sorry i've completly
>  | forgotten these above),  they are allready separated.

>  how do you suppose to do that?
>  map_features and proposed_tags are not all the valid tags
>  that people are using on the map.

i guess some statistics of tag use already exists (tagwatch)
If the statistics shows
highway: 54284238754
higway:12
...

one can reasonably and safely assume that "higway"s are typos and can
be changed to highway - preferably manually, after reviewing other
tags of said objects, eg
higway=residential can be safely changed, but
higway=hig (making it up as i go :) ) cannot be changed as it isn't
clear if author made a mistake or is it on purpose.

It is fairly easy to find such nodes with osmxapi, but very hard (next
to impossible) with the main api.

A limping workaround would be to make an application that finds such
objects using osmxapi, and then allows their editing (by id) within
the main api.

A better solution would be to allow some searching in the main api,
but that opens up another box of worms (namefinder issues, performance
issues, apps using main api instead of osmxapi...).

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

[OSM-dev] 0.6 - older trackpoints /was: Re: 0.6

[Stefan Baebler]

On Tue, May 6, 2008 at 10:25 AM, SteveC <[EMAIL PROTECTED]> wrote:
>  is there anything else that should be put in 0.6? Probably a good idea
>  to talk about it now.

As project is getting older there are many relatively old GPS tracks in the db.
Older tracks are probably already mapped, or out of date (road network
does change over longer periods), or less accurate (GPS receivers are
improving over time, reducing the errors).
Motorways near bigger cities are sometimes recorded way too often as
mapper's ways converge on their way to/from home.

Having all the ancient trackpoints downloaded into editor can make it
difficult to work with because of the sheer quantity and because it is
can be hard to map recent changes in the road network.

I propose to introduce the ability to limit the age of trackpoints
downloaded into the client by introducing an extra parameter "age",
eg:

GET 
/api/0.6/trackpoints?bbox=left,bottom,right,top&age=ageInDays&page=pageNumber

To keep it simple age parameter would be in days (can have decimals
for finer precision)
To give tracks such time to be mapped age would be calculated from the
date of trace upload, not the age of individual trackpoint (unless
there is a technical or performance reason to do the opposite).

Editors could choose to use it or not. If they do implement such
filtering there should be an easy way to show all, also older
trackpoints (eg a checkbox in the download dialog).
API might introduce some reasonable default (eg 2 years or longer).

A simpler workaround (with no API change) would be to sort the
trackpoints in the existing api from most recent to oldest and clients
can request as many pages as needed, allowing editors to easily ask
for more (older) trackpoints as needed by requesting additional pages
(and limit first batch to eg 20 pages).

any thoughts?

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] slippymap doesnt wrap around +-180deg

[Stefan Baebler]

On Fri, Apr 11, 2008 at 11:46 AM, Tom Hughes <[EMAIL PROTECTED]> wrote:
>  > Should be fairly easy to fix that with openlayers:
>  > http://openlayers.org/dev/examples/wrapDateLine.html
>
>  I don't think so. The tiles look like they are wrapping round just
>  fine at the moment - the problem is that the name crosses the tile

Actualy no, it doesn't wrap around 180 at the moment.
See the bug on URL initially provided by Luka
http://www.openstreetmap.org/?lat=-41.47&lon=-179.23&zoom=6
and try moving the map to the right (seeing more to the left).

>  boundary and the renderer hasn't rendered the other half of the
>  name on the other tile I think.
That is another, totally separate rendering problem.

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] slippymap doesnt wrap around +-180deg

[Stefan Baebler]

Should be fairly easy to fix that with openlayers:
http://openlayers.org/dev/examples/wrapDateLine.html

Stefan

2008/4/3 Dirk-Lüder Kreie <[EMAIL PROTECTED]>:
> -BEGIN PGP SIGNED MESSAGE-
>  Hash: SHA1
>
>  Luka Frelih schrieb:
>
> > hello!
>  >
>  > i just noticed that our slippymap resists crossing the 180 degree
>  > meridian (intl date line).
>  > try zooming in on new zealand at this link:
>  > http://www.openstreetmap.org/?lat=-41.47&lon=-179.23&zoom=6
>  >
>  > i guess this is unintended. not sure whether we inherited this bug from
>  > openlayers or introduced it ourselves. either way i guess someone here
>  > will know what's going on.
>  >
>
>  Placename truncated at 180° meridian
>  http://www.openstreetmap.org/?lat=-8.7&lon=179.999&zoom=5&layers=B0FT
>
>  also giving lon=180 to slippymap incorrectly jumps to 0° longitude.
>
>  - --
>
>  Dirk-Lüder "Deelkar" Kreie
>  Bremen - 53.0952°N 8.8652°E
>
>  -BEGIN PGP SIGNATURE-
>  Version: GnuPG v2.0.7 (GNU/Linux)
>  Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
>  iD8DBQFH9BHOFUbODdpRVDwRAgGgAJoCv3w/4YhBPLt2fMsAWBR7lFYwfgCfekZD
>  2Hw8f8uO7OBUUSTn0zYdTxs=
>  =J7GB
>  -END PGP SIGNATURE-
>
>
> ___
>  dev mailing list
>  dev@openstreetmap.org
>  http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] slippymap doesnt wrap around +-180deg

[Stefan Baebler]

John, when going West on this map on the given URL the longitude -179 
(179W) should become +179 (179E), but it rolls over to +91979, giving 
the url:
http://johnmckerrell.com/map/#t=mapnik&lat=-41.86956&lon=91979.537&zf=7
Better, but not exactly perfect either :)

Stefan

John McKerrell wrote:
> Try this instead:
>
> http://johnmckerrell.com/map/#t=mapnik&lat=-41.96766&lon=-179.73632&zf=7 
> 
>
>
> On 2 Apr 2008, at 14:49, Luka Frelih wrote:
>> hello!
>>
>> i just noticed that our slippymap resists crossing the 180 degree 
>> meridian (intl date line).
>> try zooming in on new zealand at this link: 
>> http://www.openstreetmap.org/?lat=-41.47&lon=-179.23&zoom=6 
>> 
>>
>> i guess this is unintended. not sure whether we inherited this bug from 
>> openlayers or introduced it ourselves. either way i guess someone here 
>> will know what's going on.
>>
>> best,
>> LF
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org 
>> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
> 
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>   


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] user display names added to planet dump

[Stefan Baebler]

Has this change anything to do with today's planet unavailability / delay?
Planet .gz file is normally available early in the morning, but
nothing so far today (except for daily and hourly change files made by
Osmosis).

Problems escaping some usernames or just different scheduling perhaps?

Stefan

On Sat, Mar 22, 2008 at 1:41 PM, Jon Burgess <[EMAIL PROTECTED]> wrote:
> Next weeks planet dump will include the user display name for those
>  users that have made their edits public. e.g.
>
>timestamp="2007-01-29T08:48:14Z" user="80n"/>
>timestamp="2006-11-04T18:15:03Z" user="RalfZ">
> 
> 
>   
>timestamp="2005-04-18T14:12:45Z" user="sxpert"/>
>timestamp="2005-12-01T12:32:02Z"/>
>
>
>  Jon
>
>
>
>  ___
>  dev mailing list
>  dev@openstreetmap.org
>  http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osm2pgsql.exe

[Stefan Baebler]

Good morning!

I've put it aside this for the moment, but not before coming to
conclusion that osm2pgsql.exe is always throwing this (deceiving
authentication problem) message regardless of the real connection
problem (eg socket not open). Or so it seems.
Hope it helps and sorry for late reply.

Stefan


On Feb 19, 2008 8:06 AM, Lauri Hahne <[EMAIL PROTECTED]> wrote:
> On 19/02/2008, Jon Burgess <[EMAIL PROTECTED]> wrote:
> > All the systems that I have used osm2pgsql on have utilised the "Ident
> > authentication over local sockets"[1]. The comments in the Postgres
> > description imply that you can not use this on Windows. I know some
> > people are definitely using osm2pgsql.exe successfully. I don't know how
> > they've set it up, maybe they have used the permissive 'trust'
> > authentication option. Maybe you can try this or wait until a new
> > version os osm2pgsql.exe is produced from the latest SVN code.
>
> Trust authentication works. It allows you to connect to db without a
> password. Just remember to change passwords back on after importing
> and also remember that your postgres username must match your Windows
> username.
>
>
> --
> Lauri Hahne
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osmosis bz2 performance

[Stefan Baebler]

Martijn van Oosterhout wrote:
> On Feb 7, 2008 2:01 PM, Stefan Baebler <[EMAIL PROTECTED]> wrote:
>>>> Would "-" have a meaning somewhere else? read xml change, write xml
>>>> ... all file operations perhaps?
>>> Yep, ideally this should be done for all tasks with a file argument.
>> It still won't be possible to use 2 input streams (eg planet.osm +
>> planet.osc) on stdin, but at least one (probably bigger source) as it
>> was now with /dev/stdin. Named pipes could solve that, but AFAIK they
>> are not supported in Java. Sockets could work, but are not as
>> convenient for scripting.
> 
> Actually, named pipes would work, since osmosis doesn't seek. Something like:
> 
> mkfifo /tmp/buffer
> bzcat x.bz2 >/tmp/buffer &
> osmosis  --read /tmp/buffer
> rm /tmp/buffer
> 
> Should do fine.
interesting, tnx!
This could be a workaround for the newly introduced aliases.

I've added the "-" alias to the other xml file reading and writing tasks:
http://trac.openstreetmap.org/changeset/6844
Now it is possible to do
wget -O - http://... |  bzcat -c | osmosis --rx - --bb ... --wx - | 
bzip2 > excerpt.osm.bz2
which performs considerably better than Apache's bzip implementation in 
Java, which is built into osmosis.

Enjoy,
Stefan


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osmosis bz2 performance

[Stefan Baebler]

> > Would "-" have a meaning somewhere else? read xml change, write xml
> > ... all file operations perhaps?
> Yep, ideally this should be done for all tasks with a file argument.
It still won't be possible to use 2 input streams (eg planet.osm +
planet.osc) on stdin, but at least one (probably bigger source) as it
was now with /dev/stdin. Named pipes could solve that, but AFAIK they
are not supported in Java. Sockets could work, but are not as
convenient for scripting.

Khm, how about reading http URLs directly ? :)
...And writing to ftp locations (eg to your car when in wifi range) :))

> The wiki will need to be updated eventually but I only do that once I
> release a new version.  That may not happen for a little while because
> I'm about to take three weeks off ...
I envy you! Enjoy!
If someone would urgently need unreleased features i can make and send
him the .jar in the mean time.

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osmosis bz2 performance

[Stefan Baebler]

Brett Henderson wrote:
>> Can Osmosis get some alias for /dev/stdin (eg "-" would mean stdin on
>> reading tasks and stdout on writing tasks, or we can reserve and
>> handle differently "stdin" and "stdout" file names) ?
>> If we agree on this design i can implement it.
>>   
>> 
> Yep, if you're prepared to make the change then I'm more than happy to 
> accept a patch.  "-" sounds like a good approach to me.  I don't think 
> it would be a complicated change, but if you have any questions about 
> how osmosis is structured just ask.
>   
Well, the patch is in.
See http://trac.openstreetmap.org/changeset/6782

I tested it and ItWorksForMe(tm), but feel free to modify it or poke me 
about it :)
Would "-" have a meaning somewhere else? read xml change, write xml ... 
all file operations perhaps?

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osmosis bz2 performance

[Stefan Baebler]

On Feb 6, 2008 2:44 PM, Brett Henderson <[EMAIL PROTECTED]> wrote:
> Hi Stefan,
>
> The apache bz2 implementation was the only version I could find, if
> there's an alternative I'd like to hear about it.  The apache
> implementation is pure java and I suspect that is the main difference.
> The gzip implementation uses Inflater and Deflator classes which have a
> bunch of native methods.  I see the bz2 support in osmosis as being a
I was suspecting this, tnx for confirmation.

> convenience aid, but if I'm processing large files I use the native
> bzip2 command line tools and pipe the data into osmosis.  For example.
Probably a brief note in the Wiki could save quite a bit of someone's
time. Added to wiki.

> bzcat planet.bz2 | osmosis --rx /dev/stdin --wn
Nice on unixes, but I can't seem to read "/dev/stdin" in Cygwin.
I tried that to pipe the planet straight from wget -> osmosis without
writing the planet to file.
putting native bzip in between would improve perofrmance quite a bit.
wget -O - http:// | bzip -c | osmosis -rx /dev/stdin ...

Can Osmosis get some alias for /dev/stdin (eg "-" would mean stdin on
reading tasks and stdout on writing tasks, or we can reserve and
handle differently "stdin" and "stdout" file names) ?
If we agree on this design i can implement it.

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

[OSM-dev] osmosis bz2 performance

[Stefan Baebler]

Hi!

Today i was pleasantly surprised to see osmosis finished processing
today's planet.osm.gz in only 2 hours. Normally it took approx 6 hours
on the same machine, same bbox, same osmosis version (0.24), but
slightly smaller bz2 version of the whole planet.

http://osm.baebler.net/data/log.txt

Previous benchmark of compression algorithms on planet files:
On Jan 1, 2008 3:47 PM, Bruce Cowan <[EMAIL PROTECTED]> wrote:
> Testing on the latest UK planet (I'm not going to download the real
> planet), I got the following results (all the default settings):
>
> Original:
> 650625406 bytes
>
> gzip:
>  74858343 bytes, 37.198s to compress, 33.005s to decompress.
> 88.5% saving.
>
> bzip2:
>  57581322 bytes, 7m25.169s to compress, 45.983s to decompress.
> 91.1% saving, 23.0% over gzip.
>
> lzma:
>  38175614 bytes, 21m39.595s to compress, 36.187s to decompress.
> 94.1% saving, 49.0% over gzip, 33.7% over bzip2.

Bruce's test: UK planet decompression time with native debian tools:
bz2 / gz ratio = 46s / 33s = 1.39 ... 39% "overtime"

My test: whole planet decompression (+extracting bbox and writing it
to bz2!) time with Osmosis:
bz2 / gz ratio = 6h / 2h = 3 ... 200% "overtime"
At least.
If we say that it took roughly 1h in both cases for decompressed osm
processing and writing out the extracted bbox (compressed to bz2 in
both cases) and substract it from total times to get the bz2
decompression time, the ratio gets even worse:
bz2 / gz ratio = 5h / 1h = 5 ... 400% "overtime"

It seems that apache's bz2 implementation that is used in Osmosis is
very slow compared to the gz implementation. Could it simply be due to
Java or are other bz2 implementations in Java better?

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] osm2pgsql.exe

[Stefan Baebler]

Artem Pavlenko wrote:
> Hello windows users,
>
> osm2pgsql.exe has arrived : http://artem.dev.openstreetmap.org/files/ 
> osm2pgsql_latest.exe.zip
>   
Tnx!

But where did i go wrong?
pgsql 8.2 + posgis
database named gis with postgis extensions, owned by new pqsql user 
"Stefan" (matches my windows username) with maximum permissions
access: all databases, all users from localhost, "trust" authentication 
method (=no passwords)
disabled windows firewall (although pgadmin can connect even with 
firewall enabled, and also telnet to port 5432 looks successful)
yet it still claims a password should be provided somewhere...

$>osm2pgsql_latest.exe -v data\slovenia-071024.osm.bz2
osm2pgsql win32

Using projection SRS 3395 (WGS84 Mercator)
Setting up table: planet_osm_point
Connection to database failed: fe_sendauth: no password supplied

any idea?

tnx,
Stefan

CREATE ROLE "Stefan" LOGIN
  SUPERUSER INHERIT CREATEDB CREATEROLE;
UPDATE pg_authid SET rolcatupdate=true WHERE OID=34261::oid;

CREATE DATABASE gis
  WITH OWNER = "Stefan"
   ENCODING = 'UTF8'
   TABLESPACE = pg_default;
GRANT ALL ON DATABASE gis TO public;
GRANT ALL ON DATABASE gis TO "Stefan";
COMMENT ON DATABASE gis IS 'OSM';


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

On Jan 4, 2008 11:22 AM, 80n <[EMAIL PROTECTED]> wrote:
> The 100 character truncation was a bug in Osmxapi, which is fixed now.
Goodie.

> About 600 tags (out of 180 million) are affected in Osmxapi's database,
> these will get corrected as they percolate through from the osmosis feed.
Surprisingly high number, given that the test was rather synthetic,
with extremely high proportion of accented characters. But i guess
this i not uncommon in exotic parts of the world (Asia comes to mind).

Anyway, I've just changed the test node (in potlatch this time) so it
should appear in the next hourly changeset.

Keeping my fingers crossed and an aye on
http://osmxapi.hypercube.telascience.org/api/0.5/node%5bplace=town%5d%5bbbox=14.5,46.1,14.8,46.2%5d

Stefan

> On Jan 4, 2008 10:17 AM, Stefan Baebler <[EMAIL PROTECTED]> wrote:
> > Khm, but where to set the new limit?
> >
> > According to http://www.facstaff.bucknell.edu/rbeard/name.html
> > it should be 310 bytes = (
> >
> length("Krungthepmahanakonbowornratanakosinmahintarayudyayamahadiloponoparatanarajthaniburiromudomrajniwesmahasatarnamornpimarnavatarsatitsakattiyavisanukamphrasit")
> > -1) * 2 to accomodate slightly shorter names, with all of the
> > characters being exotic, needing 2 bytes to encode :)
> >
> > Or the limits can be simply taken from the official OSM schema (for
> > ways at least, tags of nodes are a mess with semicolons).
> >
> > Stefan
> >
> >
> >
> >
> >
> >
> > On Jan 4, 2008 9:45 AM, 80n <[EMAIL PROTECTED]> wrote:
> > > Hmmm... yes it's truncating at 100 characters.  Working on a fix...
> > >
> > >
> > >
> > > On Jan 4, 2008 7:26 AM, Stefan Baebler <[EMAIL PROTECTED]> wrote:
> > > > Hi again!
> > > >
> > > > Osmxapi behaves much better now, but there is a problem with my test
> node
> > > > in planet.osm it is:
> > > >  > > >
> > > > lon="14.7445634">
> > > > 
> > > > 
> > > >
> > > > 
> > > >
> > > > 
> > > >
> > > > 
> > > > 
> > > > while
> > > >
> > >
> http://osmxapi.hypercube.telascience.org/api/0.5/node%5bplace=town%5d%5bbbox=14.5,46.1,14.8,46.2%5d
> > > > gives
> > > >  > > >
> > > > timestamp="2007-12-22T05:59:49Z">
> > > > 
> > > >
> > > > 
> > > >
> > > > 
> > > >
> > > > 
> > > > 
> > > >
> > > > Note that the last 2 characters in note tag should be "Ââ".
> > > > Planet.osm is ok, but osmxapi seems to misinterpret some characters.
> > > > any ideas?
> > > >
> > > > UTF characters in hourly diffs and their import into osmxapi still
> need
> > > > to be checked.
> > > >
> > > > Stefan
> > > >
> > > >
> > > >
> > >
> > >
> >
>
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

Khm, but where to set the new limit?

According to http://www.facstaff.bucknell.edu/rbeard/name.html
it should be 310 bytes = (
length("Krungthepmahanakonbowornratanakosinmahintarayudyayamahadiloponoparatanarajthaniburiromudomrajniwesmahasatarnamornpimarnavatarsatitsakattiyavisanukamphrasit")
-1) * 2 to accomodate slightly shorter names, with all of the
characters being exotic, needing 2 bytes to encode :)

Or the limits can be simply taken from the official OSM schema (for
ways at least, tags of nodes are a mess with semicolons).

Stefan



On Jan 4, 2008 9:45 AM, 80n <[EMAIL PROTECTED]> wrote:
> Hmmm... yes it's truncating at 100 characters.  Working on a fix...
>
>
>
> On Jan 4, 2008 7:26 AM, Stefan Baebler <[EMAIL PROTECTED]> wrote:
> > Hi again!
> >
> > Osmxapi behaves much better now, but there is a problem with my test node
> > in planet.osm it is:
> >  >
> > lon="14.7445634">
> > 
> > 
> >
> > 
> >
> > 
> >
> > 
> > 
> > while
> >
> http://osmxapi.hypercube.telascience.org/api/0.5/node%5bplace=town%5d%5bbbox=14.5,46.1,14.8,46.2%5d
> > gives
> >  >
> > timestamp="2007-12-22T05:59:49Z">
> > 
> >
> > 
> >
> > 
> >
> > 
> > 
> >
> > Note that the last 2 characters in note tag should be "Ââ".
> > Planet.osm is ok, but osmxapi seems to misinterpret some characters.
> > any ideas?
> >
> > UTF characters in hourly diffs and their import into osmxapi still need
> > to be checked.
> >
> > Stefan
> >
> >
> >
>
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

Hi again!

Osmxapi behaves much better now, but there is a problem with my test node
in planet.osm it is:







while
http://osmxapi.hypercube.telascience.org/api/0.5/node%5bplace=town%5d%5bbbox=14.5,46.1,14.8,46.2%5d
gives







Note that the last 2 characters in note tag should be "Ââ".
Planet.osm is ok, but osmxapi seems to misinterpret some characters.
any ideas?

UTF characters in hourly diffs and their import into osmxapi still need 
to be checked.

Stefan



___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

Brett Henderson wrote:
> Stefan Baebler wrote:
>   
>> On Dec 21, 2007 7:41 AM, Brett Henderson <[EMAIL PROTECTED]> wrote:
>>   
>> 
>>> I'm a dufus, I may have found the problem.  I didn't have the production
>>> encoding hack enabled on the hourly diffs, I've enabled it now.
>>> Presumably this means that most non ascii characters were being mangled.
>>> 
>>>   
>> s*it happens :)
>>
>> Perhaps writing some of the commandline parameters (all sans
>> passwords) into the header of xml might be a good idea to know how
>> data was obtained. Of course this would only reveal last osmosis
>> operation on the given dataset, but better this than no info at all.
>> When opening such file as input stream with osmosis this info could be
>> shown (either by default or in verbose mode).
>>   
>> 
> Perhaps, but it's not very simple.  I'm not sure how you'd pick the 
> relevant options to include without just dumping the entire command line 
> into the file.  That leads to problems figuring out which parts of the 
> command line should be masked out and getting access to the orginal 
> command line in the first place.  It could be messy.
>   
IMHO just passwords should be masked, otherwise whole commandline can be 
put in there.
It would help answering the most common questions "what's in this 
file?", "how was the data filtered?", "how do i run osmosis to get such 
file?"
> However my main issue is that I'm very hesitant to add kludges to 
> support features of limited value.  Given the generic nature of osmosis 
> it is hard to add metadata that is useful.  A similar one is that 
> osmosis isn't currently supporting the bounds xml element because it is 
> difficult to do so in a meaningful way.
>   
There are also external files (polygons, DB config files...) that are 
not included in the command line (the name is mentioned to give some 
idea though).
To go further whole process could be somehow preserved by nesting input 
streams into operations, then wrapping that into output stream, 
preserving the original ... :)













... :)
>>   
>> 
>>> Let me know if you see it occurring on any new hourly diffs.  Daily
>>> diffs already had the prod encoding hack enabled so if they contain UTF8
>>> issues please let me know.
>>>   
>> I will check my problematic nodes tonight and report if there are any 
>> problems.
>> 
Latest change 
(http://planet.openstreetmap.org/hourly/2007122205-2007122206.osc.gz ) 
contains my problematic node, with extra utf characters in note tag, 
looking perfect:








however, osmxapi doesn't respond to
http://osmxapi.hypercube.telascience.org/api/0.5/node%5bplace=town%5d%5bbbox=14.5,46.1,14.8,46.2%5d
Oops, did it kill it or it died of some other reason?

Stefan



___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

On Dec 21, 2007 7:41 AM, Brett Henderson <[EMAIL PROTECTED]> wrote:
> I'm a dufus, I may have found the problem.  I didn't have the production
> encoding hack enabled on the hourly diffs, I've enabled it now.
> Presumably this means that most non ascii characters were being mangled.
s*it happens :)

Perhaps writing some of the commandline parameters (all sans
passwords) into the header of xml might be a good idea to know how
data was obtained. Of course this would only reveal last osmosis
operation on the given dataset, but better this than no info at all.
When opening such file as input stream with osmosis this info could be
shown (either by default or in verbose mode).

> Let me know if you see it occurring on any new hourly diffs.  Daily
> diffs already had the prod encoding hack enabled so if they contain UTF8
> issues please let me know.
I will check my problematic nodes tonight and report if there are any problems.

> It is easy for me to re-generate the hourly diffs if necessary, I just
> have to modify the timestamp file and it will go back in time and
> re-generate up to the current time.  If anybody wishes me to do this let
> me know.
It is probably easier to start with importing fresh planet dump into
informationfreeway, than it would be to import every hourly dump ever
since they were introduced. Then daily , then continue with hourly...
until we find some other problem and start all over :)

Stefan

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev

Re: [OSM-dev] UTF-8 problems in informationfreeway?

[Stefan Baebler]

So, the solution is to just provide a patch with more cases for escaping in
http://trac.openstreetmap.org/browser/applications/utils/osmosis/src/com/bretth/osmosis/core/xml/common/ProductionDbDataDecoder.java
http://trac.openstreetmap.org/browser/applications/utils/osmosis/src/com/bretth/osmosis/core/xml/common/ProductionDbDataEncoder.java
and hope they work fine?

It would of course be better in a long run to fix the main DB, but I'm
not sure what all this brings along. Probably a lot.

Stefan

On Dec 19, 2007 10:36 PM, Brett Henderson <[EMAIL PROTECTED]> wrote:
> Hi All,
>
> I've lost my home ADSL (won't line sync, tried two modems, tried different
> leads, doesn't seem to be my end) so I'm mostly offline.  As a result I'm
> unlikely to get onto this issue in the short term.  With Christmas
> approaching I'm bracing myself for a long'ish outage.
>
> If anybody wishes to take a look, the hacked character encoding class is
> named ProductionDbCharset and has two related classes named
> ProductionDbDataEncoder and ProductionDbDataDecoder.
>
> The classes are instantiated within BaseXmlWriter which is extended by the
> XmlWriter class for writing osm files and XmlChangeWriter for osc files.
> The hack works by just passing the doubly encoded data through the osmosis
> pipeline then fixing it before writing to xml.
>
> Not sure how easy it will be to fix without access to a doubly encoded
> database though.
>
> Brett
>
>
>
> On 12/20/07, Martijn van Oosterhout < [EMAIL PROTECTED]> wrote:
> >
> >
> >
> > On Dec 18, 2007 1:04 PM, Stefan Baebler < [EMAIL PROTECTED]> wrote:
> > > I somehow assumed utf8 would be the default choice by now. Also
> > > http://wiki.openstreetmap.org/index.php/Database_schema
> > > mentions utf8 explicitly for every table individually.
> > >
> > > Why does main api work nicely then?
> > > Why are full planet dumps ok?
> >
> > There's an encoding issue in that what the ruby server thinks it is is
> > different from what the datavase encoding actually is. The net result
> > is that the data is encoded *twice*. For example (not actual codes,
> > just examples):
> >
> > Original char: character 0xef
> > Encoded as: 0xc3 0xaf
> > Stored as: 0xc0 0xc3 0xc0 0xbf
> >
> > > And more importantly:
> > > How can same magic be used to get properly utf8 encoded hourly changes
> (.osc)?
> >
> > Osmosis is in Java which is smart enough to not let you do stupid
> > thing like getting the database connection encoding wrong. It's just a
> > question of fixing the de-double-encoding-hack in osmosis. It doesn't
> > help that it's a *windows* encoding in the first step.
> >
> > Have a nice day,
> > --
> > Martijn van Oosterhout <[EMAIL PROTECTED]> http://svana.org/kleptog/
> >
> > ___
> >
> > dev mailing list
> > dev@openstreetmap.org
> > http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
> >
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev