Re: [OSM-dev] [Maps-l] History API Server

[80n]

Peter
Here's an implementation of the first item on your list (ways not
relations):
http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version/full

The following example, for Pharaoh's Island in the River Thames shows a
series of edits with different versions of nodes depending on which version
of the way you look at:

http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full

This should work for any versions of any way since yesterday, but as the
full history is not yet loaded older revisions are not yet available.

80n



On Wed, Dec 9, 2009 at 8:27 PM, Peter Körner wrote:

> Hi
>
> Now that we have a full Histoy Planet Dump (Thank you Lars, Matt and all
> who helped with that!), we can again start talking about a History
> Server and a History Api.
>
> What I'm thinking of is an extension of the 0.6 API, that includes calls
> like this:
>
> GET /api/0.6/[way|relation]/#id/#version/full
> GET /api/0.6/node/#id/#version/ways
> GET /api/0.6/node/#id/ways?time=#time
> GET /api/0.6/[node|way|relation]/#id?time=#time
> GET /api/0.6/[node|way|relation]/#id/full?time=#time
> GET /api/0.6/map?time=#time
>
> and maybe others like
>   GET /api/0.6/[nodes|ways|relations]/#id1,#id2,...,#idN
>   GET /api/0.6/[nodes|ways|relations]/#id1/#ver1/.../#idN/#verN
>
> This would make it easy for a client to fetch the state of a city at any
> given time. It would also enable a client to get the complete geometry
> of a way or a relation in any given version or at any given time.
>
> This would make it easy to create animations like the "year of edits"
> for a town or maybe also a country on demand and with real mapnik styles.
>
> The version-based calls (especially GET /api/0.6/node/#id/#version/ways)
> would make it possible to visualize the changes made in a given changeset:
>   if a way was changed, it is contained with each node and each node's
>   version, so a client can fetch all those nodes from api and re-build
>   the way's exact geometry before and after the change
>
>   if a node is changed, the ways depending on it are not included in the
>   changeset and with the current api it's not possible to fetch the
>   geometry of the ways depending on this node as they were before the
>   change
>
> I know that there are implications on how versions are counted (one
> version of a way could stand for more than one geometry, as its nodes
> could have been moved without the version of the way being incremented)
> but they can be solved by simply defining sth. like
>   GET /api/0.6/way/#id/#version/full returns the full geometry of the
>   way #id, as it was when version #version of was created.
>
> Still all possible geometries of the way could be accessed via the time
> predicates.
>
> When taking this idea further, a historic api could even implement
> "minor versions" on ways/relations, thus enabling a client to address
> any change in the geometry of a way (should read: any change on one of
> it's nodes) explicitly.
>
> Such an api could create heavy load on the underlying database, what is
> AFAIR the main reason why such calls aren't implemented in the current
> api. But unlike on the main api, queries to an such a history api don't
> need to be answered in miliseconds. When a /map query for a city on 1st
> jan 2007 takes 15 minutes, well that's ok. I's still faster than to
> download the corresponding planet dump, import int into postgis and
> catch up with the diffs, so what?
>
> My questions to you are
>  - do you think such an server/service/api would be of use for a greater
>audience?
>  - who would be willing to develop, and who would be able to host such a
>service?
>  - do you think these calls can be implemented without having 10 queries
>crash the database?
>
> I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336
> GB of raw HD space -- physical, bot logical!) and enough power to run
> such an api -- that's why I CCed Maps-L -- maybe Ævar can say if he
> thinks this would be possible.
>
> Thank you for reading and commenting,
> Peter
>
> ___
> Maps-l mailing list
> map...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Maps-l] History API Server

[Peter Körner]

> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full
> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full
> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full
> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
> 
> This should work for any versions of any way since yesterday, but as the 
> full history is not yet loaded older revisions are not yet available.


Very cool thing.. Maybe I'll be able to implement some visualization 
from this, at least i think it would be cool just to try it.

If we'd have GET /api/0.6/node/#id/#version/ways we'd have all api calls 
together, needed for a very cool changeset-changes viewer and a revert 
tool *yeehaww*

Peter

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Maps-l] History API Server

[John Smith]

2009/12/10 Peter Körner :
>> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full
>> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full
>> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full
>> http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
>>
>> This should work for any versions of any way since yesterday, but as the
>> full history is not yet loaded older revisions are not yet available.
>
>
> Very cool thing.. Maybe I'll be able to implement some visualization
> from this, at least i think it would be cool just to try it.
>
> If we'd have GET /api/0.6/node/#id/#version/ways we'd have all api calls
> together, needed for a very cool changeset-changes viewer and a revert
> tool *yeehaww*

What about deleted nodes/ways/relations, but instead of giving an ID
how about a bounding box so you can undo a delete?

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Export of GPX data ("planet.gpx")

[Simone Cortesi]

On Wed, Dec 9, 2009 at 00:04, Lars Francke  wrote:
>> I'd take a look at it if no one else is working on it but if someone
>> else is interested I'd gladly step back :)
>
> As no one has stepped forward with any ideas or offers to implement
> this I'll take a shot at it. This also means that I get to define the
> format of the metadata. Once I've worked that out I'll post again.
>
> A quick question to the admins: Do I have anything to worry about
> regarding the server infrastructure?

Thanks Lars,
this is longtime needed data. We have been asked several time for a GPX dump.

thank you!

-S

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [Maps-l] History API Server

[Peter Körner]

> What about deleted nodes/ways/relations, but instead of giving an ID
> how about a bounding box so you can undo a delete?

For the moment I'm thinking about a changeset revert. In a changeset 
you'll get deleted objects marked as such. Finding a deleted node (or 
better: the changeset in which a node was deleted) in an area is another 
thing.

But one step after the another..

Peter

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[josm-dev] restart JOSM

[Dieter Muecke]

Instead of restarting JOSM by hand after plugin update the code snippet below 
does it programmatically. 
It works on Mac OSX but before I submit a patch I would like know how do the 
same
on Windows and Linux.

Kind Regards,
Dieter

try {
File file = File.createTempFile("josm-restart-", ".sh");
PrintWriter pw = new PrintWriter(new FileWriter(file));
pw.println("sleep 1");
pw.println("open -a JOSM.app");
pw.close();
Runtime.getRuntime().exec("/bin/sh " + file.getAbsolutePath());
System.exit(0);

} catch (IOException e) {
e.printStackTrace();
}




___
josm-dev mailing list
josm-...@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev

[OSM-dev] Structured error messages from API

[Matthias Julius]

There are requests in JOSM's trac to improve the handling of API
errors.  To do that JOSM needs to get a better understanding on what
is wrong with the data.

Currently, JOSM is parsing the error strings the API is returning.
This is far from ideal because they are not structured, not documented
and might change without warning.

To improve things I would like to see the API extended to return meta
data about errors (error type, id of offending element, ...) in a
structured way.  There are a couple of ways to that (that came to my mind):

- change the Error header
- add home-made HTTP headers (X-Error-Type ...)
- add pseudo headers to the body
- return a XML document containing all the info

(see also http://trac.openstreetmap.org/ticket/2544)

What do you think?

How important is that to people on the receiving end (application
developers)?

Any suggestions how the format should be?

Matthias

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ian Dees]

On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius wrote:

> There are requests in JOSM's trac to improve the handling of API
> errors.  To do that JOSM needs to get a better understanding on what
> is wrong with the data.
>
> Currently, JOSM is parsing the error strings the API is returning.
> This is far from ideal because they are not structured, not documented
> and might change without warning.
>
> To improve things I would like to see the API extended to return meta
> data about errors (error type, id of offending element, ...) in a
> structured way.  There are a couple of ways to that (that came to my mind):
>
> - change the Error header
> - add home-made HTTP headers (X-Error-Type ...)
> - add pseudo headers to the body
> - return a XML document containing all the info
>
> (see also http://trac.openstreetmap.org/ticket/2544)
>
> What do you think?
>
> How important is that to people on the receiving end (application
> developers)?
>
>
On a related note, I would be very interested in seeing a progress of some
kind being returned when doing a osmChange upload. I realize that it is
difficult (because it could fail after spitting out lots of seemingly valid
IDs), but if it was documented it would be doable.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Peter Körner]

> On a related note, I would be very interested in seeing a progress of 
> some kind being returned when doing a osmChange upload. I realize that 
> it is difficult (because it could fail after spitting out lots of 
> seemingly valid IDs), but if it was documented it would be doable.

Are you talking about getting a "completed xx%" message in the result of 
the open http connection?

Peter

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ian Dees]

On Thu, Dec 10, 2009 at 10:52 AM, Peter Körner wrote:

> On a related note, I would be very interested in seeing a progress of some
>> kind being returned when doing a osmChange upload. I realize that it is
>> difficult (because it could fail after spitting out lots of seemingly valid
>> IDs), but if it was documented it would be doable.
>>
>
> Are you talking about getting a "completed xx%" message in the result of
> the open http connection?
>
>
Yes. To implement that, the HTTP server would need to respond with streaming
XML of some sort over the open HTTP response channel.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ævar Arnfjörð Bjarmason]

On Thu, Dec 10, 2009 at 16:20, Ian Dees  wrote:
> On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius 
> wrote:
>>
>> There are requests in JOSM's trac to improve the handling of API
>> errors.  To do that JOSM needs to get a better understanding on what
>> is wrong with the data.
>>
>> Currently, JOSM is parsing the error strings the API is returning.
>> This is far from ideal because they are not structured, not documented
>> and might change without warning.
>>
>> To improve things I would like to see the API extended to return meta
>> data about errors (error type, id of offending element, ...) in a
>> structured way.  There are a couple of ways to that (that came to my
>> mind):
>>
>> - change the Error header
>> - add home-made HTTP headers (X-Error-Type ...)
>> - add pseudo headers to the body
>> - return a XML document containing all the info
>>
>> (see also http://trac.openstreetmap.org/ticket/2544)
>>
>> What do you think?
>>
>> How important is that to people on the receiving end (application
>> developers)?
>>
>
> On a related note, I would be very interested in seeing a progress of some
> kind being returned when doing a osmChange upload. I realize that it is
> difficult (because it could fail after spitting out lots of seemingly valid
> IDs), but if it was documented it would be doable.

You mean showing upload progress in JOSM as opposed to the current
cylon impression? That could be implemented by counting the number of
bytes of the osmChange request that have been successfully sent over
the wire. That's how upload progress bars are usually implemented.

Obviously the upload could fail but that's another issue.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Peter Körner]

> You mean showing upload progress in JOSM as opposed to the current
> cylon impression? That could be implemented by counting the number of
> bytes of the osmChange request that have been successfully sent over
> the wire. That's how upload progress bars are usually implemented.
> 
> Obviously the upload could fail but that's another issue.

That looks somehow more intelligent than streaming the progress from the 
server back to the client.

Peter

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ævar Arnfjörð Bjarmason]

On Thu, Dec 10, 2009 at 17:01, Peter Körner  wrote:
>> You mean showing upload progress in JOSM as opposed to the current
>> cylon impression? That could be implemented by counting the number of
>> bytes of the osmChange request that have been successfully sent over
>> the wire. That's how upload progress bars are usually implemented.
>>
>> Obviously the upload could fail but that's another issue.
>
> That looks somehow more intelligent than streaming the progress from the
> server back to the client.

It is:) When I upload a file with Google Chrome it always shows the
progress of the upload. This isn't done with some ad-hoc streaming XML
from the server, the client is just counting how many bytes it has
sent over the wire and how large a percentage that is of the total.

I looked at the relevant code in JOSM once but I couldn't find a way
to do it (but I'm not familiar with Java). In Perl with
HTTP::Request::Common you can do e.g.:

$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1
my $req = HTTP::Request::Common->new( ... );
$req->content( sub {
my $chunk = &$gen();
my $length = length $chunk;

warn "I'm now uploading a chunk of length $length, of a total
of $total bytes";

return $chunk;
 } );
 my $result = LWP::UserAgent->request( $req );

(See this code for a practical example of this that I wrote:
http://cpansearch.perl.org/src/CPB/Flickr-Upload-1.32/Upload.pm)

I presume Java has some library to do this as well but I couldn't find it.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ian Dees]

On Thu, Dec 10, 2009 at 10:55 AM, Ævar Arnfjörð Bjarmason
wrote:

> On Thu, Dec 10, 2009 at 16:20, Ian Dees  wrote:
> > On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius 
> > wrote:
> >>
> >> There are requests in JOSM's trac to improve the handling of API
> >> errors.  To do that JOSM needs to get a better understanding on what
> >> is wrong with the data.
> >>
> >> Currently, JOSM is parsing the error strings the API is returning.
> >> This is far from ideal because they are not structured, not documented
> >> and might change without warning.
> >>
> >> To improve things I would like to see the API extended to return meta
> >> data about errors (error type, id of offending element, ...) in a
> >> structured way.  There are a couple of ways to that (that came to my
> >> mind):
> >>
> >> - change the Error header
> >> - add home-made HTTP headers (X-Error-Type ...)
> >> - add pseudo headers to the body
> >> - return a XML document containing all the info
> >>
> >> (see also http://trac.openstreetmap.org/ticket/2544)
> >>
> >> What do you think?
> >>
> >> How important is that to people on the receiving end (application
> >> developers)?
> >>
> >
> > On a related note, I would be very interested in seeing a progress of
> some
> > kind being returned when doing a osmChange upload. I realize that it is
> > difficult (because it could fail after spitting out lots of seemingly
> valid
> > IDs), but if it was documented it would be doable.
>
> You mean showing upload progress in JOSM as opposed to the current
> cylon impression? That could be implemented by counting the number of
> bytes of the osmChange request that have been successfully sent over
> the wire. That's how upload progress bars are usually implemented.
>
> Obviously the upload could fail but that's another issue.
>

The number of bytes uploaded != the progress of the changeset upload process
(although it's better than what's there now). I'm assuming the server reads
the entire file to memory of some sort before parsing the XML and applying
changes to the database? If so, the upload of the osmc file could happen in
a few seconds but it could then take lots more time for the server to apply
the changeset. At that point, the tool will only be able to block until it
times out or receives a response from the server.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Karl Guggisberg]

For JOSM, the structured data currently "embedded" in the error message is
important. Examples are object ids of already deleted objects (410 Gone) or
a date (the close date of a changeset in a 409 Conflict).

I'd prefer a parseable error document in case of http error codes,
preferably in XML. This would not be much of a change because the content of
the 'Error' http header is already replied as error document too (sometimes
for some error cases). 

It would be nice if the OSM API replied a message in english *and* and in
the language supplied in the "Accept-Language" http header. 

Example:

   
  Upload of an object failed.
  Hochladen eines Objekts ist fehlgeschlagen.
  
  


Regards
Karl

-Ursprüngliche Nachricht-
Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im
Auftrag von Matthias Julius
Gesendet: Donnerstag, 10. Dezember 2009 17:16
An: dev@openstreetmap.org
Betreff: [OSM-dev] Structured error messages from API

There are requests in JOSM's trac to improve the handling of API errors.  To
do that JOSM needs to get a better understanding on what is wrong with the
data.

Currently, JOSM is parsing the error strings the API is returning.
This is far from ideal because they are not structured, not documented and
might change without warning.

To improve things I would like to see the API extended to return meta data
about errors (error type, id of offending element, ...) in a structured way.
There are a couple of ways to that (that came to my mind):

- change the Error header
- add home-made HTTP headers (X-Error-Type ...)
- add pseudo headers to the body
- return a XML document containing all the info

(see also http://trac.openstreetmap.org/ticket/2544)

What do you think?

How important is that to people on the receiving end (application
developers)?

Any suggestions how the format should be?

Matthias

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Matthias Julius]

Ian Dees  writes:

> On Thu, Dec 10, 2009 at 10:52 AM, Peter Körner wrote:
>
>> On a related note, I would be very interested in seeing a progress of some
>>> kind being returned when doing a osmChange upload. I realize that it is
>>> difficult (because it could fail after spitting out lots of seemingly valid
>>> IDs), but if it was documented it would be doable.
>>>
>>
>> Are you talking about getting a "completed xx%" message in the result of
>> the open http connection?
>>
>>
> Yes. To implement that, the HTTP server would need to respond with streaming
> XML of some sort over the open HTTP response channel.

But it couln't send Status and Error headers anymore when something
goes wrong because they're sent already at that point.

That would be a very intrusive change.

Matthias

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Karl Guggisberg]

> You mean showing upload progress in JOSM as opposed to the current cylon 
> impression? That could be implemented by counting the number 
> of bytes of the osmChange request that have been successfully sent over the 
> wire. That's how upload progress bars are usually implemented.

It's not the upload or download which takes most of the time but the processing 
on the server.
Uploading an osmChange has three phases:

1. upload the osmChange document   -  can take some time, we could give 
feedback information based on the bytes sent, but can be neglected compared to 
phase 2
2. process on the server - this takes most of the time, no feedback here 
3. download the diffResult - can be neglected compared to phase 2. Again, we 
could give feedback based on the bytes read from the server, but it's not worth 
the effort

Actually, I implemented something based on the approach Avar proposes but I 
wasn't satisfied and didn't check it in.

Regards
Karl 

-Ursprüngliche Nachricht-
Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im 
Auftrag von Ævar Arnfjörð Bjarmason
Gesendet: Donnerstag, 10. Dezember 2009 17:55
An: Ian Dees
Cc: dev@openstreetmap.org
Betreff: Re: [OSM-dev] Structured error messages from API

On Thu, Dec 10, 2009 at 16:20, Ian Dees  wrote:
> On Thu, Dec 10, 2009 at 10:16 AM, Matthias Julius 
> 
> wrote:
>>
>> There are requests in JOSM's trac to improve the handling of API 
>> errors.  To do that JOSM needs to get a better understanding on what 
>> is wrong with the data.
>>
>> Currently, JOSM is parsing the error strings the API is returning.
>> This is far from ideal because they are not structured, not 
>> documented and might change without warning.
>>
>> To improve things I would like to see the API extended to return meta 
>> data about errors (error type, id of offending element, ...) in a 
>> structured way.  There are a couple of ways to that (that came to my
>> mind):
>>
>> - change the Error header
>> - add home-made HTTP headers (X-Error-Type ...)
>> - add pseudo headers to the body
>> - return a XML document containing all the info
>>
>> (see also http://trac.openstreetmap.org/ticket/2544)
>>
>> What do you think?
>>
>> How important is that to people on the receiving end (application 
>> developers)?
>>
>
> On a related note, I would be very interested in seeing a progress of 
> some kind being returned when doing a osmChange upload. I realize that 
> it is difficult (because it could fail after spitting out lots of 
> seemingly valid IDs), but if it was documented it would be doable.

You mean showing upload progress in JOSM as opposed to the current cylon 
impression? That could be implemented by counting the number of bytes of the 
osmChange request that have been successfully sent over the wire. That's how 
upload progress bars are usually implemented.

Obviously the upload could fail but that's another issue.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Karl Guggisberg]

There is alreay ProgressInputStream
  
http://josm.openstreetmap.de/browser/trunk/src/org/openstreetmap/josm/io/ProgressInputStream.java
in the JOSM code base and a "ProgressOutputStream" could be built easily.

But again, I don't think this would help much. If we really wanted to display 
progress information, we would need a kind of streaming reply from the server, 
as proposed by others on this thread.

Regards
Karl 


-Ursprüngliche Nachricht-
Von: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] Im 
Auftrag von Ævar Arnfjörð Bjarmason
Gesendet: Donnerstag, 10. Dezember 2009 18:09
An: Peter Körner
Cc: dev@openstreetmap.org
Betreff: Re: [OSM-dev] Structured error messages from API

On Thu, Dec 10, 2009 at 17:01, Peter Körner  wrote:
>> You mean showing upload progress in JOSM as opposed to the current 
>> cylon impression? That could be implemented by counting the number of 
>> bytes of the osmChange request that have been successfully sent over 
>> the wire. That's how upload progress bars are usually implemented.
>>
>> Obviously the upload could fail but that's another issue.
>
> That looks somehow more intelligent than streaming the progress from 
> the server back to the client.

It is:) When I upload a file with Google Chrome it always shows the progress of 
the upload. This isn't done with some ad-hoc streaming XML from the server, the 
client is just counting how many bytes it has sent over the wire and how large 
a percentage that is of the total.

I looked at the relevant code in JOSM once but I couldn't find a way to do it 
(but I'm not familiar with Java). In Perl with HTTP::Request::Common you can do 
e.g.:

$HTTP::Request::Common::DYNAMIC_FILE_UPLOAD = 1
my $req = HTTP::Request::Common->new( ... );
$req->content( sub {
my $chunk = &$gen();
my $length = length $chunk;

warn "I'm now uploading a chunk of length $length, of a total of $total 
bytes";

return $chunk;
 } );
 my $result = LWP::UserAgent->request( $req );

(See this code for a practical example of this that I wrote:
http://cpansearch.perl.org/src/CPB/Flickr-Upload-1.32/Upload.pm)

I presume Java has some library to do this as well but I couldn't find it.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Nominatim Install Problems

[Jason Surratt]

Hello,

I have about 4 million addresses that I need to geocode. These addresses are 
mostly in the US, but about 200K of them are in Europe and a smattering 
elsewhere. I already have an approach for geocoding US addresses, but I'd like 
to try using Nominatim for geocoding the non-US addresses.

Rather than inappropriately slamming the Nominatim server with bulk requests 
I'm trying to install Nominatim on one of my Linux boxes. However, I'm running 
into some problems getting it installed.

I installed osm2pgsql using the instructions here: 
http://wiki.openstreetmap.org/wiki/Osm2pgsql

Then I used this 

 as a template for ingesting the planet-latest.osm data and building the 
indexes. The exact commands I've run are at the bottom of the email.

The only error messages of consequence that I receive relate to inserting data 
into placex from place during the gazetteer-index.sql. This appears to be due 
to an SRID constraint, but that seems appropriate as the tables have different 
SRIDs. The other ERROR messages relate to missing "IF EXISTS" in drop commands.

After the import completes I can access the search page and the searches 
complete, however it always returns an empty result set. I'm not sure what I 
should look at in the database for clues, but when poking around the word table 
it appears that only data related to the import*.sql files is available.

Any help would be greatly appreciated.

Cheers!

-jason

Version info:
gazetteer/osm2pgsql - SVN revision 19015
CentOS 5.3
PostgreSQL 8.3.8
PHP 5.1.6
PostGIS 1.3.6-1

Commands used for ingest:

pg_ctl -D /dfs/data1/pgsql-data/ -l /var/log/pgsql stop
sleep 3

rm -rf /dfs/data1/pgsql-data || exit -1
initdb -D /dfs/data1/pgsql-data || exit -1

pg_ctl -D /dfs/data1/pgsql-data/ -l /var/log/pgsql start
sleep 3

createuser -s osm || exit -1
createdb -E UTF8 -O osm gazetteer || exit -1
createlang plpgsql gazetteer || exit -1

psql -d gazetteer -f /usr/share/pgsql/contrib/_int.sql || exit -1
psql -d gazetteer -f /usr/share/pgsql/contrib/pg_trgm.sql || exit -1
# README says "lwpostgis.sql", but I assume "lwpostgis-64.sql" is appropriate 
for a 64bit machine
psql -d gazetteer -f /usr/share/pgsql/contrib/lwpostgis-64.sql || exit -1
psql -d gazetteer -f /usr/share/pgsql/contrib/spatial_ref_sys.sql || exit -1
psql -d gazetteer -f /usr/share/osm2pgsql/900913.sql || exit -1

echo running osm2pgsql

# I receive the same errors when using the planet-latest.osm. to make tests 
faster I've been testing w/ the
# cloudmade denmark.osm file acquired here:
# http://downloads.cloudmade.com/europe/denmark#breadcrumbs
/usr/share/osm2pgsql/osm2pgsql -lsc -C 2048 -O gazetteer -U postgres -m -d 
gazetteer /dfs/data1/osm/denmark.osm.bz2 || exit -1
cd /var/lib/pgsql/gazetteer

echo osm2pgsql done

createuser -SDR www-data
createuser -SDR apache

cat import_worldboundaries.sql | sed -e s/twain/osm/g | psql -d gazetteer || 
exit

# README says to run these after the import*sql files, but I get function 
missing errors if I do that.
echo loading functions
# The first time through causes errors b/c of dependencies. The second pass 
errors are the interesting ones.
cat gazetteer-functions.sql | psql -d gazetteer 2> /dev/null > /dev/null || 
exit -1
echo loading tables
cat gazetteer-tables.sql | psql -d gazetteer || exit -1
echo loading functions again
cat gazetteer-functions.sql | psql -d gazetteer || exit -1

cat import_country_name.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_gb_postcodearea.sql | sed -e s/twain/osm/g | psql -d gazetteer || 
exit
cat import_gb_postcode.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_specialwords.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_us_statecounty.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_state.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit

psql -d gazetteer -f gazetteer-index.sql


echo all done


Jason R. Surratt
SPADAC
email: jason.surr...@spadac.com

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ævar Arnfjörð Bjarmason]

On Thu, Dec 10, 2009 at 17:25, Karl Guggisberg
 wrote:
>> You mean showing upload progress in JOSM as opposed to the current cylon 
>> impression? That could be implemented by counting the number
>> of bytes of the osmChange request that have been successfully sent over the 
>> wire. That's how upload progress bars are usually implemented.
>
> It's not the upload or download which takes most of the time but the 
> processing on the server.
> Uploading an osmChange has three phases:

Indeed, but I understood it from Ian's post "when doing a[sic]
osmChange upload" that he was only interested in step #1.

> 1. upload the osmChange document   -  can take some time, we could give 
> feedback information based on the bytes sent, but can be neglected compared 
> to phase 2

Actually when I do uploads uploading the file itself and processing on
the server seems to take around the same time, judging from network
sniffing I've done. Results may vary though.

> 2. process on the server - this takes most of the time, no feedback here

Yup, PostgreSQL can't tell you where it's at in the query:
http://wiki.postgresql.org/wiki/Query_progress_indication

> 3. download the diffResult - can be neglected compared to phase 2. Again, we 
> could give feedback based on the bytes read from the server, but it's not 
> worth the effort
>
> Actually, I implemented something based on the approach Avar proposes but I 
> wasn't satisfied and didn't check it in.

I've filed a bug to track this, for what it's worth:
http://josm.openstreetmap.de/ticket/4145

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Ian Dees]

On Thu, Dec 10, 2009 at 11:39 AM, Ævar Arnfjörð Bjarmason
wrote:

> On Thu, Dec 10, 2009 at 17:25, Karl Guggisberg
>  wrote:
> >> You mean showing upload progress in JOSM as opposed to the current cylon
> impression? That could be implemented by counting the number
> >> of bytes of the osmChange request that have been successfully sent over
> the wire. That's how upload progress bars are usually implemented.
> >
> > It's not the upload or download which takes most of the time but the
> processing on the server.
> > Uploading an osmChange has three phases:
>
> Indeed, but I understood it from Ian's post "when doing a[sic]
> osmChange upload" that he was only interested in step #1.
>

My interest is in getting feedback from the API as it applies large
changesets, so I'm mostly interested in #2 as the actual HTTP POST of the
data file is a relatively small part of the transaction.


>
> > 1. upload the osmChange document   -  can take some time, we could give
> feedback information based on the bytes sent, but can be neglected compared
> to phase 2
>
> Actually when I do uploads uploading the file itself and processing on
> the server seems to take around the same time, judging from network
> sniffing I've done. Results may vary though.
>

When uploading large diffs (changeset reverts, new data imports), time spent
sending the data over the network is insignificant compared to the time
spent applying to the server.


>
> > 2. process on the server - this takes most of the time, no feedback here
>
> Yup, PostgreSQL can't tell you where it's at in the query:
> http://wiki.postgresql.org/wiki/Query_progress_indication
>

Presumably the API server performs many SQL queries to update the database,
so the API application could provide feedback as it goes through the steps
of updating based on the changeset.

Someone (sorry my shortterm memory is pretty crummy) pointed out that this
would probably break the API's current error system as the server would no
longer be able to send error headers once it starts sending streaming
response information.


>
> > 3. download the diffResult - can be neglected compared to phase 2. Again,
> we could give feedback based on the bytes read from the server, but it's not
> worth the effort
> >
> > Actually, I implemented something based on the approach Avar proposes but
> I wasn't satisfied and didn't check it in.
>
> I've filed a bug to track this, for what it's worth:
> http://josm.openstreetmap.de/ticket/4145
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Nominatim Install Problems

[Jason Surratt]

I just noticed the Geocoding mailing list. My bad. I will repost there.

Thanks!

Jason R. Surratt
SPADAC
email: jason.surr...@spadac.com

From: dev-boun...@openstreetmap.org [mailto:dev-boun...@openstreetmap.org] On 
Behalf Of Jason Surratt
Sent: Thursday, December 10, 2009 12:38 PM
To: dev@openstreetmap.org
Subject: [OSM-dev] Nominatim Install Problems

Hello,

I have about 4 million addresses that I need to geocode. These addresses are 
mostly in the US, but about 200K of them are in Europe and a smattering 
elsewhere. I already have an approach for geocoding US addresses, but I'd like 
to try using Nominatim for geocoding the non-US addresses.

Rather than inappropriately slamming the Nominatim server with bulk requests 
I'm trying to install Nominatim on one of my Linux boxes. However, I'm running 
into some problems getting it installed.

I installed osm2pgsql using the instructions here: 
http://wiki.openstreetmap.org/wiki/Osm2pgsql

Then I used this 

 as a template for ingesting the planet-latest.osm data and building the 
indexes. The exact commands I've run are at the bottom of the email.

The only error messages of consequence that I receive relate to inserting data 
into placex from place during the gazetteer-index.sql. This appears to be due 
to an SRID constraint, but that seems appropriate as the tables have different 
SRIDs. The other ERROR messages relate to missing "IF EXISTS" in drop commands.

After the import completes I can access the search page and the searches 
complete, however it always returns an empty result set. I'm not sure what I 
should look at in the database for clues, but when poking around the word table 
it appears that only data related to the import*.sql files is available.

Any help would be greatly appreciated.

Cheers!

-jason

Version info:
gazetteer/osm2pgsql - SVN revision 19015
CentOS 5.3
PostgreSQL 8.3.8
PHP 5.1.6
PostGIS 1.3.6-1

Commands used for ingest:

pg_ctl -D /dfs/data1/pgsql-data/ -l /var/log/pgsql stop
sleep 3

rm -rf /dfs/data1/pgsql-data || exit -1
initdb -D /dfs/data1/pgsql-data || exit -1

pg_ctl -D /dfs/data1/pgsql-data/ -l /var/log/pgsql start
sleep 3

createuser -s osm || exit -1
createdb -E UTF8 -O osm gazetteer || exit -1
createlang plpgsql gazetteer || exit -1

psql -d gazetteer -f /usr/share/pgsql/contrib/_int.sql || exit -1
psql -d gazetteer -f /usr/share/pgsql/contrib/pg_trgm.sql || exit -1
# README says "lwpostgis.sql", but I assume "lwpostgis-64.sql" is appropriate 
for a 64bit machine
psql -d gazetteer -f /usr/share/pgsql/contrib/lwpostgis-64.sql || exit -1
psql -d gazetteer -f /usr/share/pgsql/contrib/spatial_ref_sys.sql || exit -1
psql -d gazetteer -f /usr/share/osm2pgsql/900913.sql || exit -1

echo running osm2pgsql

# I receive the same errors when using the planet-latest.osm. to make tests 
faster I've been testing w/ the
# cloudmade denmark.osm file acquired here:
# http://downloads.cloudmade.com/europe/denmark#breadcrumbs
/usr/share/osm2pgsql/osm2pgsql -lsc -C 2048 -O gazetteer -U postgres -m -d 
gazetteer /dfs/data1/osm/denmark.osm.bz2 || exit -1
cd /var/lib/pgsql/gazetteer

echo osm2pgsql done

createuser -SDR www-data
createuser -SDR apache

cat import_worldboundaries.sql | sed -e s/twain/osm/g | psql -d gazetteer || 
exit

# README says to run these after the import*sql files, but I get function 
missing errors if I do that.
echo loading functions
# The first time through causes errors b/c of dependencies. The second pass 
errors are the interesting ones.
cat gazetteer-functions.sql | psql -d gazetteer 2> /dev/null > /dev/null || 
exit -1
echo loading tables
cat gazetteer-tables.sql | psql -d gazetteer || exit -1
echo loading functions again
cat gazetteer-functions.sql | psql -d gazetteer || exit -1

cat import_country_name.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_gb_postcodearea.sql | sed -e s/twain/osm/g | psql -d gazetteer || 
exit
cat import_gb_postcode.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_specialwords.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_us_statecounty.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit
cat import_state.sql | sed -e s/twain/osm/g | psql -d gazetteer || exit

psql -d gazetteer -f gazetteer-index.sql


echo all done


Jason R. Surratt
SPADAC
email: jason.surr...@spadac.com

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Structured error messages from API

[Tom Hughes]

On 10/12/09 17:39, Ævar Arnfjörð Bjarmason wrote:

>> 2. process on the server - this takes most of the time, no feedback here
>
> Yup, PostgreSQL can't tell you where it's at in the query:
> http://wiki.postgresql.org/wiki/Query_progress_indication

Um... It's not like it's a single query. It will be thousands of 
individual queries each of which will, on their own, complete virtually 
instantaneously.

On top of all that there is time parsing the XML, time running other 
ruby code to validate things, etc, etc.

Basically, even if sending a streaming response from rails was possible 
(which I'm not sure it is - it's certainly hard) there is no simple way 
to measure progress.

Tom

-- 
Tom Hughes (t...@compton.nu)
http://www.compton.nu/

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [josm-dev] restart JOSM

[Andre Hinrichs]

Am Donnerstag, den 10.12.2009, 15:56 +0100 schrieb Dirk Stöcker:
> On Thu, 10 Dec 2009, Dieter Muecke wrote:
> 
> > Instead of restarting JOSM by hand after plugin update the code snippet 
> > below does it programmatically. It works on Mac OSX but before I submit 
> > a patch I would like know how do the same on Windows and Linux.
> 
> Why do you want restarting JOSM after plugin update?
> 
> The plugin update on startup is done before loading the plugins, so 
> already the new plugins are loaded.

Well, that's true for updating plugins but not for activating or
deactivating plugins. Don't know if the patch can handle it but it would
be a rather good thing if I would not be forced to restart JOSM after
changing the plugin settings.

Andre



___
josm-dev mailing list
josm-...@openstreetmap.org
http://lists.openstreetmap.org/listinfo/josm-dev

[OSM-dev] Polygon buffer tool (for Osmosis)

[Frank Steggink]

Hi,

I've created a small Python tool which buffers the polygon in a polygon 
file (for Osmosis), and creates a new polygon file. It can be found 
here: [1]. For the buffering PostGIS is used. All Canadian polygon files 
from [2] have been tested and are working.

Requirements: Python (2.6, but 2.5 likely works too), PostGIS (1.3+), 
PyGreSQL
More info can be found in the file itself, or by typing "./polybuffer.py 
-h" in the command prompt.

Please let me know any issues you experience with this tool.

Cheers,

Frank

[1] 
http://svn.openstreetmap.org/applications/utils/osm-extract/polygons/polybuffer.py
[2] http://www.maproom.psu.edu/dcw/


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Nominatim Install Problems

[Peter Körner]

Jason Surratt schrieb:
> I just noticed the Geocoding mailing list. My bad. I will repost there.

Please also post here if you got a solution.


Peter

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev