[osmosis-dev] using osmosis to import full history planet

2011-03-03 Thread Martijn van Exel 

Hi all,

We have been trying to import an extract (bbox) of the full history 
planet file into a PostGIS database using the osmosis db schema. Without 
much success so far. We described the steps we took here: 
https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en


As you can see we have been following the history-specific threads on 
this list and dev, but we got stuck. Maybe something is amiss with the 
planet file, or should we be looking elsewhere?


The salient error message for the latest attempt:

Caused by: org.postgresql.util.PSQLException: ERROR: null value in 
column "relation_version" violates not-null constraint

 Where: COPY relation_members, line 1: "16136292794W 0"

What we want to end up with is a PostGIS database that has all current 
and historic nodes, ways and relations + their attributes for analysis 
purposes.


Any idea what's going on here? Should we be doing this on a Linux machine?

Thanks
Martijn

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-03 Thread Martijn van Exel 

Op 03-03-11 15:07, Martijn van Exel schreef:

Hi all,

We have been trying to import an extract (bbox) of the full history 
planet file into a PostGIS database using the osmosis db schema. 
Without much success so far. We described the steps we took here: 
https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en



..or the published version of the Google Doc for readability...
https://docs.google.com/document/pub?id=1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o


--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-04 Thread Brett Henderson 
Hi Martijn,

The history tasks you're using are part of some experimental work that Peter
Körner was working on and they're not part of the normal Osmosis
distribution.

Perhaps Peter can provide some assistance ...

Brett

On Fri, Mar 4, 2011 at 1:09 AM, Martijn van Exel  wrote:

>  Op 03-03-11 15:07, Martijn van Exel schreef:
>
> Hi all,
>
> We have been trying to import an extract (bbox) of the full history planet
> file into a PostGIS database using the osmosis db schema. Without much
> success so far. We described the steps we took here:
> https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en
>
>  ..or the published version of the Google Doc for readability...
>
> https://docs.google.com/document/pub?id=1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o
>
>
>
> --
> Martijn van Exel
> Senior Researcher
> -
> Geodan S&R
> President Kennedylaan 1
> 1079 MB Amsterdam (NL)
> -
> Tel: +31 (0)20 - 5711 318
> Fax: +31 (0)20 - 5711 333
> -
> E-mail: mart...@geodan.nl
> Website: www.geodan.nl
> KvK-nummer: 33 247475
> Disclaimer: www.geodan.nl/disclaimer
> -
>
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/osmosis-dev
>
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-05 Thread Martijn van Exel 

On 3/4/11 10:18 AM, Brett Henderson wrote:

Hi Martijn,

The history tasks you're using are part of some experimental work that
Peter Körner was working on and they're not part of the normal Osmosis
distribution.

Perhaps Peter can provide some assistance ...

Brett


Hi Brett,

Thanks, I hope so too. We're trying Frederik's history extract script[1] 
to get an extract for the Amsterdam area using the same BBOX we fed to 
osmosis. I'm curious if the result will be different from the osmosis 
output. Frederik, what was your reason for writing this script?


Best - Martijn

[1] 
http://trac.openstreetmap.org/browser/applications/utils/osm-extract/history-excerpt.pl

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-05 Thread Frederik Ramm 

Matijn,

Martijn van Exel wrote:
Thanks, I hope so too. We're trying Frederik's history extract script[1] 
to get an extract for the Amsterdam area using the same BBOX we fed to 
osmosis. I'm curious if the result will be different from the osmosis 
output. Frederik, what was your reason for writing this script?


Thoughtlessness. It didn't occur to me that Osmosis would process a 
history planet without complaint.


Bye
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-08 Thread Martijn van Exel 

Hi Peter and other osmosis users/devs -

We made a few attempts to import a full history planet into a PostGIS 
database and have thus far not been successful. Let me outline what we 
have done (more verbose explanation in the link below):


* Downloaded the latest full history planet from 
http://planet.openstreetmap.org/full-experimental/
* Created an extract for the Amsterdam area first using osmosis and then 
also using Frederik Ramm's history extract perl script[1]. The file 
sizes were not the same (913MB vs 646MB) , but I chalk this up to more 
whitespace trimming by the perl script.
* compiled patched osmosis and history plugin as per instructions on 
[2]. We had to use a db schema from the 0.38 osmosis as the svn version 
shows a schema version of 5, which causes an error (expect version 6).
* executed an osmosis import command on the history extract: 
patched/osmosis --read-xml amsterdam.osm --write-pgsql-history 
user="user" database="database" password="password". The result was an 
osmosis error like this:


SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to 
process COPY file /tmp/copy8882265165846891763rm.
at 
org.openstreetmap.osmosis.pgsnapshot.common.DatabaseContext.loadCopyFile(DatabaseContext.java:293)
at 
org.openstreetmap.osmosis.history.v0_6.impl.HistoryCopyFilesetLoader.run(HistoryCopyFilesetLoader.java:91)
at 
org.openstreetmap.osmosis.history.v0_6.PostgreSqlHistoryCopyWriter.complete(PostgreSqlHistoryCopyWriter.java:130)
at 
org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:110)

at java.lang.Thread.run(Thread.java:662)
Caused by: org.postgresql.util.PSQLException: ERROR: null value in 
column "relation_version" violates not-null constraint
 Where: COPY relation_members, line 1: "16136292794W 
 0"


I don't know at this point whether there is something amiss with the DB 
schema or with the planet file. I did see that the relation member that 
causes the error, way 6292794, occurs twice in the planet file, v1 and 
v2, and both versions have 'visible=true'. This can't be right, or can it?


Who can offer any hints as to what to attempt next? Should I be looking 
at the schema or at the full-planet?


Thanks!

Martijn

[1] 
http://trac.openstreetmap.org/browser/applications/utils/osm-extract/history-excerpt.pl
[2] 
http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20100826/31946e8a/attachment-0001.txt

On 3/4/2011 10:18 AM, Brett Henderson wrote:

Hi Martijn,

The history tasks you're using are part of some experimental work that
Peter Körner was working on and they're not part of the normal Osmosis
distribution.

Perhaps Peter can provide some assistance ...

Brett

On Fri, Mar 4, 2011 at 1:09 AM, Martijn van Exel mailto:mart...@geodan.nl>> wrote:

Op 03-03-11 15:07, Martijn van Exel schreef:

Hi all,

We have been trying to import an extract (bbox) of the full
history planet file into a PostGIS database using the osmosis db
schema. Without much success so far. We described the steps we
took here:

https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en


..or the published version of the Google Doc for readability...

https://docs.google.com/document/pub?id=1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o



--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail:mart...@geodan.nl  
Website:www.geodan.nl  
KvK-nummer: 33 247475
Disclaimer:www.geodan.nl/disclaimer  
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org 
http://lists.openstreetmap.org/listinfo/osmosis-dev




--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-08 Thread Peter Körner 

Am 08.03.2011 11:47, schrieb Martijn van Exel:

Who can offer any hints as to what to attempt next? Should I be looking
at the schema or at the full-planet?


Hi

the experimental plugin has not been maintained for a couple of months 
and is not compatible with the 6er scheme. You should try an older 
osmosis-version (i used 0.37 while developing, though I used a 
svn-compiled snapshot-version of 0.37.


http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.37.zip

(you could use the history .jar in this binary build, too. the patch 
just modified the plugin loader config .txt)


Please be warned: I never imported more than a few nodes into a history 
database. The used node-stores are not able to handle more than some 
thousand nodes.


If you use the plan writing without linestring or bbox builder they 
should not be used and you should be able to import a much bigger set of 
data.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-08 Thread Martijn van Exel 

Hi Peter,

Thanks for your help.

On 3/8/2011 12:11 PM, Peter Körner wrote:

Am 08.03.2011 11:47, schrieb Martijn van Exel:

Who can offer any hints as to what to attempt next? Should I be looking
at the schema or at the full-planet?


Hi

the experimental plugin has not been maintained for a couple of months
and is not compatible with the 6er scheme. You should try an older
osmosis-version (i used 0.37 while developing, though I used a
svn-compiled snapshot-version of 0.37.

http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.37.zip

(you could use the history .jar in this binary build, too. the patch
just modified the plugin loader config .txt)


Do you mean to say there is a history .jar in that build? I don't see 
one. Or do you mean that I can use the history plugin with that build?




Please be warned: I never imported more than a few nodes into a history
database. The used node-stores are not able to handle more than some
thousand nodes.

If you use the plan writing without linestring or bbox builder they
should not be used and you should be able to import a much bigger set of
data.


I do want the linestring option in the end, the BBOX I can probably do 
without. Let me first try it without the linestring builder and see if 
that works. What kind of work would it take to include the linestring 
builder with a larger dataset? Maybe I can try and pool some resources here.


Martijn

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-08 Thread Peter Körner 

Am 08.03.2011 12:47, schrieb Martijn van Exel:

Do you mean to say there is a history .jar in that build? I don't see
one. Or do you mean that I can use the history plugin with that build?
You can use the history jar you build during the last days with this 
version of osmosis.




Please be warned: I never imported more than a few nodes into a history
database. The used node-stores are not able to handle more than some
thousand nodes.

If you use the plan writing without linestring or bbox builder they
should not be used and you should be able to import a much bigger set of
data.


I do want the linestring option in the end, the BBOX I can probably do
without. Let me first try it without the linestring builder and see if
that works. What kind of work would it take to include the linestring
builder with a larger dataset? Maybe I can try and pool some resources
here.


The main problem is that we need to fiddle out which version of a node 
correlates to which version of the way. Because the way does not have a 
direct reference to the node-version, it's required to compare the 
timestamps.


This is done using the HistoryNodeStore interface. It's only 
implementation ExampleHistoryNodeStore uses two instances of 
java.util.Map to store the nodes. This is not efficient enough to run 
with larger datasets but it's well tested and good enough for developing.


A quick look shows that the linestring-builder should be in a usable 
state despite the storage problems. The last thing I was working on was 
the minor version builder, that aims to create a "half version" of a way 
each time one of its nodes got modified. This is somehow experimental.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-08 Thread Martijn van Exel 

Hi Peter,

On 3/8/2011 1:03 PM, Peter Körner wrote:

Am 08.03.2011 12:47, schrieb Martijn van Exel:

Do you mean to say there is a history .jar in that build? I don't see
one. Or do you mean that I can use the history plugin with that build?

You can use the history jar you build during the last days with this
version of osmosis.



So do I just copy the osmosis-history.jar to the 0.37 osmosis? Does it 
go into lib/default?




Please be warned: I never imported more than a few nodes into a history
database. The used node-stores are not able to handle more than some
thousand nodes.

If you use the plan writing without linestring or bbox builder they
should not be used and you should be able to import a much bigger set of
data.


I do want the linestring option in the end, the BBOX I can probably do
without. Let me first try it without the linestring builder and see if
that works. What kind of work would it take to include the linestring
builder with a larger dataset? Maybe I can try and pool some resources
here.


The main problem is that we need to fiddle out which version of a node
correlates to which version of the way. Because the way does not have a
direct reference to the node-version, it's required to compare the
timestamps.


It would make sense to me if this was being dealt with during the 
generation of the full history planet file. But things being as they 
are, timestamp-checks are unavoidable.


This is done using the HistoryNodeStore interface. It's only
implementation ExampleHistoryNodeStore uses two instances of
java.util.Map to store the nodes. This is not efficient enough to run
with larger datasets but it's well tested and good enough for developing.


Is it possible to mimic osm2pgsql's --slim behaviour, create temporary 
tables in postgres to hold all the nodes and work from those instead of 
memory?



A quick look shows that the linestring-builder should be in a usable
state despite the storage problems. The last thing I was working on was
the minor version builder, that aims to create a "half version" of a way
each time one of its nodes got modified. This is somehow experimental.


OK, if I can get it to work I will see how it behaves with more data.
Thanks again,
Martijn

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-09 Thread Martijn van Exel 

Hi Peter,

On 3/8/11 12:11 PM, Peter Körner wrote:

Am 08.03.2011 11:47, schrieb Martijn van Exel:

Who can offer any hints as to what to attempt next? Should I be looking
at the schema or at the full-planet?


Hi

the experimental plugin has not been maintained for a couple of months
and is not compatible with the 6er scheme. You should try an older
osmosis-version (i used 0.37 while developing, though I used a
svn-compiled snapshot-version of 0.37.

http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.37.zip

(you could use the history .jar in this binary build, too. the patch
just modified the plugin loader config .txt)


Dropping the osmosis-history.jar in lib/default and then running osmosis 
gives an error even on --read-xml (task does not exist) so this is 
apparently not the way to go. Should I change something else in the 0.37 
build? Should the library go someplace else?


Thanks,

Martijn

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-10 Thread Peter Körner 

Am 08.03.2011 14:39, schrieb Martijn van Exel:

Hi Peter,

On 3/8/2011 1:03 PM, Peter Körner wrote:

Am 08.03.2011 12:47, schrieb Martijn van Exel:

Do you mean to say there is a history .jar in that build? I don't see
one. Or do you mean that I can use the history plugin with that build?

You can use the history jar you build during the last days with this
version of osmosis.



So do I just copy the osmosis-history.jar to the 0.37 osmosis? Does it
go into lib/default?


I think so but I'm not sure.


This is done using the HistoryNodeStore interface. It's only
implementation ExampleHistoryNodeStore uses two instances of
java.util.Map to store the nodes. This is not efficient enough to run
with larger datasets but it's well tested and good enough for developing.


Is it possible to mimic osm2pgsql's --slim behaviour, create temporary
tables in postgres to hold all the nodes and work from those instead of
memory?


I don't think it's really necessary to create another set of tables. The 
nodes-table that is imported before the first -Tag occurs should be 
sufficient.


The process would be like that:

import all -Tags, when the first -Tag occurs, create the 
indexes and commit the transaction. Then start importing the -Tags. 
Use the alredy finished node-Table with an index on the timestamp-column 
to fetch the node-version and -coordinates matching a way-version and 
build the way-geom as well as the way-nodes-table (with version) from it.



A quick look shows that the linestring-builder should be in a usable
state despite the storage problems. The last thing I was working on was
the minor version builder, that aims to create a "half version" of a way
each time one of its nodes got modified. This is somehow experimental.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-10 Thread Peter Körner 

Am 09.03.2011 12:52, schrieb Martijn van Exel:

Dropping the osmosis-history.jar in lib/default and then running osmosis
gives an error even on --read-xml (task does not exist) so this is
apparently not the way to go. Should I change something else in the 0.37
build? Should the library go someplace else?


Well,

I just exported
http://svn.openstreetmap.org/applications/utils/osmosis/tags/0.37

and then into this folder
http://svn.toolserver.org/svnroot/mazder/osmhist/osmosis-plugin/history/

the activation-patch failed somehow so I needed to append
org.openstreetmap.osmosis.history.HistoryPluginLoader

manually to the original osmosis-plugins.conf file.

ant clean publish then built me all osmosis files.



If you are interested in working on this plugin, I'd transfer it to the 
openstreetmap svn.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] using osmosis to import full history planet

2011-03-10 Thread Martijn van Exel 

On 3/10/11 11:20 AM, Peter Körner wrote:

Am 09.03.2011 12:52, schrieb Martijn van Exel:

Dropping the osmosis-history.jar in lib/default and then running osmosis
gives an error even on --read-xml (task does not exist) so this is
apparently not the way to go. Should I change something else in the 0.37
build? Should the library go someplace else?


Well,

I just exported
http://svn.openstreetmap.org/applications/utils/osmosis/tags/0.37

and then into this folder
http://svn.toolserver.org/svnroot/mazder/osmhist/osmosis-plugin/history/

the activation-patch failed somehow so I needed to append
org.openstreetmap.osmosis.history.HistoryPluginLoader

manually to the original osmosis-plugins.conf file.

ant clean publish then built me all osmosis files.



If you are interested in working on this plugin, I'd transfer it to the
openstreetmap svn.

Peter


Hi Peter,

Yes, please do. We have some Java people here that I might be able to 
have spend some time on this.



Thanks,
Martijn

--
Martijn van Exel
Senior Researcher
-
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-
E-mail: mart...@geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev