Re: [osmosis-dev] Latest osc change file returns empty xml.
Hi, On 11/14/20 21:19, Dave F via osmosis-dev wrote: > I'm using osmosis --rri workingDirectory=. --wxc GB.osc.gz > I amended configuration.txt to point to the GB folder: > baseUrl=http://download.geofabrik.de/europe/great-britain-updates > > Running the command it completed it's task, but the osc file contained > an empty xml file: Is it possible that you have accidentally set your replication state to the current state? If you download the Great Britain file now, and set the appropriate state.txt value, then the expected result of running osmosis is an empty XML file because there are no changes available. As of this writing, the /europe/great-britain-updates/state.txt says sequenceNumber=2791 and running osmosis with that will yield an empty file, until such time as the state.txt on the server points to a sequenceNumber of 2792. > On other Q: > As the update interval for Geofabrik is 24 hours, should I amend this > value in the configuration.txt file to suit: > maxInterval = 3600 It does not make a difference; any maxInterval between 1 and 86400 should behave the same, as each would pick exactly one update per run. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis java.lang.IndexOutOfBoundsException
Hi, I think this was a problem of the Geofabrik download server. We accidentally generated our "cleaned" files (the GDPR conformant file without any user data) in a different way last night, completely dropping the UID/Username fields, instead of replacing them with 0 and empty strings as we did before. This tripped up osmosis and also older versions of osm2pgsql (<= 0.94). The pbf files are replaced with old-style, compatible files now. Nightly updates and bz2 files were not affected since they don't use the PBF format. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis 0.46 Release
Hi, On 04.05.2018 11:21, dev Yogurt wrote: > The file I'm trying to convert is > https://download.geofabrik.de/europe/hungary-latest.osm.pbf ... which, since yesterday, does not contain user names, user IDs, or changeset IDs any longer. That is probably the core of the issue. We've made that change on the Geofabrik server to comply with data protection rules. The "old" file with full user info is still available as https://osm-internaldownload.geofabrik.de/europe/hungary-latest-internal.osm.pbf and requires a login to access. I fear that we were a bit too optimistic making this change; there's just too much software out there that cannot cope with missing attributes. We'll likely change this, and set uid=0, username="", changeset=0 instead of dropping them altogether. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem
Brett, thank you for your comment. The issue is not an urgent one for me since workarounds exist, and on the many osmosis-based OSM updating machines I've been running continuously for years, this is only the second time I run into it. So it is a rare quirk, but of course I would feel better if I knew where it came from. I've re-built osmosis with Xerces 2.11.0 and this doesn't change the situation. Should I perhaps try and build a minimal "use Xerces to parse this XML file" program, and if I can replicate the problem with that, file a bug with Xerces? Or is the way in which Osmosis uses Xerces somehow special so that a simple program like that would be very unlikely to trigger the bug? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem
Hi, On 01/11/2017 10:30 AM, Frederik Ramm wrote: > SEVERE: Thread for task 1-read-xml-change failed I was a bit over-eager in shortening the stack trace. Full detail: org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file x.osc. publicId=(null), systemId=(null), lineNumber=583379, columnNumber=90. at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114) at java.lang.Thread.run(Thread.java:745) Caused by: org.xml.sax.SAXParseException; lineNumber: 583379; columnNumber: 90; Invalid byte 2 of 4-byte UTF-8 sequence. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109) ... 1 more Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence. at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown Source) at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source) at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) ... 11 more -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] osmosis wrongly claims to see UTF8 problem
Hi, I have recently had osmosis in --rri mode refuse to apply an update it had downloaded from OSM, claiming there was an UTF8 error in the file. I looked and looked but the file was fine, passed UTF8 and XML validity checks. I tried to isolate the line that gave me the "error" but isolating it made the problem go away. Only including the 583379 previous lines makes the error occur. So I now have two .osc files, one with 583380 lines and one with 583379 lines: $ wc -l x.osc y.osc 583380 x.osc 583379 y.osc their only difference is one line at the beginning of the longer file: $ diff x.osc y.osc 2d1 < But the longer one fails to process in osmosis, and the shorter one works: $ osmosis --read-xml-change x.osc --write-null-change Jan 11, 2017 10:19:41 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.43.1 ... SEVERE: Thread for task 1-read-xml-change failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file x.osc. publicId=(null), systemId=(null), lineNumber=583379, columnNumber=90. at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114) $ osmosis --read-xml-change y.osc --write-null-change Jan 11, 2017 10:20:34 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.43.1 ... Jan 11, 2017 10:20:35 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Total execution time: 1448 milliseconds. Since the line which supposedly contains the "error" is identical in both files, it can't really be an error (and the line does not contain any non-ASCII characters). Re-formatting the XML file with "xmlstarlet fo" or "xmlstarlet c14n" makes the problem go away. I've reproduced this bug on different machines with different Osmosis versions. I've tried these java versions with identical results: $ java -showversion java version "1.7.0_121" OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.14.04.1) $ java -showversion openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) I have uploaded the two .osc files here: http://www.remote.org/frederik/tmp/osmosis-bug-try-read-xml-change-write-null-change-on-these-files-which-differ-only-by-one-line.zip I'd be interested in any insights anyone has to share. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading file over http
Hi, On 06/19/2016 09:09 PM, Stephen Knox wrote: > I should have said, I am using Osmosis as a Java library and don't have > access to the command line as I am using Platform as a service, so I guess > the same applies so long as I can find a decent Bzip converter to stream to > Osmosis? Osmosis includes bzip2 support so it could decode the .bz2 itself but since Java bzip2 is much slower than the standalone commandline program, the version with bzip2 in the pipe is usually recommended. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading file over http
Hi, On 06/19/2016 04:16 PM, Stephen Knox wrote: > This might be a bit of a dumb question, but is it possible already (or > would it be possible with minor modification) to read a remote osm xml > file over http and stream it, so that the file does not have to be fully > downloaded? wget -O- http://somewhere/file.osm.bz2 | bzcat | osmosis --read-xml - --do-something-else Sadly the more efficient .osm.pbf files cannot be consumed in streaming fashion due to Osmosis attempting to seek in the file. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Upload OSM Files to private Server
Hi,
On 03/09/2016 01:08 PM, Marius Beuerle wrote:
> at first we merged our osm file and the osm file of Berlin via osm convert
> and uploaded the merged file to our server. But then following problem
> appeared: The new ways and nodes aren’t shown in the map, although they were
> correctly uploaded to our database. Does anybody has an idea why that didn’t
> work out?
Explain your toolchain. For the above to work you would have to run:
* the OSM website ("rails port")
* Osmosis regularly generating diffs from the PostgreSQL instance behind
the website
* osm2pgsql regularly importing these diffs into your rendering database
* renderd/tirex + mod_tile to make updated tiles from your rendering
database
Do you have all these components?
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Supplying postgres port to osmosis commands
Hi, On 10/10/2014 10:49 PM, Walter Nordmann wrote: > just call osmosis with host=localhost:15432 Oh dear, that was probably the reason the "port" thing was never implemented - because it is so simple to embed it in the host. Should have thought about that sooner ;) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Supplying postgres port to osmosis commands
Hi,
On 10/10/2014 09:05 PM, Justin Palmer wrote:
> I have Postgres running on a non-default port and unfortunately it
> doesn't look like osmosis will accept a custom port
Indeed it doesn't.
If you have nothing else on your port 5432, you can of course redirect
that port ("ssh localhost -L
5432:localhost:whereever-your-postgres-listens"). Else here's a patched
version of Osmosis that does support a "port" argument on all database
tasks:
https://github.com/woodpeck/osmosis/tree/database_port_support
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error on a node modified and deleted in the same changeset
Hi, (re-heating this 4 month old thread) On 07/13/2013 01:03 PM, Jocelyn Jaubert wrote: > I'm currently using a workaround, which is to set maxInterval in > configuration.txt to 0.5 day to make sure that osmosis is not trying to > merge two diffs. Then I had to add a loop to run osmosis until there are > no recent diffs. I've just been bitten by the same problem when trying to keep an Europe-only Nominatim instance up to date with Geofabrik diffs. I wonder if simply dropping the --simplify-change from Nominatim's osmosis invocation will solve things - might lose a little performance for things modified on subsequent days but hey. (One could even do this: try with --simplify-change by default but if this fails, re-run without... Nominatim already re-runs osmosis if it fails so I'd just have to add the "drop simplify-change" code.) > Maybe the best way to fix this is to add a note on a website explaining > that the diffs generated by Geofabrik are not meant for "direct" > consumption by osmosis with a big value on maxInterval ? I'll do that. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Merge multiple files
Hi, On 13.10.2013 12:54, Brett Henderson wrote: > Can this be solved with > a wrapper shell script? Here's a wrapper script I use to combine multiple osc files (called like "./myscript.sh *.osc > foo.osc"): #!/bin/bash CMDLINE=` echo "--read-xml-change $1" echo "--sort-change" shift while [[ $# > 0 ]] do echo "--read-xml-change $1" echo "--sort-change" echo "--merge-change" echo "--sort-change" shift done echo "--simc" echo "--write-xml-change -" ` /srv/osmosis/bin/osmosis $CMDLINE Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error on a node modified and deleted in the same changeset
Hi, On 07.05.2013 23:50, Jocelyn Jaubert wrote: I'm including a patch for this, that only impacts the ChangeDeriver class. I have tested it on a country, and the generated diff was correct - the only difference with the current osmosis version is the version on deleted elements. I had not followed this thread, sorry. I didn't know the Geofabrik diffs had this problem, but of course if one thinks about it, it does seem logical - comparing two excerpts, when the newer one doesn't have an object then how am I to know who deleted it when and in what version? I could run Jocelyn's patch on the Geofabrik server but technically it would create wrong osc files - if you were to compare a Geofabrik osc with one loaded from osm.org you'd find that the Geofabrik diff reports the wrong user name, wrong timestamp, and *possibly* also wrong version on a deleted object. No idea if that problem is worse than the one we have now! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Un-Redacting Stuff
Hi, with the license change we introduced the concept of "redacted" objects. Since "redacting" an old version touches that version in the database, initially such redactions made Osmosis issue diffs that contained that old version; we then introduced a quick fix to stop that: https://github.com/openstreetmap/osmosis/blob/master/apidb/src/main/java/org/openstreetmap/osmosis/apidb/v0_6/impl/EntityDao.java#L450 We're now also using "redaction" to suppress objects where a copyright violation has occurred - but mistakes are possible, so we need to have a way to un-redact things if necessary, i.e. remove the "redaction_id" from a historic version again. Simply setting the column to NULL will, again, make Osmosis issue a diff that contains the old version; this is unwanted. How could we proceed? Ideas: 1. Introduce special value "0" (not NULL) to denote an un-redacted object; leave Osmosis unchanged (so it treats NULL and 0 differently, will only issue .osc for objects with redaction_id=NULL), and modify other API code to treat 0 and NULL the same (so historic versions can be accessed through the API if redaction_id=NULL or 0). Cheap, easy, but a bit ugly. 2. Introduce an additional column "suppress_diff" to nodes/ways/relations tables; on un-redaction, set redaction_id=NULL and suppress_diff=TRUE; modify Osmosis by assing an "and not suppress_diff" to the SQL query. Would increase database size by something like 4 GB for the extra column. 3. Introduce an additional table "un-redacted objects", store object type, version, and id; when an object is un-redacted, add it to that table and clear the object's redaction_id, then modify the Osmosis query to only output objects that are not found in that table. Uses little space but makes diff creation slower. There might be more... Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Question regarding the replication file structure
Hi, I'm toying with the idea of offering regionalised diffs - i.e. a series of daily diffs for every regional extract that download.geofabrik.de has to offer. To make it easy for consumers to keep their extracts up to date, I thought about making an Osmosis-style directory for each extract, e.g. something like download.geofabrik.de/openstreetmap/europe/germany/nordrhein-westfalen/000/000/001.osc.gz or so. Just to be safe: What are the conventions that I will have to follow so that this works seamlessly with existing clients? Simply have a xxx.osc.gz and matching xxx.state.txt in the leaf directory, count from 000 to 999 then wrap to the next directory, and have the most recent state.txt file at the root directory as well - anything else? If the frequency wasn't exactly daily - if, say, because of some sort of glitch there was extract for one day and therefore the diff is missing, or if there were two extracts in one day - would that matter? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pgsnapshot composite index results
Hi,
On 22.01.2013 11:17, Paweł Paprota wrote:
So every WMS map request generates something like this:
SELECT "type",encode(ST_AsBinary(ST_Force_2D("way")),'base64') as "way"
FROM "public"."view_railways" WHERE ("way" && ST_GeomFromText('POLYGON
((2031543.337480622 6414748.28685562, 2031543.337480622
6578323.527363132, 2195118.5779881305 6578323.527
363132, 2195118.5779881305 6414748.28685562, 2031543.337480622
6414748.28685562))', 900913) AND "type" = 'rail')
Such requests are ideally served by a conditional index (i.e. when you
create the index with a "WHERE type=rail" clause). Since your WMS is
probably not running custom queries, you can pre-create these indexes
and they're not even terribly expensive to create. Having a large number
of them slows down updates but having them for the 5-10 most
time-consuming SELECTs is certainly worth it.
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Geometries for the relations table of pgsnapshot
Hi, On 25.10.2012 02:37, Paul Norman wrote: I am wondering what would be involved with getting a geometry column built for the relations table. There is a *lot* of open source code out there that does this already, or at least parts of this: * Multipolygon builder in Jochen's Osmium * Multipolygon builder in ImpOSM * Multipolygon builder in osm2pgsql * Multipolygon builder in osm2ogr (all implementations are completely separate) Also osm2pgsql has code to build linear geometries for routes, and I expect some of the others as well. There's also potentially reusable code from Lonvia who uses, as far as I know, a pgsnapshot database to compute hiking route geometries. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] applying diffs to the new planet
Hi, On 16.09.2012 18:38, Christian H. Bruhn wrote: I downloaded the ODbL-Planet (PBF-format). How can apply ALL the (hourly-)diffs? How should the stat-file look like? Simply set the sequence number to 0 and it should work. So I tried without a state-file. I didn't know that it works without a state file at all! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Msg error with osmosis when bboxing
Hi, On 05/13/2012 02:56 PM, Séverin MENARD wrote: I have a small issue with osmosis. Already post this message on OSM help (no answer) I saw your question there but had no idea what the problem could be. The message you are seeing would typically occur if something is logically wrong with your Osmosis command line, e.g. you read a file but don't write the results to anywhere or so. For example, if I enter osmosis --read-xml file="dummy" then I get the message The following named pipes () and 1 default pipes have not been terminated with appropriate output sinks. Osmosis wouldn't even try to read the "dummy" file. If, however, I run your command on my machine: osmosis --read-xml file="D:/OSM/Salvador/brazil.osm" --bounding-polygon file="D:/OSM/Salvador/Bahia_contour.poly" --write-xml file="D:/OSM/Salvador/bahia.osm" then I get the message Unable to read XML file D:/OSM/Salvador/brazil.osm because Osmosis has correctly parsed the command line and tried to execute it. So there's nothing wrong with your command line as far as I can see. I therefore think that something in your osmosis.bat must be broken, leading to Osmosis being called with not enough parameters or something crazy. Maybe you should add an "echo" statement to your osmosis.bat so you can see what command is run. (It is also possible, but less likely, that you have some strange Java installation that ruins command line arguments.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Remove entities in a bounding box leave other intact
Hi, On 01/16/2012 07:37 PM, Jaume Figueras i Jové wrote: I've been looking different Osmosis options but I have no idea on how to do this processing. You can use the --bp task and define a polygon with an exclamation mark before the polygon ID to cut something out, see http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format Example: test 1 0.000E+00 0.E+00 1.000E+01 0.E+00 1.000E+01 1.E+01 0.000E+01 1.E+01 0.000E+00 0.E+00 END !2 0.200E+01 0.2000E+01 0.800E+01 0.2000E+01 0.800E+01 0.8000E+01 0.200E+01 0.8000E+01 0.200E+01 0.2000E+01 END END Bye Frederk -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet files appear truncated to osmosis, was: [OSM-talk] old planet files with 'nolt' in the name
Hi, On 01/15/2012 05:31 PM, Martijn van Exel wrote: It seems that the problem is not with the particular file. I tried the same process with a few other files, all give the same error around line 4000. I'm officially confused. I used osmosis a lot before on the same machine. There are different ways to construct a bz2 file, and we did have issues in the past with Java's bz2 reading and the way we would create the bz2 in a multi-threaded fashion. I was under the impression that this was fixed, but can you try to replace your mvexel@lima$ /osm/software/osmosis-0.39/bin/osmosis --rx /osm/planet/historical/planet-090701.osm.bz2 ... with bzcat /osm/planet/historical/planet-090701.osm.bz2 | /osm/software/osmosis-0.39/bin/osmosis --rx - ... and see if the problem persists? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] full OSM mirror with Osmosis
Hi, is there anyone on this list who runs a full OSM replication server with Osmsois (i.e. import a full history planet file, keep both historic and "current" tables, and regularly apply diffs, all using the apidb schema)? I would be interested in timings for initial import and diff application and of course in the specs for the hardware. Also if someone has tried and failed ;) Reason I'm asking is that I am thinking to set up a kind of "shadow DB" that has an ODbL compliant version of everything, so everyone can easily see how their area would look like if the license change was done today. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Merge huge count of files
Hi, > What is the best strategy to merge a huge count (e.g. 100x100 matrix) > together with a minimum of needed memory? You don't say what kind of files you are merging but it is possible that the best strategy is not using Osmosis at all for this task. If you have plain simple OSM XML files then the following will merge them super fast: (echo ''; cat *osm|grep -v ""') > newfile.osm You can add an Osmosis sort step if you want. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Using PBF for temp files
Hi, when Osmosis writes temporary files, using the SimpleObjectStore, it uses a relatively primitive serialisation format and runs gzip on that. Given that we now have rather fast and effective OSM data storage with the PBF reader and writer, I wonder what it would take to switch from writing classic temp files to using (probably uncompressed) PBF as the temporary storage format. I've rummaged around in the code for a bit but at least it wasn't straightforward to do since the temporary storage code uses "containers" for everything whereas the PBF code doesn't. Do you reckon it would be easy to do, and can somebody (Brett? Scott?) outline what you think would have to be done? Then I'll have a go at implementing that. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] No geometry for ways and relations after osmosis import into PostGIS
Hi, bigspoon wrote: I now see that the 'simple' schema script produces a ways and relations table that have no geometry associated with them on purpose. duh. The ways reference any node IDs in the nodes column. Unless of course you als execute pgsimple_schema_0.6_linestring.sql which creates a column named "linestring", the presence of which will then make Osmosis build the line strings for you. (It doesn't build polygons or lines for relations though.) 1. Rendering: It seems the advantage to using osmosis with PostGIS (over osm2pgsql and imposm) is to have a database where once can calc/apply faster diffs, updates or clippings using osmosis functions. Generally the advantage (or niche?) of Osmosis is that you retain the original data objects; you would thus tend to use Osmosis if it is important to you to have access to all OSM objects and all their tags, the object ID, version number and all that. Much of that is not required for rendering, a lot of it even is a burden. osm2pgsql (and to an even greater degree imposm) will optimise data for rendering, dropping what is not necessary for rendering. Fast extraction of bounding boxes is an important requirement for rendering too; indeed the databases generated by these tools will do so even faster than Osmosis-imported ones becaue there's less data in the first place. Are any osmosis schemas used by the community to do rendering (with mapnik or mapserver)? I'm not aware of anyone doing that (since, as explained above, someone whose primary aim is rendering would tend not to use Osmosis) but it would of course be possible. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Running a crop task to multiple output
Hi. Laurent Gregoire wrote: Hi all, Is it possible to crop an input osm file to multiple regions, and output each region to a different file, all in one pass? See http://blog.geofabrik.de/?p=75 for a big example. Since writing that blog post, I have introduced some optimisations (mainly placing --buffer tasks at the right locations, after lots of back and forth discussion and many helpful hints from Brett and Scott); I have been meaning to write an update to that post but haven't done it yet. But the basic syntax works, in any case. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] --used-node performance and a possible way to improve it
Hi, Jochen Topf wrote: I have thoght about these things quite a lot, too. I agree with Igor that we are bumping into the limits of what the current Osmosis pipelining model can do. We seem to be reaching a point where people are creating specialist programs to do things that are traditionally done with Osmosis - when until recently it was the other way round, you needed less and less specialist programs because Osmosis did it all. For example, there's "osmconvert" (http://wiki.openstreetmap.org/wiki/Osmconvert), a tool that can read/write OSM, apply diffs, do bounding box, polygon, and tag filtering, and people published benchmarks in the forum where that program was almost 5 times as fast as Osmosis. The program is certainly not a marvel of software design (6000 lines of rather unreadable C code in one source file, and it uses NO LIBRARIES except lz, not even an XML parser or a geo library or Protobuf code - all hand-coded) - but still, it seems to have a growing user base because it is fast and can be easily installed/compiled everywhere. (Also, the tag filtering seems to be a bit easier than the tee/tf/used-way/used-node/merge orgy that you currently need to perform in Osmosis if you want to extract something like "all forests".) It used to be that, compared to other means of achieving the same, Osmosis was perhaps minimally more clumsy and perhaps minimally slower, but I always used, and still use, Osmosis for everything because it was so convenient to have this one multi-tool. (Maybe except when I was using grep.) - But when others start to do the same in a fifth of the time then maybe it is really time to think; Split Osmosis into two parts, one is the low-level code for OSM objects, readers and writers etc. And one is the "control plane" code that stitches pipelines together etc. ... maybe the approach you are sketching would also allow one to somehow hack together "shortcuts" that would allow the building blocks to be used without anything not necessary for the task at hand. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Merge 2 maps
Hi, Martin wrote: maybe it's a bug. I've already posted it on the german section of the osm-forum (http://forum.openstreetmap.org/viewtopic.php?id=11591). I'm trying to merge 2 maps from geofabrik. But I still get not connected ways. I am in the process of changing the Geofabrik extracts to use the "fullWays" option so that ways crossing the clipping border will be included in full. But at the moment they still may be cut off at the border, so Osmosis will receive two ways with the same id, but one containing some nodes from area 1, the other containing nodes from area 2. If you're trying to do an Alsace/Baden-Wurttemberg extract, you must use europe.osm.pbf as your base file and then cut out the area of interst. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] using osmosis to import full history planet
Matijn, Martijn van Exel wrote: Thanks, I hope so too. We're trying Frederik's history extract script[1] to get an extract for the Amsterdam area using the same BBOX we fed to osmosis. I'm curious if the result will be different from the osmosis output. Frederik, what was your reason for writing this script? Thoughtlessness. It didn't occur to me that Osmosis would process a history planet without complaint. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Scott, On 02/18/11 14:15, Scott Crosby wrote: I used two bitsets for every output file. One indicating which nodes were already output and another (built when processing ways) indicating what node ID's were missed and will need to be grabbed on the next pass. I have another two pair of bitsets for ways and relations. Thats around 400mb of RAM per output. I see. Means I overestimated a little ;) Really, almost everything you want is already done from the mkgmap splitter crosby_integration branch. Just make it support non-rectangular regions, and track output/missed entities with bitsets. I'd say about 4 hours and 2gb+400mb/region to generate as many outputs as you want. I'll have a look at that. * all nodes in the bounding box * all ways using any of these nodes * all nodes used by any of these ways even if outside * all relations using any of these nodes or ways o all nodes and ways used by any of these relations even if outside o but NOT all nodes used by a way drawn in through a relation. (The points marked "*" are what the API does; the API does not do the "o" marked points even though users could be interested in them.) I did not know that it was allowed to miss the things in "o". That makes the job easier. The API doesn't do them on purpose even though, having access to an indexed database, it could easily add the "o" points. Reason being that other relation members may not even be geographically nearby, and if you load a small part of European countryside you might not be interested in receiving the full detail about the Madrid-Peking long distance cycle route just because that happens to pass through your area of interest ;) There's a second approach that could split the entire planet into extracts, simultaneously, in 2-3 minutes; a new planet format that I'm working on that is geographically sorted. Progress is ongoing, but slow. Sounds interesting - and difficult, in case of the Madrid-Peking long distance cycle route ;) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Hi, On 02/18/11 12:47, André Joost wrote: You can use minutely diffs, which gives you a permanently current database. Provided that applying a minutely diff (possily even while the extraction requests are running) does indeed take less than a minute. Which, I repeat myself, I believe to be unlikely. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Igor, On 02/18/11 11:40, Igor Podolskiy wrote: just a random thought: what's wrong with using --dataset-bounding-box? Importing the planet file into a database and doing a bunch of queries against is equivalent to creating a single disk buffer for all bb tasks (the database _is_ the disk buffer, if you want). This still isn't very elegant as it requires two passes (import into DB and export to PBFs) but is IMHO more elegant than selection lists. I currently use bounding polygons so I'd have to add a --dataset-bounding-polygon task for that but that should be possible. The major problem is finding a database that would be able to import the data, index it, and fulfil my requests, all within less than 24 hours. From what I hear, PostGIS takes a week for the import step alone but I want to produce fresh extracts every day. So I would have to import into PostGIS and then apply the daily diffs, but I could only do that if applying a daily diff and making all the bounding-polygon requests would take much less than a day and somehow I suspect that's not going to work. Don't get me wrong, maybe there's a reason why dataset tasks are unsuitable - but I've not found any... Maybe the dbb task itself needs improvement - but in this case, this should be the way to go. I doubt that a performant solution can ever be found with PostGIS but I'd love to hear from someone who can prove me wrong. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Improving completeWays/completeRelations performance
Hi, in the long run I'd like to change the Geofabrik extracts so that they have the completeWays/completeRelations feature enabled. It's a pain because that totally breaks the elegant and well-performing streaming mode in Osmosis but it would really make the extracts more usable, and more in line with what people get from the API. My biggest concern is the disk space used for temporary storage. If I read things correctly, a temporary storage of the input stream is created for each --bb or --bp task. So if you do something like osmosis --rb planet --tee 5 --bb ... --wb europe --bb ... --wb asia --bb ... --wb america --bb ... --wb australia --bb ... --wb africa then you will temporarily have 5 copies of the planet file lying around. So while, if there was only one copy of it, I could still hope to make use of linux file system buffers and a lot of RAM to soften the negative impact of file storage, that will kill performance for sure. I wonder if there is a way to at least reduce this to *one* temporary storage. The easiest thing I could imagine would be a new "multi-bb" (or "multi-bp") task that basically combines the tee and bb. That would be less elegant and would probably also be less efficient because it would not use multiple threads, but it could easily use one shared temporary storage. But I've been thinking: With the the high performance of PBF reading, a two-pass operation should become possible. Simply read the input file twice, determining which objects to copy in pass 1, and actually copying them in pass 2. I'm just not sure how that could be made to fit in Osmosis. One way could be creating a special type of file, a "selection list", from a given entity stream. A new task "--write-seelction-list" would dump the IDs of all nodes, ways, and relations that were either present or referenced in the entity stream: osmosis --rb planet --tee 5 --bb ... --write-selection-list europe.sel --bb ... --write-selection-list asia.sel --bb ... --write-selection-list america.sel --bb ... --write-selection-list australia.sel --bb ... --write-selection-list africa.sel Then, in a second pass, one would use a new task "--apply-selection-list" to actually filter the objects: osmosis --rb planet --tee 5 --apply-selection-list europe.sel --wb europe --apply-selection-list asia.sel --wb asia ... The selection lists would be quite big, and would for efficiency have to be fully kept in memory, so the above jobs could probably eat 20 GB of RAM easily (1.5 billion objects, IDs have 64 bit, hash table overhead). Also, what I have sketched above would be able to give you * all nodes in the bounding box * all ways using any of these nodes * all nodes used by any of these ways even if outside * all relations using any of these nodes or ways o all nodes and ways used by any of these relations even if outside o but NOT all nodes used by a way drawn in through a relation. (The points marked "*" are what the API does; the API does not do the "o" marked points even though users could be interested in them.) Does anybody have any thoughts about this; maybe a different approach still? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Suggestion: option to include changesets in historic bbox extracts
Hi, On 02/10/11 10:45, Eric Marsden wrote: I use osmosis to produce bbox extracts from the experimental "full history" planet dumps. I wasn't even aware that this was possible! How does Osmosis select nodes, ways, and relations for inclusion when using history data? If version 3 of a way contains node x which is in the bbox, but versions 1 and 2 didn't contain any nodes, what will happen - will it include v3, or v3 and all later versions, or all versions of that way? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Ring Grouping outer/inner information
Hi, Peter Schrammel wrote: according to http://wiki.openstreetmap.org/wiki/Relation:multipolygon/Algorithm I have to know of which type a way is (inner,outer). I actually can't figure out how to get this information out of my osmosis database to build correct polygons and do some checks. Am I missing something? Don't I need this info to build multipolygons? The algorithm actually suggests that you find this out by yourself instead of relying on the inner/outer roles. But if you want to use these roles, they are not properties of the way, but they are roles contained in the relation member table (depending on what schema you are using). The same way could be inner to one, and outer to another polygon. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Scott Crosby wrote: I've run a test over all of the files, computing a CRC of each block, and prettyprinting every block that differs. The two files are identical except for the reported osmosis version: That's comforting (to me at least). Does the same breakage happen of pbf2osm is run on the same file in windows? I'll have to defer that to Henning - I am not sure if anyone has even built pbf2osm on Windows? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, now I have the files ready for your inspection, on http://www.geofabrik.de/tmp/ MD5 sums: 9d278ee963925ed34701431b8573e0bf europe-created-on-linux.osm.pbf c4186d9197225e97f5bc935be09881e8 europe-created-on-windows.osm.pbf file sizes: 4448335290 2010-12-14 14:24 europe-created-on-linux.osm.pbf 4448335278 2010-12-14 11:38 europe-created-on-windows.osm.pbf The "europe-created-on-linux" was created by me with Osmsois SNAPSHOT-r24405, a simple --write-pbf task with compress=deflate. Henning says that simply trying to convert that to XML with Osmosis fails on his windows machine, while it works all right here. The "europe-created-on-windows" was produced from the .osm.bz2 that I used, and seems to work under Windows. (More precisely, what I am doing is I read a planet, cut out Europe, and use a --tee 2 to write a .osm.pbf and a raw XML at the same time; that raw XML is later compressed to bz2 and that was then used by Henning to produce his PBF.) I'm sorry that the files have to be so large but people report no problems with smaller files... Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Peter Körner wrote: I planned to do sth. for this tomorrow: Download the current europe.osm.bz2, convert it to pbf. Download the current europe.osm.pbf Great. - User aighes on talk-de seems to be trying out the same with today's files. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Scott Crosby wrote: If they do that conversion on windows, does that generate a pbf that has the same md5sum as the geofabrik generated file? (And if not, then I'd like both of those files!) I'll try to assemble a proper test case with the help of one of the windows users. * converting the Geofabrik-generated .osm.pbf to XML will copy exactly 10,600,000 nodes to the XML file and then stop, without error, adding a proper "" but no ways or relations. Is this with pbf2osm or osmosis or both? This is when trying an Osmosis-on-Linux-generated .osm.pbf to .osm with Osmosis-on-Windows. pbf2osm is not part of this particular problem. Is the 10,600,000 nodes from different files or the same file? The resulting XML file has 10,600,000 nodes. Does anybody have an idea why this might be? The europe.osm.pbf can be processed without problems on Linux, with Osmsis as well as with the mkgmap splitter. Did the users having this problem successfully work with geofabrik extracts from last week or last month on their windows machines? Yes, they claim they did. The only change that I made is upgrading from 0.37 to latest SVN, and adding the cascadingRelations=true option which should really only influence relations. Other than that, bizarre. I had some random crashes with the mkgmap splitter, until I found this bug: http://bugs.sun.com/view_bug.do?bug_id=698 and started running with -XX:-ReduceInitialCardMarks I'm wondering if this is some issue with corruption with the JDK that's rare and causes spurrious correlations. That would explain why smaller files are less likely to hit the issue, whatever it is. I think one of the error reporters indeed seems to have some sort of hardware (RAM) fault but it is unlikely that all of them do. I don't have development environment on windows to do any experiments of my own. Same here although I might be able to conjure something up on an old laptop. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] pbf problems under windows
Hi, I have a number of reports where Windows users complain that they cannot process my (osmosis-generated) .osm.pbf extracts under Windows. For some it seems to break with country extracts but it seems that everybody has problems with the Europe extract. These problems seem to affect simple Osmosis operations (like --read-pbf --write-xml) as well as the mkgmap splitter. The seem to occur with 32bit and 64bit machines. Users who researched the issue in more detail said on talk-de that: * using the Geofabrik-generated .osm.bz2 and converting it to .osm.pbf on Windows will yield an europe.osm.pbf that *can* be processed on Windows. * converting the Geofabrik-generated .osm.pbf to XML will copy exactly 10,600,000 nodes to the XML file and then stop, without error, adding a proper "" but no ways or relations. Does anybody have an idea why this might be? The europe.osm.pbf can be processed without problems on Linux, with Osmsis as well as with the mkgmap splitter. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] proper usage of osmosis when writing changes to osm api db
Samir, Samir Faci (Dev) wrote: I'm not sure if this is helpful or not It was well intended but the original author inquired about an API DB database, which is populated directly by osmosis, without osm2pgsql. It uses a different database schema and does not have a "slim" mode. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Another small Osmosis PBF problem
Hi, whenever relations without userid are temporarily stored by Osmosis, they end up changed somehow. If I take this file: changeset="1" /> and convert it to PBF: osmosis --rx test.osm --write-bin test.osm.pbf I end up with a 58 byte file that can be converted back to the above with either osmosis or pbf2osm. But when I run osmosis --rx test.osm --sort --write-bin test.osm.pbf then the resulting file has 65 bytes. It can still be converted back to the above with osmosis, but if I run pbf2osm on it I get: timestamp="2010-06-25T11:33:43Z"/> Somehow the information that user ID and user name were unset has been lost when the relation was temporarily stored. Can it be fixed? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improve relation extraction handling
Brett, Brett Henderson wrote: I'm thinking of adding a "cascadingRelations=yes" flag to the --bp/--bb tasks that would * process relations not containing other relations as usual, * throw all other relations onto a stack and process them in a suitable order afterwards I'd try to use the existing completeXXX options as a starting point. I have done that now and committed the enhancements in r24342. I ran some tests of my own and I think everything works as desired. I'd love to amend the tests as well but it seems I am lacking either the proper system or the proper brains to run the existing tests at all. The BoundingBoxFilterTest and PolygonFilterTest work in 0.38 and they now fail in trunk; the AreaFilterTest fails in 0.38 as well as in trunk: test: [mkdir] Created dir: /home/fred/svn.openstreetmap.org/applications/utils/osmosis/tags/0.38/areafilter/report/test [junit] Running org.openstreetmap.osmosis.areafilter.v0_6.AreaFilterTest [junit] Tests run: 5, Failures: 5, Errors: 0, Time elapsed: 0.348 sec How do I find out why exactly these tests fail? Running ant with -v only gives lines like [junit] junit.framework.TestListener: startTest(testProcessBoundContainer1) [junit] junit.framework.TestListener: addError(testProcessBoundContainer1, org.openstreetmap.osmosis.areafilter.v0_6.PolygonFilter.(Lorg/openstreetmap/osmosis/core/filter/common/IdTrackerType;Ljava/io/File;ZZZ)V) [junit] junit.framework.TestListener: addError(testProcessBoundContainer1, null) [junit] junit.framework.TestListener: endTest(testProcessBoundContainer1) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf syntax?
Hi, Samir Faci (Dev) wrote: osmosis --read-replication-interval --simplify-change --write-pbf You'd want something like osmosis --read-replication-interval --simplify-change --read-pbf oldfile.osm.pbf --apply-change --write-pbf newfile.osm.pbf If your area of interest is not the whole world, you would add a --bb or --bp after the --apply-change. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Improve relation extraction handling
Hi,
I'm a major user of the --bp task in Osmosis, for the nightly
Geofabrik country extracts. I cannot use the complete-something flags as
that would take far too much time, and on the whole the
"clip-incomplete-entities" mode seems not to be too bad for users.
There's one thing that has popped up on this list however ("lost
relations while clipping") which is an increasing concern to our users
and which I hope I can fix. The problem is that with cascading
relations, Osmosis will only include a relation if one of its children
has been included already, so if
node N1 is inside the clipping area,
relation R1 contains node N1,
relation R2 contains relation R1
then relation R1 will alway be included in the output, but relation R2
will only be included if it comes after R1 in the input file.
Now while using the temporary storage for nodes and ways would be too
expensive for the nightly Geofabrik build, I think that temporarily
storing relations would be acceptable (as there aren't too many). I'm
thinking of adding a "cascadingRelations=yes" flag to the --bp/--bb
tasks that would
* process relations not containing other relations as usual,
* throw all other relations onto a stack and process them in a suitable
order afterwards
I am aware that determining a "suitable order" could be difficult if you
have many levels of cascasing (or even circular) relations, and that the
output file would not be ordered properly, but I think such an option
would mean a big improvement at a relatively low cost.
1. Any comments?
2. Anyone done something like that already?
3. Is any of Osmosis' internal storage mechanisms suitable for buffering
relations in this way or do I have to invent something new?
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [OpenStreetMap] #2620: osmosis can't parse un-uploaded 0.6 .osm files (from JOSM, other tools) due to requiring version=""
Hi, Ævar Arnfjörð Bjarmason wrote: I've only submitted one very minor patch to osmosis so I'm not at all familiar with its architecture, but is it really so hard to just ignore version fields if they're negative integers? Is that the only problem? Would proper processing of JOSM-generated files not also include * being able to work with missing date, username, userid attributes * ignoring objects that have action="delete" * alternatively carrying through any existing action attribute? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with 0.36
Hi, Brett Henderson wrote: I should read all of my emails before responding :-) And I should read the change logs before hunting a bug ;) Don't rush it with the 0.37, I can live with using different versions for the different tasks I have to perform for now. Then again, I always thought I was just doing plain ordinary stuff that everyone else does too but if what I do somehow uncovers bugs then maybe I should really try out the 0.37 snapshot with binary format and see if I can find any bugs there ;) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with 0.36
Hi,
Frederik Ramm wrote:
I'll debug this further - currently unsure if there's something wrong
with the input data or what. The problem occurs about 1h50m hours into
processing the job. so it looks more like it is towards the end.
Ok, that was easy; it looks like clipIncompleteEntities is broken in
0.36 because Way.getWayNodes is documented as returning a read-only
list, and AreaFilter does this if clipIncompleteEntities is true:
// Remove node references for nodes that are unavailable.
for (Iterator i = filteredWay.getWayNodes().iterator();
i.hasNext();) {
WayNode nodeReference = i.next();
if (!availableNodes.get(nodeReference.getNodeId())) {
i.remove();
}
}
I'll continue using 0.36 for downloading the hourly diffs and go back to
an earlier version for other processing.
Bye
Frederik
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Problem with 0.36 (was: Re: [OSM-dev] Osmosis hourly replication broken?)
Hi, moving this from dev to osmosis-dev: Frederik Ramm wrote: 0.36 seems to work. Anything prior, probably not. Then again hardly anyone uses the hourlies I guess. I'm seeing another problem, after upgrading to 0.36. It's not related to the diffs any more, these are fine; but my job that cuts out continents and worked fine with 0.32: osmosis --fast-read-xml-0.6 current-planet.osm --buffer --tee 6 --bp file=africa.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 africa.osm --bp file=asia.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 asia.osm --bp file=australia-oceania.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 australia-oceania.osm --bp file=central-america.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 central-america.osm --bp file=europe.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 europe.osm --bp file=south-america.poly clipIncompleteEntities=true idTrackerType=BitSet --write-xml-0.6 south-america.osm now results in the following error: SEVERE: Thread for task 2-buffer failed java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableCollection$1.remove(Collections.java:1028) at org.openstreetmap.osmosis.areafilter.v0_6.AreaFilter.emitWay(AreaFilter.java:277) at org.openstreetmap.osmosis.areafilter.v0_6.AreaFilter.process(AreaFilter.java:191) at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(WayContainer.java:60) at org.openstreetmap.osmosis.areafilter.v0_6.AreaFilter.process(AreaFilter.java:104) at org.openstreetmap.osmosis.core.tee.v0_6.EntityTee$ProxySinkSource.process(EntityTee.java:124) at org.openstreetmap.osmosis.core.tee.v0_6.EntityTee.process(EntityTee.java:70) at org.openstreetmap.osmosis.core.buffer.v0_6.EntityBuffer.run(EntityBuffer.java:74) at java.lang.Thread.run(Thread.java:636) I'll debug this further - currently unsure if there's something wrong with the input data or what. The problem occurs about 1h50m hours into processing the job. so it looks more like it is towards the end. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] read-change-interval stuck
Hi, Nakor wrote: I have a setup where I first run osmosis to download changes (--rri) then in a second step stuff the result into osm2pgsql. About once every 1-2 months, this breaks, and osmosis hangs forever. When I notice it a few hours later and strace the osmosis process, I find that it somehow tries to read something from the network and only gets timeouts. Do you mean you download to an xml file and then feed the xml file into the database? Yes. That way I can always be sure not to start loading something into the DB before it has fully downloaded. Probably paranoia on my part, since transactions should make sure everything is right, but ... Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] read-change-interval stuck
Hi, Nakor wrote: I have a local database that I update with daily diffs. This usually takes aorund I hour. I started today's update 6 hours ago and it is still running. Where could it come from? I have a setup where I first run osmosis to download changes (--rri) then in a second step stuff the result into osm2pgsql. About once every 1-2 months, this breaks, and osmosis hangs forever. When I notice it a few hours later and strace the osmosis process, I find that it somehow tries to read something from the network and only gets timeouts. Incidentally, last time this happened was yesterday (Thursday) morning at 9:14 MEST. I then kill and restart the process and it usually works. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] No Default pipes available?
Hi, Markus wrote: bin\osmosis.bat --rri --simc --rx planet.old.osm.gz --ac --wx planet.new.osm.gz but it didn't finish for some reason. Any indication what the reason might be? If run on a full planet, this can easily take a few hours. and now I have a update.osm.gz file. One further question is what command to use to merge a planet file with the update file? That would then be --rxc update.osm.gz --rx planet.old.osm.gz --ac --wx planet.new.osm.gz But again, that will take a while. If you have more than one hard disk, things can be improved by having it write the new file to another disk. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] No Default pipes available?
Hi, Markus wrote: bin\osmosis.bat -–wxc update.osm.gz You'd have to do a bin\osmosis.bat --rri --simc --wxc update.osm.gz or if you want to apply it to a planet file directly, bin\osmosis.bar --rri --simc --rx planet.old.osm.gz --ac --wx planet.new.osm.gz Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Project Restructure
Brett, Brett Henderson wrote: I haven't created a 0.36 detailed usage page on the wiki yet but will do so before releasing. There are a few other random goodies in the latest development release that I'd like to release in the near future. Will there be significant changes in ordinary Osmosis usage, or is the change largely transparent for users? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error while applying updates to planet.osm
Tyler, sounds like either your planet or your change file is corrupt. Check the files with "xmllint --stream". Re-download the one that is broken. > Next, I ran the commands below to get my updates and simplify them: > osmosis --rri workingDirectory=. --wxc update.osm.gz > gzip -d -c update.osm.gz | /usr/local/osmosis/bin/osmosis > --read-xml-change file=/dev/stdin --simplify-change --write-xml-change > file=- | gzip -9 -c> update.unique.osm.gz > > Finally, I attempted to apply these changes to my planet.osm file: > bzcat planet-100618.osm.bz2 | /usr/local/osmosis/bin/osmosis > --read-xml-change file="update.unique.osm.gz" --read-xml file=/dev/stdin > --apply-change --write-xml file=- | bzip2 -c > planet.osm.bz2 All that can be shortened to this one line: bzcat planet-100618.osm.bz2 | osmosis --rri --simc --rx - --ac --wx - | bzip2 > planet.osm.bz2 But it is probably good to do it in single steps while still testing the process. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] bounding-box does not support data provided by default pipe
Hi, Oliver Schrenk wrote: >> bzcat $(MAPS.OSM)/planet-080611.osm.bz2 | $(TOOLS.OSMOSIS) >> --read-xml-0.5 enableDateParsing=no file=/dev/stdin --bounding-box >> top=$(TOP) left=$(LEFT) bottom=$(BOTTOM) right=$(RIGHT) --write-xml >> file=080611.osm Perhaps throw in a --migrate just before the --bounding-box! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] bounding-box does not support data provided by default pipe
Hi, Oliver Schrenk wrote: > Has 0.3 been deprecated/removed? Can I use an older version for extracting bb > from 0.3 maps? What's the case for 0.4 maps? 0.3 and 0.4 are basically the same and can be processed with the same software. Osmosis never supported either, it starts at 0.5. There is a tool in SVN which converts from 0.4 (and therefore also 0.3) to 0.5 (applications/utils/conv05). However that will probably not run on the whole planet file, or take too long; you will have to use applications/utils/osm-extract/planetosm-excerpt-area.pl to cut out an area from the old file. That perl script either supports 0.4 in its current form, or maybe you'll have to check out an older version from SVN. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postGIS: bbox columns?
David, David Carmean wrote: > Thanks for the clarification. Indeed the linestrings are the primary > desired feature in my use cases (as will be polygons constructed from > the closed ways). It is easy to make polygons from closed ways, but quite hard to make polygons from multipolygon relations (at least it's not something that you can do well with a simple st_buildarea). Depending on what exactly you want, you may want to look into the file we're offering (weekly update) at http://download.geofabrik.de/polygon/, it is a PostGIS dump with multipolygon geometries for every single closed way *and* multipolygon relation in OSM. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Unable to write String to the store
Hi, Nakor wrote: > I was trying to extract a bounding box from planet-100519.osm.bz2 and > after one hour or so osmosis stopped running with the error message at > the end. You seem to have used one of the --complete-something options. These are real performance killers because they have to cache the full planet in a temp file. Don't use them unless you really have to. If the extract you are making is relatively small, then it can make sense to first do an extract with --clip-incomplete-entities for something that is slightly larger than the bit you want, and then make an excerpt *from that* with the --complete option. > I have 175 Gb free on the disk where the planet file resides. Is this > not enough? Or is osmosis using another disk for its temporary files? 175 GB is enough because the temp files are gzipped. But it uses /tmp unless you export JAVACMD_OPTIONS="-Djava.io.tmpdir=/my/pretty/little/tempdir" before you run it. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postGIS: bbox columns?
David, David Carmean wrote: > As described above, this index can be used to perform a quick first-pass > query based on just the table indices (proxy bounding boxes) and present > the main query with a pre-filtered set of candidates. I probably have not made myself clear enough. It is obvious that having a proper linestring in your table, together with a GIST index, beats having just a bbox, and it is unlikely that any application will be unable to make use of the index. HOWEVER, it takes much longer to import the full linestrings than to just import a bbox in the first place. So if your application, for whatever reason, does not need pre-built geometries, but just an overall idea of where in the world a certain way is, then the bbox builder mechanism in Osmosis allows you to skip linestring generation altogether, saving time and storage space. The bbox column does not present added value if you intend to import the full linestrings. It is only useful if you want a lean database without full linestrings. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postGIS: bbox columns?
Hi, David Carmean wrote: > Who/what uses the ways.bbox column? Is there something/somebody > that cannot use the linestring gist index, but instead requires > an explicit bounding box geometry? Is the bbox column not meant to be a quicker alternative to a full linestring (quicker in importing, not quicker in using)? I.e. you won't have a linestring to base your GIST index on if you use that as intended. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Can't load osm file to PostGRESql
Hi, Ibrahim Bouchrika wrote: > Your idea seams nifty, although it's giving an error on the --rx command. I don't know Osmosis well enough to figure out where the error comes from. Is it just an error in the syntax or is it because of the osm-file? > Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog. Can you make sure your empty.osm and your amsterdam.osm begin with the characters http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems importing .osm files into postgis
Brett, > I think i set up the database correcly is, I used the > pgsql_simple_schema_0.6.sql schema from osmosis. > I also used the 900913.sql file. > > What is the 900913.sql file for? That's an INSERT command that adds the EPSG:900913 projection to PostGIS's spatial_ref_sys table. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Cryptic error message
Hi, > org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task > 2-bounding-box does not support data provided by default pipe stored at > level 1 in the default pipe stack. You must either "upgrade" your data to 0.6 by adding a "--migrate" after the --read-xml-0.5, or you must use the 0.5 version of all tasks by adding "-0.5" to their name (--bounding-box-0.5 etc.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis Release 0.35
Brett, Brett Henderson wrote: > There are no major new features, just an assortment of bugs That's how we love our Osmosis ;-) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet import time
Hi, Jens Lehmann wrote: > A solution would be to include it in the planet or provide it as a > separate file, which can be loaded into the database. Well it isn't even in our central database so it would have to be computed at the time we dump the planet file which already is an expensive enough operation ;-) > Can the long running bbox query be aborted/killed without resulting in a > corrupt database or do I have to start the import from scratch for > testing the Java based solution? I'm not sure if Osmosis uses one transaction for the whole import or several. Sure you can kill and you will not have a corrupt database but not an usable one either I believe! Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet import time
Jens, Jens Lehmann wrote: > UPDATE ways SET bbox = (SELECT Envelope(Collect(geom)) FROM nodes JOIN > way_nodes ON way_nodes.node_id = nodes.id WHERE way_nodes.way_id = ways.id) > > How long will this query take? Is there a way to monitor its progress? > Why isn't the content of the column just copied in instead of > recomputing everything from scratch? Well you have to compute it somewhere because it is not in the planet. But you can use the osmosis --wp option enableBboxBuilder to have the boox built in Java and then copied into PostGIS rather than having it built in PostGIS. Doing it in Java is faster but I don't know by how much. See http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql_.28--wp.29 Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: > The bottom line for my investigation of mapping tools is that I need > to have a local database of the world to perform lookups and > rendering of maps on a "standard" Windows based desktop computer in a > reasonably quick fashion (i.e. pan/zoom operations complete in 1 to 5 > seconds). > > Performance wise, for my scenario, I'm not sure which would be > better: rendering as tiled images or rendering as SVG. > My application will be using WPF which can render XAML based graphic > paths, which can be derived from SVG. There is a utility called > Osmarender that I've heard of and I understand it can render to SVG. There is a rather large gap between your ambitions and your current knowledge ;-) Mapnik can be made to render any section of the planet to an image in 5 seconds but you will need to tweak the style sheet somehow. Rendering to XML is possible but the XML that Mapnik produces is highly complex and it is not unlikely that you'll stretch the capabilites of your library with it (go to www.openstreetmap.org and use the export tab to generate SVG, and try that out). Osmarender is too slow for your purpose. There's no reason why you must use tiled images. You can also make Mapnik render exactly the section you need. > I think > Osmarender would work with a MySQL database. No, Osmarender does not use databases at all. > Would you guess if > rendering as SVG and then converting to XAML be faster than working > with images rendered on-the-fly? My guess is: slower by a factor of 5. > If I go with rendering as tiled images, then I have to consider > rendering on-the-fly or pre-rendering in advance. It is probably a bit painful to get Mapnik + Postgres running properly on a Windows machine, *especially* if that machine is somewhere at your user's site and not fully controlled by you. Tiles are, in contrast, dead-easy. They only have one drawback: There are a lot of them. > If I pre-rendered > in advance, then I suppose I would run Mapnik over the entire > osm2pgsql generated database and that would in turn generate yet > another database of tiled images? No, it generates individual files, not a database. If you want to cover the whole planet at zoom level 18 you will need to produce about 100 billion tiles and they will together use about 100 Terabytes of storage. This means you will have to think more about what you really need. > Any thoughts on size of a database > of 256x256 PNG images for entire "planet.osm" contents? Or, is > rendering on-the-fly going to allow me to perform my rendering of > user requests in a few seconds? Probably, if nobody else uses the machine. Can you not simply download tiles from a web service? That would solve a hell of a lot of problems for you. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: > You mentioned that osm2pgsql needs more RAM (16GB+ during > import). Do you know how much RAM would be required for normal > lookup and rendering usage with either slim or non-slim usage? For normal lookup it doesn't make a difference. It should run on a 4 GB machine without problems. > Also, I think a schema that didn't have history would be better for > my purposes (less data) as I only need lookup and rendering > capabilities (i.e. read-only). I think then osm2pgsql is going to be your friend. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: > I appreciate your quick response! I am not surprised by your answer. > Any idea on how much disk space would be required by a PostGIS > database that would hold the entire planet.osm file data? I think it would be between 100 and 150 GB for either schema; but only 30 GB if you do option 2 (osm2pgsql) in non-slim mode. In non-slim mode, osm2pgsql needs more RAM (at least 16 GB during import) but produces a smaller, and non-updatable, database. > I'm not sure what the "simple schema" is. I assume that is different > than resolution #2 that you indicated in your reply? Yes, osmosis and osm2pgsql use different schemas. Osmosis supports the APIDB schema and the "Simple" schema while osm2pgsql has its own schema, available in "slim" and "non-slim" varieties. From a data modeling perspective - and I'm on somewhat thin ice here, happy for anyone to chip in and confirm or protest - the APIDB schema is the most powerful because it has all the data plus history. The "Simple" schema, as well as the Osm2pgsql slim-mode schema, still have a full OSM data snapshot (and you could theoretically convert from one to the other, or from APIDB into one of them) but they don't have history. The non-slim osm2pgsql schema, lastly, carries only a processed selection of OSM data (exactly what is needed for the map - nothing more). You could theoretically convert from any of the others to non-slim osm2pgsql, but never in the other direction. > I assume that > in answer #2, there is a standard schema that Mapnik assumes must be > in place in the database it will query and that schema is different > than the "simple schema" specified in answer #1? Yes. To me more precise, Mapnik itself can be made to work with any schema, even the "Osmosis simple" schema, but the schema created by osm2pgsql and the style sheet commonly used in OSM work together and if you were to use Mapnik with the Osmosis simple schema you'd have to write your own style sheet. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: > My question concerns how to do this. One option would be to use Osmosis > to generate a local database containing the XML from “planet.osm” such > that it could be queried in a variety of ways quickly. However, for my > simplistic use case of simply extracting a bounding box worth of XML, > would Osmosis perform this operation quick enough such that a user of my > application could specify the bounding box, Osmosis would extract that > bounding box directly from the “planet.osm” file No. That requires parsing the whole file and I have yet to encounter a machine that can do this under 1 hour. I can think of two ways how you could handle your problem. 1. Import full planet into a PostGIS database using Osmosis' "Simple Schema". Then use the -dbb task to extract data from that. I have not used that but the data extraction should run quickly. The process the extracted data with a renderer of your choice to yield an image. 2. Import full planet into a PostGIS database using osm2pgsql. Then use Mapnik directly to render an image for the area of interest. If all you need is a PNG map image or so, the second option is recommended. The first path is less traveled. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Database is empty after five days of importing
Hi, Klemens wrote: > That makes sense. My requirement is that I must continue querying the > old data while importing the new, larger data - which can take a long > time as we know. How can I achieve this? I would suggest to create a second database instance (same server process, different database name, same table names), import into that, then drop the old database and rename the newly imported. This will give you a short service interruption. If you cannot tolerate any service interruption, you can use the "pgbouncer" proxy which can be reconfigured to connect to another database while still serving active connections to the old database. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Database is empty after five days of importing
Hi, Klemens wrote: > So at some time in the future when I want > to import Europe into a DB that contains Austria I will have to create a > delta .osm first using osmosis? Possible, but impractical, because you would spend more time to create the delta than you save by re-using the existing data. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] help by merge 4 files
Jan, Jan Tappenbeck wrote: > can anybody tell me the reason why not do - here is the report: True, finding out the *acutal* problem among 100 lines of Osmosis error reporting is sometimes difficult. > SCHWERWIEGEND: Thread for task 1-read-xml failed > org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline > entities are not sorted This seems to be the problem. If you insert a "--sort" (with some Osmosis versions you may need "--sort-0.6" after *each* --rx spec then it should work. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Need help understanding osmosis OsmChange structure
Hi, Daniel Caris wrote: > Im trying to understand the structure behind the change sets (OsmChange) > produced by osmosis (in particular the daily updates). A little > confused in regards to the ordering of modify and create xml-nodes, what > the changeset id’s refer to, what version numbers refer to, and the > general order in which changes (modify, create or delete) are processed in. If you don't know what a changeset or an object version is, then what are you trying to do with OsmChange files? The ordering is not specified, but generally the order is first nodes, then ways, then relations, and among each group ascending by id then version; you will find as many modify/create/delete xml nodes as necessitated by this order. Which order changes are processed in depends on whoever processes them and for what purpose. A typical Osmosis --apply-change task will consume one stream of entities and one stream of changes and expect them both to be sorted, applying changes one after the other. Changesets are an OSM concept to group a number of changes and apply a comment and possibly futher information to them. There is no strict relation to OsmChange files; all changes contained in one Changeset could be expressed as one OsmChange file, but you could also have one OsmChange file that describes everything changed within a certain time and that might only contain part of a Changeset. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [RFE] Add bounding shape to OSM dumps
Hi, Apollinaris Schoell wrote: > osmosis supports 2 options to keep ways and relations intact. but geofabrik > extracts don't use it as far as I know. > completeWays > completeRelations That is correct: These options exist, and are not used, because if we'd use them, the nightly build would take something like three days instead of six hours. Osmosis works very well in streaming mode and these options make it impossible to stream - Osmosis needs to create a temporary copy of the full dataset and is rather inefficient at it. We typically do something like osmosis --rx file.osm --tee 20 --bp file=country1.poly --wx country1.osm --bp file=country2.poly --wx country2.osm... which would causes Osmosis to make 20 temporary full copies of the data and write them to disk in "completeWays" mode. We use "clipIncompleteEntities" which means that Osmosis removes references to those nodes outside the polygon from any ways or relations. It would be good if Osmosis could somehow flag the clipped entities so that processing software could at least know that there is something wrong, or incomplete, with them. Adding the actual polygon used for clipping could of course be done but it will not automatically enable proper filling. Assume this: | +-|+-+ | | | | | |<--- filled area +-|+-+ | |<-- clipping boundary After clipping with clipIncompleteEntities, this will lead to | |+-+ || | || | |+-+ | | Even if you know where the clipping boundary is, you cannot extend the object towards that boundary properly because you are missing the nodes beyond the boundary. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
Hi, Simon Nuttall wrote: > 2010/1/10 Simon Nuttall : >> I think both these points are more a message for geofabrik.de than me >> because I'm using the diff's downloaded from their site. > > Sorry, I wasn't clear, I meant to say that CycleStreets uses the > ireland and great_britain extracts that we download from geofabrik.de, > and their system seems to be using 0.31.1 now. The problem was that in switching from --rci to --rri I didn't add the --simc filter, which led to duplicate objects in the extracts. It should be fixed now. I don't think that it was related to the Osmosis version, but switched to 0.32 nonetheless. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] completeRelations flag for --bp/--bb not working
Hi, consider this test case (run with 0.31.3): Cutting out from this a bbox that only contains node #1, such as osmosis --rx test.osm --bb left=1 right=90 top=53.5 bottom=52.5 --wx - one would expect to find in the output: * without extra flags: node #1, relation #1 referring to nodes #1,#2 --> pass * with clipIncompleteEntities: node #1, relation #1 refering to node #1 --> pass * with completeRelations: nodes #1,#2, relation #1 referring to both --> fail; instead, you get node #1 only, but relations #1 and #2. It seems that completeRelations always acts as "copy all relations from input to output no matter what" and has no influence on handling nodes and ways. Re-writing the same example to use ways, and the trying it with completeWays=true, yields the expected result. The Wiki documentation on completeRelations says: "Include all available relations which are members of relations which have at least one member in the bounding polygon." This describes neither the behaviour I had expected, nor the behaviour I have observed. I believe the intended behaviour is really "Include all available *objects* ..."- no? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Osmosis --apply-change vs --merge-change performance
Hi, say I have a planet file and want to apply two daily diffs to it. Which of the following would result in faster execution: --rxc diff1.osc --rxc diff2.osc --mc --rx planet.osm --ac --wx new.osm or --rxc diff1.osc --rxc diff2.osc --rx planet.osm --ac --ac --wx new.osm ? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways -> polygons?
Hi, David Carmean wrote: > Not a wizard, but "tenacious" (or OCD, take your pick). It's beginning to > seem like PostGIS has the tools to accomplish what I need. > Have already managed to have it find closed ways and create polygon > geometries > for them. I'll update y'all on my progress when I make some more. There is a procedural algorithm about making proper polygon geometries from relations specified here: http://wiki.openstreetmap.org/wiki/Relation:multipolygon/Algorithm If you have a PostGIS implementation that achieves the same, or roughly the same, feel free to add it! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways -> polygons?
Hi, David Carmean wrote: > If nothing else it appears that I could write a query using PostGIS functions > that will do what I want: > > http://postgis.refractions.net/documentation/manual-1.3/ch06.html#IsClosed Be advised that multipolygons may consist of any number of outer and inner ways which do not have to be closed themselves, they only have to form a closed ring. It will certainly be hard (but certainly not impossible) to do this with SQL alone. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways -> polygons?
Hi, Lennard wrote: > David Carmean wrote: > >> I found that osm2pgsql omitted several attributes that I require. Also, the >> schema created by osmosis is more suitable to my needs. > > If these attributes are the user, uid, version and timestamp, check out > http://trac.openstreetmap.org/changeset/19144 And if they are other attributes, then it is very likely that you will be able to modify the osm2pgsql style file to suit your needs. This is probably going to be easier than trying to build proper multipolygons from data imported by osmosis, unless you are a real SQL wizard. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways -> polygons?
Hi, David Carmean wrote: > I'm attempting to use OSM data, as imported to PostGIS via Osmosis, as > layers in a project in some GIS software; I've had some success crafting > queries to pull highways into a separate table, and am now working on > hydrologic features. My current roadblock is: how/where can I convert > closed OSM ways into polygons? And then of course there will be > multipolygons. Are you aware of osm2pgsql, a tool which has nothing to do with Osmosis but does roughly what you want? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Replication Changes
Hi, Brett Henderson wrote: > There is a new (as yet undocumented) task in osmosis called > --simplify-change which allows a full history diff to be collapsed into > an older style delta diff. In a scenario where you use --rri to retrieve a set of combined minute diffs, and if you continue with --wxc to feed stuff into osm2pgsql, would an intermediate --simplify-change then make sense, or does --rri do that automatically for you? I'ts amazing how far Osmosis has come in the 2.5 years since it was first announced (http://lists.openstreetmap.org/pipermail/dev/2007-June/005390.html - "There are a few features on my todo list ... Upgrade to support 0.4.") and how it is now a central element to much of what's going on in OSM. You were a total unknown at the time (or at least not present on the lists!) and initially I thought Osmosis was a slightly quirky tool that would probably not live long because its inventor would be likely to lose interest. Yet you produced something that profoundly influences the way most of us work with OSM. Hats off to that! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] duplication of nodes on apply-change
Hi, Maxim Dubinin wrote: > MD> But after I do: > MD> osmosis --read-xml-change file="data/updates/20091112-20091113.osc.gz" > MD> --read-xml file="data/rus.osm" --apply-change --write-xml > file="data/rus-new.osm" > > MD> I'm getting two versions of the same node in the output: Is it possible that I am resposible for this because I recently changed the sorting function to treat something with version=x+1 as "larger than" something with version=x, instead of "equal"? (SVN r18230, change to EntityByTypeThenIdComparator...) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Relations with > 65k members
Hi, Brett Henderson wrote: > I've updated the Relation class to use a 32-bit int instead of 16-bit > char when persisting temp data to disk (which I assume is where it > broke). The problem occured in the Relation's "store" routine where it attempted to make a Char value of the relation member count. I am, however, surprised that you say this has anything to do with persisting temp data to disk, as the problem occured when doing a --rri --wxc combination which only handled a rather small amount of data. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Has the behaviour of Osmosis changed regarding deletions in .osc?
Hi, Brett Henderson wrote: >I always thought that osmosis would only write the object "header" > (the deleted, and omit the tags and members. Now I see that it in fact > includes the full object. Has this behaviour changed some time, or has > this always been the case? > > > Nothing should have changed in that space. The Osmosis > --write-xml-change task will write everything that is passed to it in > the pipeline. So if the original change derivation task provides tags > and members they'll be written to the output file. Oh right, then that was my misunderstanding. I remembered from looking at the planet .osc files, and then tried it out by comparing two .osm files and thought the behaviour must have changed. Interestingly, the .osc files on planet seem to contain the full set of tags for deleted *nodes*, but no tags for deleted ways or relations. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Relations with > 65k members
Hi, Frederik Ramm wrote: > I thought about simply patching Osmosis to use an integer but was > informed by Jon Burgess that osm2pgsql (which is next in my toolchain) > would carp on > 65k members as well Turns out osm2pgsql in its default config allows only up to 32767 members. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Relations with > 65k members
Hi, we just had a problem with diff updates because someone created relations with more than 65k members and Osmosis tries to store them as a char which raised an exception. I did a quick hack that would simply drop the excessive members with a warning: http://www.remote.org/frederik/tmp/remove_excessive_members.diff I'm not checking that in. I thought about simply patching Osmosis to use an integer but was informed by Jon Burgess that osm2pgsql (which is next in my toolchain) would carp on > 65k members as well so I didn't bother - especially as it turned out that the relations in question were an error in somebody's automated import and have been meanwhile fixed by Lennard (they contained the same element over and over). Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Has the behaviour of Osmosis changed regarding deletions in .osc?
Hi, I always thought that osmosis would only write the object "header" (the http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Sorting (and test whining)
Hi, Brett Henderson wrote: >> This was necessary in order to process replicate diffs which sometimes >> contain the same object twice, causing the old implementation to >> complain about change streams "not being sorted" (as "being sorted", >> for Osmosis, means that every object has to be greater than the >> previous). > Which task was complaining? I had intended to create a new task which > would take a (sorted) change stream with full changes and collapse them > into a single change per entity. This new task would allow existing > tasks such as --apply-change to continue to be used as is. In my case it was --merge-changes which complained. I had downloaded some changes with --rri, and later did the same thing again and wanted to merge the two before then processing the unified result with osm2pgsql. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Sorting (and test whining)
Hi, I have committed a change to trunk that, when dealing with 0.6 files, also looks at the version of an object for comparison and sorting (not only type and id). This was necessary in order to process replicate diffs which sometimes contain the same object twice, causing the old implementation to complain about change streams "not being sorted" (as "being sorted", for Osmosis, means that every object has to be greater than the previous). If this causes problems for anybody then another option would have been to leave the sorting code unchanged and instead replace the "greater than" requirement in the sorted stream validator with a "greater or equal" requirement. Bye Frederik PS: Can we remove those tests the failure of which can be safely ignored? So that someone who checks out and builds Osmosis has a chance to build it without tests failing, and only if you do "ant --with-all-bells-and-whistles-and-exotic-features" do you get the current "something always fails but don't bother" functionality? ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] java.io.IOException: No space left on device
Hi, James P. Biagioni wrote: > When I try to cut a bounding box out of the Planet.osm file, with the > completeWays=yes option on, I receive the error: java.io.IOException: > No space left on device This option creates a large (10s of GB) temporary file. Is it possible that the temporary file is created on another disk partition where you do not have so much space? The size of the temp file is proportional to the size of the input file. So if you e.g. cut a little area out of the full planet, you will have a full-size temporary file. In such a situation performance will improve dramatically if you first cut out a slighty larger area than required WITHOUT completeWays=yes, and only then cut out the smaller area - in this case only the stuff you cut out first needs to go to the temp file: osmosis --rx planet.osm --bb clipIncompleteEntities=yes --bb completeWays=Yes --wx cutout.osm Of course this will lose any nodes which are outside of the larger bbox. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Dropping 0.5 tasks
Hi, Brett Henderson wrote: > I'm in the process of releasing 0.31 so that people have a known stable > version to download that supports 0.6. It still includes all of the > existing 0.5 tasks. > > So, would there be any objections to dropping 0.5 support? It's not > urgent, but I'd like to get rid of them at some stage. There are some minor tools out there which don't yet support 0.6 (some simply do a version compare and balk if it isn't 0.5 - easy for a programmer to fix but maybe not so easy for every user). Only recently someone complained that the Ruby osmlib does not support 0.6. I told them to simply run the 0.6 file through osmosis and --write-xml-0.5... but of course 0.6-to-0.5-translation could also be achieved by an XSLT one-liner I guess. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [OSM-dev] [OSM-talk] osmosis Problem with bounding-polygon and v0.6?
Hi, Karl Newman wrote: > If the fault does lie > with the bzip2 code, then I'm surprised you're the only one that's reported > a problem so far. Nobody in their right mind lets Osmosis compress or decompress bz2 ;-) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] ivy problem
Hi, I'm trying to build Osmosis trunk and I get an error from Ivy: [ivy:resolve] [FAILED ] org.codehaus.woodstox#wstx-lgpl;3.2.8!wstx-lgpl.jar: invalid sha1: expected=8898bbf8f5b17f5e59dde8324eceba6469171766 computed=833a57ee3dc3aa7ac347c8dea98dc38de42e4938 (10409ms) Hacker that I am, I grepped through the files for the "expected" value, hoping to be able to simply change it so that the above check succeeds, but I could not find it. Can somebody tell me how this can be fixed, or can I possibly build Osmosis without "wstx", whatever that may be? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
