Re: [osmosis-dev] Latest osc change file returns empty xml.
Hi, On 11/14/20 21:19, Dave F via osmosis-dev wrote: > I'm using osmosis --rri workingDirectory=. --wxc GB.osc.gz > I amended configuration.txt to point to the GB folder: > baseUrl=http://download.geofabrik.de/europe/great-britain-updates > > Running the command it completed it's task, but the osc file contained > an empty xml file: Is it possible that you have accidentally set your replication state to the current state? If you download the Great Britain file now, and set the appropriate state.txt value, then the expected result of running osmosis is an empty XML file because there are no changes available. As of this writing, the /europe/great-britain-updates/state.txt says sequenceNumber=2791 and running osmosis with that will yield an empty file, until such time as the state.txt on the server points to a sequenceNumber of 2792. > On other Q: > As the update interval for Geofabrik is 24 hours, should I amend this > value in the configuration.txt file to suit: > maxInterval = 3600 It does not make a difference; any maxInterval between 1 and 86400 should behave the same, as each would pick exactly one update per run. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis java.lang.IndexOutOfBoundsException
Hi, I think this was a problem of the Geofabrik download server. We accidentally generated our "cleaned" files (the GDPR conformant file without any user data) in a different way last night, completely dropping the UID/Username fields, instead of replacing them with 0 and empty strings as we did before. This tripped up osmosis and also older versions of osm2pgsql (<= 0.94). The pbf files are replaced with old-style, compatible files now. Nightly updates and bz2 files were not affected since they don't use the PBF format. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] osmosis wrongly claims to see UTF8 problem
Hi, On 01/11/2017 10:30 AM, Frederik Ramm wrote: > SEVERE: Thread for task 1-read-xml-change failed I was a bit over-eager in shortening the stack trace. Full detail: org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file x.osc. publicId=(null), systemId=(null), lineNumber=583379, columnNumber=90. at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114) at java.lang.Thread.run(Thread.java:745) Caused by: org.xml.sax.SAXParseException; lineNumber: 583379; columnNumber: 90; Invalid byte 2 of 4-byte UTF-8 sequence. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:109) ... 1 more Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence. at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown Source) at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source) at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) ... 11 more -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] osmosis wrongly claims to see UTF8 problem
Hi, I have recently had osmosis in --rri mode refuse to apply an update it had downloaded from OSM, claiming there was an UTF8 error in the file. I looked and looked but the file was fine, passed UTF8 and XML validity checks. I tried to isolate the line that gave me the "error" but isolating it made the problem go away. Only including the 583379 previous lines makes the error occur. So I now have two .osc files, one with 583380 lines and one with 583379 lines: $ wc -l x.osc y.osc 583380 x.osc 583379 y.osc their only difference is one line at the beginning of the longer file: $ diff x.osc y.osc 2d1 < But the longer one fails to process in osmosis, and the shorter one works: $ osmosis --read-xml-change x.osc --write-null-change Jan 11, 2017 10:19:41 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.43.1 ... SEVERE: Thread for task 1-read-xml-change failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file x.osc. publicId=(null), systemId=(null), lineNumber=583379, columnNumber=90. at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:114) $ osmosis --read-xml-change y.osc --write-null-change Jan 11, 2017 10:20:34 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.43.1 ... Jan 11, 2017 10:20:35 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Total execution time: 1448 milliseconds. Since the line which supposedly contains the "error" is identical in both files, it can't really be an error (and the line does not contain any non-ASCII characters). Re-formatting the XML file with "xmlstarlet fo" or "xmlstarlet c14n" makes the problem go away. I've reproduced this bug on different machines with different Osmosis versions. I've tried these java versions with identical results: $ java -showversion java version "1.7.0_121" OpenJDK Runtime Environment (IcedTea 2.6.8) (7u121-2.6.8-1ubuntu0.14.04.1) $ java -showversion openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) I have uploaded the two .osc files here: http://www.remote.org/frederik/tmp/osmosis-bug-try-read-xml-change-write-null-change-on-these-files-which-differ-only-by-one-line.zip I'd be interested in any insights anyone has to share. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading file over http
Hi, On 06/19/2016 09:09 PM, Stephen Knox wrote: > I should have said, I am using Osmosis as a Java library and don't have > access to the command line as I am using Platform as a service, so I guess > the same applies so long as I can find a decent Bzip converter to stream to > Osmosis? Osmosis includes bzip2 support so it could decode the .bz2 itself but since Java bzip2 is much slower than the standalone commandline program, the version with bzip2 in the pipe is usually recommended. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Supplying postgres port to osmosis commands
Hi, On 10/10/2014 10:49 PM, Walter Nordmann wrote: just call osmosis with host=localhost:15432 Oh dear, that was probably the reason the port thing was never implemented - because it is so simple to embed it in the host. Should have thought about that sooner ;) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error on a node modified and deleted in the same changeset
Hi, (re-heating this 4 month old thread) On 07/13/2013 01:03 PM, Jocelyn Jaubert wrote: I'm currently using a workaround, which is to set maxInterval in configuration.txt to 0.5 day to make sure that osmosis is not trying to merge two diffs. Then I had to add a loop to run osmosis until there are no recent diffs. I've just been bitten by the same problem when trying to keep an Europe-only Nominatim instance up to date with Geofabrik diffs. I wonder if simply dropping the --simplify-change from Nominatim's osmosis invocation will solve things - might lose a little performance for things modified on subsequent days but hey. (One could even do this: try with --simplify-change by default but if this fails, re-run without... Nominatim already re-runs osmosis if it fails so I'd just have to add the drop simplify-change code.) Maybe the best way to fix this is to add a note on a website explaining that the diffs generated by Geofabrik are not meant for direct consumption by osmosis with a big value on maxInterval ? I'll do that. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error on a node modified and deleted in the same changeset
Hi, On 07.05.2013 23:50, Jocelyn Jaubert wrote: I'm including a patch for this, that only impacts the ChangeDeriver class. I have tested it on a country, and the generated diff was correct - the only difference with the current osmosis version is the version on deleted elements. I had not followed this thread, sorry. I didn't know the Geofabrik diffs had this problem, but of course if one thinks about it, it does seem logical - comparing two excerpts, when the newer one doesn't have an object then how am I to know who deleted it when and in what version? I could run Jocelyn's patch on the Geofabrik server but technically it would create wrong osc files - if you were to compare a Geofabrik osc with one loaded from osm.org you'd find that the Geofabrik diff reports the wrong user name, wrong timestamp, and *possibly* also wrong version on a deleted object. No idea if that problem is worse than the one we have now! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Un-Redacting Stuff
Hi, with the license change we introduced the concept of redacted objects. Since redacting an old version touches that version in the database, initially such redactions made Osmosis issue diffs that contained that old version; we then introduced a quick fix to stop that: https://github.com/openstreetmap/osmosis/blob/master/apidb/src/main/java/org/openstreetmap/osmosis/apidb/v0_6/impl/EntityDao.java#L450 We're now also using redaction to suppress objects where a copyright violation has occurred - but mistakes are possible, so we need to have a way to un-redact things if necessary, i.e. remove the redaction_id from a historic version again. Simply setting the column to NULL will, again, make Osmosis issue a diff that contains the old version; this is unwanted. How could we proceed? Ideas: 1. Introduce special value 0 (not NULL) to denote an un-redacted object; leave Osmosis unchanged (so it treats NULL and 0 differently, will only issue .osc for objects with redaction_id=NULL), and modify other API code to treat 0 and NULL the same (so historic versions can be accessed through the API if redaction_id=NULL or 0). Cheap, easy, but a bit ugly. 2. Introduce an additional column suppress_diff to nodes/ways/relations tables; on un-redaction, set redaction_id=NULL and suppress_diff=TRUE; modify Osmosis by assing an and not suppress_diff to the SQL query. Would increase database size by something like 4 GB for the extra column. 3. Introduce an additional table un-redacted objects, store object type, version, and id; when an object is un-redacted, add it to that table and clear the object's redaction_id, then modify the Osmosis query to only output objects that are not found in that table. Uses little space but makes diff creation slower. There might be more... Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Remove entities in a bounding box leave other intact
Hi, On 01/16/2012 07:37 PM, Jaume Figueras i Jové wrote: I've been looking different Osmosis options but I have no idea on how to do this processing. You can use the --bp task and define a polygon with an exclamation mark before the polygon ID to cut something out, see http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format Example: test 1 0.000E+00 0.E+00 1.000E+01 0.E+00 1.000E+01 1.E+01 0.000E+01 1.E+01 0.000E+00 0.E+00 END !2 0.200E+01 0.2000E+01 0.800E+01 0.2000E+01 0.800E+01 0.8000E+01 0.200E+01 0.8000E+01 0.200E+01 0.2000E+01 END END Bye Frederk -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] full OSM mirror with Osmosis
Hi, is there anyone on this list who runs a full OSM replication server with Osmsois (i.e. import a full history planet file, keep both historic and current tables, and regularly apply diffs, all using the apidb schema)? I would be interested in timings for initial import and diff application and of course in the specs for the hardware. Also if someone has tried and failed ;) Reason I'm asking is that I am thinking to set up a kind of shadow DB that has an ODbL compliant version of everything, so everyone can easily see how their area would look like if the license change was done today. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] --used-node performance and a possible way to improve it
Hi, Jochen Topf wrote: I have thoght about these things quite a lot, too. I agree with Igor that we are bumping into the limits of what the current Osmosis pipelining model can do. We seem to be reaching a point where people are creating specialist programs to do things that are traditionally done with Osmosis - when until recently it was the other way round, you needed less and less specialist programs because Osmosis did it all. For example, there's osmconvert (http://wiki.openstreetmap.org/wiki/Osmconvert), a tool that can read/write OSM, apply diffs, do bounding box, polygon, and tag filtering, and people published benchmarks in the forum where that program was almost 5 times as fast as Osmosis. The program is certainly not a marvel of software design (6000 lines of rather unreadable C code in one source file, and it uses NO LIBRARIES except lz, not even an XML parser or a geo library or Protobuf code - all hand-coded) - but still, it seems to have a growing user base because it is fast and can be easily installed/compiled everywhere. (Also, the tag filtering seems to be a bit easier than the tee/tf/used-way/used-node/merge orgy that you currently need to perform in Osmosis if you want to extract something like all forests.) It used to be that, compared to other means of achieving the same, Osmosis was perhaps minimally more clumsy and perhaps minimally slower, but I always used, and still use, Osmosis for everything because it was so convenient to have this one multi-tool. (Maybe except when I was using grep.) - But when others start to do the same in a fifth of the time then maybe it is really time to think; Split Osmosis into two parts, one is the low-level code for OSM objects, readers and writers etc. And one is the control plane code that stitches pipelines together etc. ... maybe the approach you are sketching would also allow one to somehow hack together shortcuts that would allow the building blocks to be used without anything not necessary for the task at hand. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Igor, On 02/18/11 11:40, Igor Podolskiy wrote: just a random thought: what's wrong with using --dataset-bounding-box? Importing the planet file into a database and doing a bunch of queries against is equivalent to creating a single disk buffer for all bb tasks (the database _is_ the disk buffer, if you want). This still isn't very elegant as it requires two passes (import into DB and export to PBFs) but is IMHO more elegant than selection lists. I currently use bounding polygons so I'd have to add a --dataset-bounding-polygon task for that but that should be possible. The major problem is finding a database that would be able to import the data, index it, and fulfil my requests, all within less than 24 hours. From what I hear, PostGIS takes a week for the import step alone but I want to produce fresh extracts every day. So I would have to import into PostGIS and then apply the daily diffs, but I could only do that if applying a daily diff and making all the bounding-polygon requests would take much less than a day and somehow I suspect that's not going to work. Don't get me wrong, maybe there's a reason why dataset tasks are unsuitable - but I've not found any... Maybe the dbb task itself needs improvement - but in this case, this should be the way to go. I doubt that a performant solution can ever be found with PostGIS but I'd love to hear from someone who can prove me wrong. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Scott, On 02/18/11 14:15, Scott Crosby wrote: I used two bitsets for every output file. One indicating which nodes were already output and another (built when processing ways) indicating what node ID's were missed and will need to be grabbed on the next pass. I have another two pair of bitsets for ways and relations. Thats around 400mb of RAM per output. I see. Means I overestimated a little ;) Really, almost everything you want is already done from the mkgmap splitter crosby_integration branch. Just make it support non-rectangular regions, and track output/missed entities with bitsets. I'd say about 4 hours and 2gb+400mb/region to generate as many outputs as you want. I'll have a look at that. * all nodes in the bounding box * all ways using any of these nodes * all nodes used by any of these ways even if outside * all relations using any of these nodes or ways o all nodes and ways used by any of these relations even if outside o but NOT all nodes used by a way drawn in through a relation. (The points marked * are what the API does; the API does not do the o marked points even though users could be interested in them.) I did not know that it was allowed to miss the things in o. That makes the job easier. The API doesn't do them on purpose even though, having access to an indexed database, it could easily add the o points. Reason being that other relation members may not even be geographically nearby, and if you load a small part of European countryside you might not be interested in receiving the full detail about the Madrid-Peking long distance cycle route just because that happens to pass through your area of interest ;) There's a second approach that could split the entire planet into extracts, simultaneously, in 2-3 minutes; a new planet format that I'm working on that is geographically sorted. Progress is ongoing, but slow. Sounds interesting - and difficult, in case of the Madrid-Peking long distance cycle route ;) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Scott Crosby wrote: I've run a test over all of the files, computing a CRC of each block, and prettyprinting every block that differs. The two files are identical except for the reported osmosis version: That's comforting (to me at least). Does the same breakage happen of pbf2osm is run on the same file in windows? I'll have to defer that to Henning - I am not sure if anyone has even built pbf2osm on Windows? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Scott Crosby wrote: If they do that conversion on windows, does that generate a pbf that has the same md5sum as the geofabrik generated file? (And if not, then I'd like both of those files!) I'll try to assemble a proper test case with the help of one of the windows users. * converting the Geofabrik-generated .osm.pbf to XML will copy exactly 10,600,000 nodes to the XML file and then stop, without error, adding a proper /osm but no ways or relations. Is this with pbf2osm or osmosis or both? This is when trying an Osmosis-on-Linux-generated .osm.pbf to .osm with Osmosis-on-Windows. pbf2osm is not part of this particular problem. Is the 10,600,000 nodes from different files or the same file? The resulting XML file has 10,600,000 nodes. Does anybody have an idea why this might be? The europe.osm.pbf can be processed without problems on Linux, with Osmsis as well as with the mkgmap splitter. Did the users having this problem successfully work with geofabrik extracts from last week or last month on their windows machines? Yes, they claim they did. The only change that I made is upgrading from 0.37 to latest SVN, and adding the cascadingRelations=true option which should really only influence relations. Other than that, bizarre. I had some random crashes with the mkgmap splitter, until I found this bug: http://bugs.sun.com/view_bug.do?bug_id=698 and started running with -XX:-ReduceInitialCardMarks I'm wondering if this is some issue with corruption with the JDK that's rare and causes spurrious correlations. That would explain why smaller files are less likely to hit the issue, whatever it is. I think one of the error reporters indeed seems to have some sort of hardware (RAM) fault but it is unlikely that all of them do. I don't have development environment on windows to do any experiments of my own. Same here although I might be able to conjure something up on an old laptop. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] pbf problems under windows
Hi, Peter Körner wrote: I planned to do sth. for this tomorrow: Download the current europe.osm.bz2, convert it to pbf. Download the current europe.osm.pbf Great. - User aighes on talk-de seems to be trying out the same with today's files. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] proper usage of osmosis when writing changes to osm api db
Samir, Samir Faci (Dev) wrote: I'm not sure if this is helpful or not It was well intended but the original author inquired about an API DB database, which is populated directly by osmosis, without osm2pgsql. It uses a different database schema and does not have a slim mode. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improve relation extraction handling
Brett, Brett Henderson wrote: I'm thinking of adding a cascadingRelations=yes flag to the --bp/--bb tasks that would * process relations not containing other relations as usual, * throw all other relations onto a stack and process them in a suitable order afterwards I'd try to use the existing completeXXX options as a starting point. I have done that now and committed the enhancements in r24342. I ran some tests of my own and I think everything works as desired. I'd love to amend the tests as well but it seems I am lacking either the proper system or the proper brains to run the existing tests at all. The BoundingBoxFilterTest and PolygonFilterTest work in 0.38 and they now fail in trunk; the AreaFilterTest fails in 0.38 as well as in trunk: test: [mkdir] Created dir: /home/fred/svn.openstreetmap.org/applications/utils/osmosis/tags/0.38/areafilter/report/test [junit] Running org.openstreetmap.osmosis.areafilter.v0_6.AreaFilterTest [junit] Tests run: 5, Failures: 5, Errors: 0, Time elapsed: 0.348 sec How do I find out why exactly these tests fail? Running ant with -v only gives lines like [junit] junit.framework.TestListener: startTest(testProcessBoundContainer1) [junit] junit.framework.TestListener: addError(testProcessBoundContainer1, org.openstreetmap.osmosis.areafilter.v0_6.PolygonFilter.init(Lorg/openstreetmap/osmosis/core/filter/common/IdTrackerType;Ljava/io/File;ZZZ)V) [junit] junit.framework.TestListener: addError(testProcessBoundContainer1, null) [junit] junit.framework.TestListener: endTest(testProcessBoundContainer1) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Improve relation extraction handling
Hi, I'm a major user of the --bp task in Osmosis, for the nightly Geofabrik country extracts. I cannot use the complete-something flags as that would take far too much time, and on the whole the clip-incomplete-entities mode seems not to be too bad for users. There's one thing that has popped up on this list however (lost relations while clipping) which is an increasing concern to our users and which I hope I can fix. The problem is that with cascading relations, Osmosis will only include a relation if one of its children has been included already, so if node N1 is inside the clipping area, relation R1 contains node N1, relation R2 contains relation R1 then relation R1 will alway be included in the output, but relation R2 will only be included if it comes after R1 in the input file. Now while using the temporary storage for nodes and ways would be too expensive for the nightly Geofabrik build, I think that temporarily storing relations would be acceptable (as there aren't too many). I'm thinking of adding a cascadingRelations=yes flag to the --bp/--bb tasks that would * process relations not containing other relations as usual, * throw all other relations onto a stack and process them in a suitable order afterwards I am aware that determining a suitable order could be difficult if you have many levels of cascasing (or even circular) relations, and that the output file would not be ordered properly, but I think such an option would mean a big improvement at a relatively low cost. 1. Any comments? 2. Anyone done something like that already? 3. Is any of Osmosis' internal storage mechanisms suitable for buffering relations in this way or do I have to invent something new? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] No Default pipes available?
Hi, Markus wrote: bin\osmosis.bat -–wxc update.osm.gz You'd have to do a bin\osmosis.bat --rri --simc --wxc update.osm.gz or if you want to apply it to a planet file directly, bin\osmosis.bar --rri --simc --rx planet.old.osm.gz --ac --wx planet.new.osm.gz Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Project Restructure
Brett, Brett Henderson wrote: I haven't created a 0.36 detailed usage page on the wiki yet but will do so before releasing. There are a few other random goodies in the latest development release that I'd like to release in the near future. Will there be significant changes in ordinary Osmosis usage, or is the change largely transparent for users? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error while applying updates to planet.osm
Tyler, sounds like either your planet or your change file is corrupt. Check the files with xmllint --stream. Re-download the one that is broken. Next, I ran the commands below to get my updates and simplify them: osmosis --rri workingDirectory=. --wxc update.osm.gz gzip -d -c update.osm.gz | /usr/local/osmosis/bin/osmosis --read-xml-change file=/dev/stdin --simplify-change --write-xml-change file=- | gzip -9 -c update.unique.osm.gz Finally, I attempted to apply these changes to my planet.osm file: bzcat planet-100618.osm.bz2 | /usr/local/osmosis/bin/osmosis --read-xml-change file=update.unique.osm.gz --read-xml file=/dev/stdin --apply-change --write-xml file=- | bzip2 -c planet.osm.bz2 All that can be shortened to this one line: bzcat planet-100618.osm.bz2 | osmosis --rri --simc --rx - --ac --wx - | bzip2 planet.osm.bz2 But it is probably good to do it in single steps while still testing the process. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] bounding-box does not support data provided by default pipe
Hi, Oliver Schrenk wrote: Has 0.3 been deprecated/removed? Can I use an older version for extracting bb from 0.3 maps? What's the case for 0.4 maps? 0.3 and 0.4 are basically the same and can be processed with the same software. Osmosis never supported either, it starts at 0.5. There is a tool in SVN which converts from 0.4 (and therefore also 0.3) to 0.5 (applications/utils/conv05). However that will probably not run on the whole planet file, or take too long; you will have to use applications/utils/osm-extract/planetosm-excerpt-area.pl to cut out an area from the old file. That perl script either supports 0.4 in its current form, or maybe you'll have to check out an older version from SVN. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postGIS: bbox columns?
Hi, David Carmean wrote: Who/what uses the ways.bbox column? Is there something/somebody that cannot use the linestring gist index, but instead requires an explicit bounding box geometry? Is the bbox column not meant to be a quicker alternative to a full linestring (quicker in importing, not quicker in using)? I.e. you won't have a linestring to base your GIST index on if you use that as intended. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Unable to write String to the store
Hi, Nakor wrote: I was trying to extract a bounding box from planet-100519.osm.bz2 and after one hour or so osmosis stopped running with the error message at the end. You seem to have used one of the --complete-something options. These are real performance killers because they have to cache the full planet in a temp file. Don't use them unless you really have to. If the extract you are making is relatively small, then it can make sense to first do an extract with --clip-incomplete-entities for something that is slightly larger than the bit you want, and then make an excerpt *from that* with the --complete option. I have 175 Gb free on the disk where the planet file resides. Is this not enough? Or is osmosis using another disk for its temporary files? 175 GB is enough because the temp files are gzipped. But it uses /tmp unless you export JAVACMD_OPTIONS=-Djava.io.tmpdir=/my/pretty/little/tempdir before you run it. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Can't load osm file to PostGRESql
Hi, Ibrahim Bouchrika wrote: Your idea seams nifty, although it's giving an error on the --rx command. I don't know Osmosis well enough to figure out where the error comes from. Is it just an error in the syntax or is it because of the osm-file? Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog. Can you make sure your empty.osm and your amsterdam.osm begin with the characters osm and nothing else? Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems importing .osm files into postgis
Brett, I think i set up the database correcly is, I used the pgsql_simple_schema_0.6.sql schema from osmosis. I also used the 900913.sql file. What is the 900913.sql file for? That's an INSERT command that adds the EPSG:900913 projection to PostGIS's spatial_ref_sys table. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Cryptic error message
Hi, org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task 2-bounding-box does not support data provided by default pipe stored at level 1 in the default pipe stack. You must either upgrade your data to 0.6 by adding a --migrate after the --read-xml-0.5, or you must use the 0.5 version of all tasks by adding -0.5 to their name (--bounding-box-0.5 etc.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis Release 0.35
Brett, Brett Henderson wrote: There are no major new features, just an assortment of bugs That's how we love our Osmosis ;-) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet import time
Hi, Jens Lehmann wrote: A solution would be to include it in the planet or provide it as a separate file, which can be loaded into the database. Well it isn't even in our central database so it would have to be computed at the time we dump the planet file which already is an expensive enough operation ;-) Can the long running bbox query be aborted/killed without resulting in a corrupt database or do I have to start the import from scratch for testing the Java based solution? I'm not sure if Osmosis uses one transaction for the whole import or several. Sure you can kill and you will not have a corrupt database but not an usable one either I believe! Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: I appreciate your quick response! I am not surprised by your answer. Any idea on how much disk space would be required by a PostGIS database that would hold the entire planet.osm file data? I think it would be between 100 and 150 GB for either schema; but only 30 GB if you do option 2 (osm2pgsql) in non-slim mode. In non-slim mode, osm2pgsql needs more RAM (at least 16 GB during import) but produces a smaller, and non-updatable, database. I'm not sure what the simple schema is. I assume that is different than resolution #2 that you indicated in your reply? Yes, osmosis and osm2pgsql use different schemas. Osmosis supports the APIDB schema and the Simple schema while osm2pgsql has its own schema, available in slim and non-slim varieties. From a data modeling perspective - and I'm on somewhat thin ice here, happy for anyone to chip in and confirm or protest - the APIDB schema is the most powerful because it has all the data plus history. The Simple schema, as well as the Osm2pgsql slim-mode schema, still have a full OSM data snapshot (and you could theoretically convert from one to the other, or from APIDB into one of them) but they don't have history. The non-slim osm2pgsql schema, lastly, carries only a processed selection of OSM data (exactly what is needed for the map - nothing more). You could theoretically convert from any of the others to non-slim osm2pgsql, but never in the other direction. I assume that in answer #2, there is a standard schema that Mapnik assumes must be in place in the database it will query and that schema is different than the simple schema specified in answer #1? Yes. To me more precise, Mapnik itself can be made to work with any schema, even the Osmosis simple schema, but the schema created by osm2pgsql and the style sheet commonly used in OSM work together and if you were to use Mapnik with the Osmosis simple schema you'd have to write your own style sheet. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: You mentioned that osm2pgsql needs more RAM (16GB+ during import). Do you know how much RAM would be required for normal lookup and rendering usage with either slim or non-slim usage? For normal lookup it doesn't make a difference. It should run on a 4 GB machine without problems. Also, I think a schema that didn't have history would be better for my purposes (less data) as I only need lookup and rendering capabilities (i.e. read-only). I think then osm2pgsql is going to be your friend. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis use case
Hi, Walschlager, Gerard wrote: The bottom line for my investigation of mapping tools is that I need to have a local database of the world to perform lookups and rendering of maps on a standard Windows based desktop computer in a reasonably quick fashion (i.e. pan/zoom operations complete in 1 to 5 seconds). Performance wise, for my scenario, I'm not sure which would be better: rendering as tiled images or rendering as SVG. My application will be using WPF which can render XAML based graphic paths, which can be derived from SVG. There is a utility called Osmarender that I've heard of and I understand it can render to SVG. There is a rather large gap between your ambitions and your current knowledge ;-) Mapnik can be made to render any section of the planet to an image in 5 seconds but you will need to tweak the style sheet somehow. Rendering to XML is possible but the XML that Mapnik produces is highly complex and it is not unlikely that you'll stretch the capabilites of your library with it (go to www.openstreetmap.org and use the export tab to generate SVG, and try that out). Osmarender is too slow for your purpose. There's no reason why you must use tiled images. You can also make Mapnik render exactly the section you need. I think Osmarender would work with a MySQL database. No, Osmarender does not use databases at all. Would you guess if rendering as SVG and then converting to XAML be faster than working with images rendered on-the-fly? My guess is: slower by a factor of 5. If I go with rendering as tiled images, then I have to consider rendering on-the-fly or pre-rendering in advance. It is probably a bit painful to get Mapnik + Postgres running properly on a Windows machine, *especially* if that machine is somewhere at your user's site and not fully controlled by you. Tiles are, in contrast, dead-easy. They only have one drawback: There are a lot of them. If I pre-rendered in advance, then I suppose I would run Mapnik over the entire osm2pgsql generated database and that would in turn generate yet another database of tiled images? No, it generates individual files, not a database. If you want to cover the whole planet at zoom level 18 you will need to produce about 100 billion tiles and they will together use about 100 Terabytes of storage. This means you will have to think more about what you really need. Any thoughts on size of a database of 256x256 PNG images for entire planet.osm contents? Or, is rendering on-the-fly going to allow me to perform my rendering of user requests in a few seconds? Probably, if nobody else uses the machine. Can you not simply download tiles from a web service? That would solve a hell of a lot of problems for you. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Database is empty after five days of importing
Hi, Klemens wrote: So at some time in the future when I want to import Europe into a DB that contains Austria I will have to create a delta .osm first using osmosis? Possible, but impractical, because you would spend more time to create the delta than you save by re-using the existing data. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Database is empty after five days of importing
Hi, Klemens wrote: That makes sense. My requirement is that I must continue querying the old data while importing the new, larger data - which can take a long time as we know. How can I achieve this? I would suggest to create a second database instance (same server process, different database name, same table names), import into that, then drop the old database and rename the newly imported. This will give you a short service interruption. If you cannot tolerate any service interruption, you can use the pgbouncer proxy which can be reconfigured to connect to another database while still serving active connections to the old database. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] help by merge 4 files
Jan, Jan Tappenbeck wrote: can anybody tell me the reason why not do - here is the report: True, finding out the *acutal* problem among 100 lines of Osmosis error reporting is sometimes difficult. SCHWERWIEGEND: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are not sorted This seems to be the problem. If you insert a --sort (with some Osmosis versions you may need --sort-0.6 after *each* --rx spec then it should work. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [RFE] Add bounding shape to OSM dumps
Hi, Apollinaris Schoell wrote: osmosis supports 2 options to keep ways and relations intact. but geofabrik extracts don't use it as far as I know. completeWays completeRelations That is correct: These options exist, and are not used, because if we'd use them, the nightly build would take something like three days instead of six hours. Osmosis works very well in streaming mode and these options make it impossible to stream - Osmosis needs to create a temporary copy of the full dataset and is rather inefficient at it. We typically do something like osmosis --rx file.osm --tee 20 --bp file=country1.poly --wx country1.osm --bp file=country2.poly --wx country2.osm... which would causes Osmosis to make 20 temporary full copies of the data and write them to disk in completeWays mode. We use clipIncompleteEntities which means that Osmosis removes references to those nodes outside the polygon from any ways or relations. It would be good if Osmosis could somehow flag the clipped entities so that processing software could at least know that there is something wrong, or incomplete, with them. Adding the actual polygon used for clipping could of course be done but it will not automatically enable proper filling. Assume this: | +-|+-+ | | | | | |--- filled area +-|+-+ | |-- clipping boundary After clipping with clipIncompleteEntities, this will lead to | |+-+ || | || | |+-+ | | Even if you know where the clipping boundary is, you cannot extend the object towards that boundary properly because you are missing the nodes beyond the boundary. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
Hi, Simon Nuttall wrote: 2010/1/10 Simon Nuttall simon.nutt...@gmail.com: I think both these points are more a message for geofabrik.de than me because I'm using the diff's downloaded from their site. Sorry, I wasn't clear, I meant to say that CycleStreets uses the ireland and great_britain extracts that we download from geofabrik.de, and their system seems to be using 0.31.1 now. The problem was that in switching from --rci to --rri I didn't add the --simc filter, which led to duplicate objects in the extracts. It should be fixed now. I don't think that it was related to the Osmosis version, but switched to 0.32 nonetheless. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways - polygons?
Hi, Lennard wrote: David Carmean wrote: I found that osm2pgsql omitted several attributes that I require. Also, the schema created by osmosis is more suitable to my needs. If these attributes are the user, uid, version and timestamp, check out http://trac.openstreetmap.org/changeset/19144 And if they are other attributes, then it is very likely that you will be able to modify the osm2pgsql style file to suit your needs. This is probably going to be easier than trying to build proper multipolygons from data imported by osmosis, unless you are a real SQL wizard. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Closed ways - polygons?
Hi, David Carmean wrote: If nothing else it appears that I could write a query using PostGIS functions that will do what I want: http://postgis.refractions.net/documentation/manual-1.3/ch06.html#IsClosed Be advised that multipolygons may consist of any number of outer and inner ways which do not have to be closed themselves, they only have to form a closed ring. It will certainly be hard (but certainly not impossible) to do this with SQL alone. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Replication Changes
Hi, Brett Henderson wrote: There is a new (as yet undocumented) task in osmosis called --simplify-change which allows a full history diff to be collapsed into an older style delta diff. In a scenario where you use --rri to retrieve a set of combined minute diffs, and if you continue with --wxc to feed stuff into osm2pgsql, would an intermediate --simplify-change then make sense, or does --rri do that automatically for you? I'ts amazing how far Osmosis has come in the 2.5 years since it was first announced (http://lists.openstreetmap.org/pipermail/dev/2007-June/005390.html - There are a few features on my todo list ... Upgrade to support 0.4.) and how it is now a central element to much of what's going on in OSM. You were a total unknown at the time (or at least not present on the lists!) and initially I thought Osmosis was a slightly quirky tool that would probably not live long because its inventor would be likely to lose interest. Yet you produced something that profoundly influences the way most of us work with OSM. Hats off to that! Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] duplication of nodes on apply-change
Hi, Maxim Dubinin wrote: MD But after I do: MD osmosis --read-xml-change file=data/updates/20091112-20091113.osc.gz MD --read-xml file=data/rus.osm --apply-change --write-xml file=data/rus-new.osm MD I'm getting two versions of the same node in the output: Is it possible that I am resposible for this because I recently changed the sorting function to treat something with version=x+1 as larger than something with version=x, instead of equal? (SVN r18230, change to EntityByTypeThenIdComparator...) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Relations with 65k members
Hi, we just had a problem with diff updates because someone created relations with more than 65k members and Osmosis tries to store them as a char which raised an exception. I did a quick hack that would simply drop the excessive members with a warning: http://www.remote.org/frederik/tmp/remove_excessive_members.diff I'm not checking that in. I thought about simply patching Osmosis to use an integer but was informed by Jon Burgess that osm2pgsql (which is next in my toolchain) would carp on 65k members as well so I didn't bother - especially as it turned out that the relations in question were an error in somebody's automated import and have been meanwhile fixed by Lennard (they contained the same element over and over). Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Relations with 65k members
Hi, Frederik Ramm wrote: I thought about simply patching Osmosis to use an integer but was informed by Jon Burgess that osm2pgsql (which is next in my toolchain) would carp on 65k members as well Turns out osm2pgsql in its default config allows only up to 32767 members. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Sorting (and test whining)
Hi, Brett Henderson wrote: This was necessary in order to process replicate diffs which sometimes contain the same object twice, causing the old implementation to complain about change streams not being sorted (as being sorted, for Osmosis, means that every object has to be greater than the previous). Which task was complaining? I had intended to create a new task which would take a (sorted) change stream with full changes and collapse them into a single change per entity. This new task would allow existing tasks such as --apply-change to continue to be used as is. In my case it was --merge-changes which complained. I had downloaded some changes with --rri, and later did the same thing again and wanted to merge the two before then processing the unified result with osm2pgsql. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [OSM-dev] [OSM-talk] osmosis Problem with bounding-polygon and v0.6?
Hi, Karl Newman wrote: If the fault does lie with the bzip2 code, then I'm surprised you're the only one that's reported a problem so far. Nobody in their right mind lets Osmosis compress or decompress bz2 ;-) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
