Hi, After conversations with TomH and others, I have finally gotten around to rewriting the GPX importer in C.
Attached is r14 of my code, and I believe it to be *ALMOST* feature-complete. In essence, it functions equivalently to the ruby importer, however it consumes significantly less system resources. TomH suggested to me on IRC that the ruby importer consumed the equivalent of 30% CPU continuously on the box on which it runs. My tests, although tainted by the fact that I ran them against an effectively blank database, show that the C reimplementation takes approximately 0.4% of that, so approximately 0.12% of a CPU on average through the month. The particular gain however is that an import which took almost 2min30 with the ruby, took less than a second with my C code. Naturally, the C version is not configured by the ruby environment which does introduce issues wrt. keeping it consistent with the rest of the system, however it is configured by environment variables set in settings.sh prior to it being run. As such, it would be possible to write a ruby daemon wrapper which configured the environment and then ran my code, were such desirable. (I'm *really* not a ruby hacker, so someone else would have to do that). My code does not replicate the "delete all traces which are set to visible=0" functionality as I am unsure of when this behaviour would be required. The images are generated using libgd rather than imagemagick. As such, they are *marginally* different, but functionally equivalent. To build this, unpack the tarball, cd into the src/ subdirectory and type 'make' You will need the MySQL client library headers, and libgd's headers, along with the Expat library headers for the GPX parsing. To run it, from the directory containing src, templates etc, run: ./settings.sh src/gpx-import This assumes a localhost database, /tmp/osm/traces/* and /tmp/osm/images/* Trivial inspection of settings.sh show how to reconfigure that. I'd really appreciate any comments anyone might have, and if at all possible, a functionality review from one of the OSM admins. Modulo some kind of daemon management (gpx-import runs against its console by default), and the question about invisible traces, I believe it to be ready for deployment. Regards, Daniel Silverstone. (Kinnison on the wiki) -- Daniel Silverstone http://www.digital-scurf.org/ PGP mail accepted and encouraged. Key Id: 2BC8 4016 2068 7895
gpx-import-r14.tar.gz
Description: application/compressed-tar
_______________________________________________ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
