Re: [OSM-dev] Faster loading with scabies

2008-10-20 Thread Noel Grandin
Most modern databases have some kind of "block-load" API for efficiently importing large chunks of data into their tables. It's really only the most simple stuff that __doesn't__ slow down when importing using INSERT. Although it should plateau to a constant time/INSERT pretty quickly. Brett Hend

Re: [OSM-dev] Faster loading with scabies

2008-10-17 Thread Brett Henderson
Sascha Silbe wrote: > On Thu, Oct 16, 2008 at 09:21:54AM +1100, Brett Henderson wrote: > >> The biggest problem I found wasn't the actual processing of INSERT >> statements, it was MySQL scaling non-linearly with the number of rows. >> MyISAM tables are very fast to import regardless of number of r

Re: [OSM-dev] Faster loading with scabies

2008-10-17 Thread Brett Henderson
Joachim Zobel wrote: > Am Donnerstag, den 16.10.2008, 09:21 +1100 schrieb Brett Henderson: > >> Do you have any numbers to show what sort of performance increase can >> be expected. >> > > I can load europe.osm into the current_* tables in less than 12 hours > (on my laptop - overpowered

Re: [OSM-dev] Faster loading with scabies

2008-10-16 Thread Sascha Silbe
On Thu, Oct 16, 2008 at 09:21:54AM +1100, Brett Henderson wrote: The biggest problem I found wasn't the actual processing of INSERT statements, it was MySQL scaling non-linearly with the number of rows. MyISAM tables are very fast to import regardless of number of rows, but InnoDB seems to slow

Re: [OSM-dev] Faster loading with scabies

2008-10-16 Thread Joachim Zobel
Am Donnerstag, den 16.10.2008, 09:21 +1100 schrieb Brett Henderson: > Do you have any numbers to show what sort of performance increase can > be expected. I can load europe.osm into the current_* tables in less than 12 hours (on my laptop - overpowered CPU, 5400 disk). > If it's significant I

Re: [OSM-dev] Faster loading with scabies

2008-10-15 Thread Brett Henderson
Hi Joachim, Nice work, any info on how to improve load speed is much appreciated. On Thu, Oct 16, 2008 at 6:15 AM, Joachim Zobel <[EMAIL PROTECTED]>wrote: > Hi. > > I have written a utility named scabies to load data from osm files into > a mysql database. See > > http://www.heute-morgen.de/scab

Re: [OSM-dev] Faster loading with scabies

2008-10-15 Thread Stefan de Konink
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Joachim Zobel schreef: > Hi. > > I have written a utility named scabies to load data from osm files into > a mysql database. See > > http://www.heute-morgen.de/scabies/ > > Usage is rather straightforward, but you need lots of disk space. > scabie

[OSM-dev] Faster loading with scabies

2008-10-15 Thread Joachim Zobel
Hi. I have written a utility named scabies to load data from osm files into a mysql database. See http://www.heute-morgen.de/scabies/ Usage is rather straightforward, but you need lots of disk space. scabies_parse.pl parses the osm file and creates files that are then loaded by scabies_load.sql