On 6 Jun 2012, at 12:00pm, IQuant <sql...@iquant.co.cc> wrote: > We need to be able to run 1000's of extractors concurrently processing > different tick tapes and symbol sets. aka service bureau. The Daily > tick tapes are approx 20gb each.. 30TB repository and growing. An > extraction run take 1 - 5 minutes for small symbol sets per tape. > > An example would be to concurrently extract 5 years of a particular > stock's tick data. 1500 days x 2 minute extraction jobs across 15 VMs > each running 100 extractors. > > I'll try journal mode off and increasing page size.
Your problem appears to relate to your virtualisation layer. You yourself reported "previously ran fine and fast on bare metal." I see no indication that SQLite is doing anything weird here. The only problem you're reporting is 100% CPU utilisation which is not, of itself, a problem. It just means that CPUs are being utilised very thoroughly which is good in a process that munches through lots of data. I suspect there's some optimisation component to your settings to VMWare which is not well tuned to data-munching processes like this. Some form of caching, for instance, would be terrible for a process like this which uses data and then throws it away. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
