Experimenting with Data Warehouse - which should really be run on a more "mainstream" DB. Sqlite was supposed to be just for piloting and testing - but it's such an incredible little database engine - it's hard to let it go - so I try big things on it just for kicks - delaying the inevitable.
It very easily handles billion row/100GB tables - multi-table joins, etc. - it just chugs for a while because it's single threaded, and will gobble up memory and swap - but it gets the job done. It's quite amazing for a DB engine that's not even 1 MB. While it's happiest when it can do all the work in memory versus disk - it reads fairly quickly from disk the first time around - but not subsequently - which is why I posed the question. For now - my workaround is to attach and detach for every SQL statement - but a better solution is probably to get a server with big RAM and tune some of those OS settings - and ultimately, maybe MySQL? On Fri, Jul 13, 2012 at 9:16 PM, Keith Medcalf <kmedc...@dessus.com> wrote: > > I know the newer versions of Windows are fantastically bloated (and slower > every version), but what are you running that uses more than 16 GB of > committed memory? > > > Thanks. More RAM would clearly be helpful - but first I need a bigger > > machine that can take it. For some reason - the "home" line of PC is > > typically capped at 16GB or so. I'll Need more of a workstation to go > > higher and experiment with the settings you suggested. > > > > On Fri, Jul 13, 2012 at 8:35 PM, Keith Medcalf <kmedc...@dessus.com> > wrote: > > > > > > > > Windows is really atrociously bad at I/O. Windows has the same basic > > > model of how to perform I/O as a 6 year-old. Scratch that, the six > year > > > old could probably understand I/O better than whoever wrote/designed > the > > > crap in Windows that passes for I/O routines. > > > > > > Anyway, make sure that you have all the fanciful magical features > turned > > > OFF (they make things slower, not faster). That is all the various > > > "SpeedBooster" crap and so forth that Microsoft crappifies their OS > with to > > > give that "gee wiz" wonderful warm and fuzzy feeling to the mass > consumer > > > market. > > > > > > Second, make sure you have turned off "Large System Cache". Force > Windows > > > to forgo the magic, forgo the dreadful cache design, and do I/O > properly. > > > > > > Disable "Magical Virtual Machine Resizing"-- set a fixed pagefile size > -- > > > or better yet get sufficient RAM and disable swapping altogether -- it > is > > > pure bull droppings that you need a pagefile that is a percentage of > RAM > > > size. If it works with 4GB of RAM and a 4GB swapfile, then it will > work > > > better with 8 GB of RAM and no pagefile. > > > > > > Then increase the IOPageLockLimit to something reasonable. > > > > > > And if your DASD driver supports it, enable block-level I/O > optimization > > > and/or caching. > > --- > () ascii ribbon campaign against html e-mail > /\ www.asciiribbon.org > > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
