Running a single extractor on a VM runs near bare metal speeds. I can run concurrent extractors up to the # of cores allocated to the VM and sustain decent throughput. Once I go above this count the VM's get crippled.. Pegged at 100% - Disk, RAM and LAN loads are miniscule.
For a test we ran 50 extractors across 2 VM's (8 core and 4 core) and it took 14 hours to extract a single symbol. A single exctractor running on one 4 core VM can do the job in 50 hours. The same run limited to running 12 extractors finished in 4 hours. There is obviously an issue with the virtualization but no clear fix or workaround were aware of. Our objective is to extract and process all tick tapes in one shot. <<< Response to >>> Unless pegging the CPU causes the hypervisor to preempt the guest's synchronization primitives and cause priority inversion, in which case OP's throughput would plummet. If throughput remains reasonably stable in the VM, though, I would agree that a pegged CPU isn't necessarily a problem. Another possibility: the guest OS idle loop will probably count as CPU usage on the host, and with heavy load the guest won't necessarily attempt to park the cores it was using in between scheduling decisions; on bare metal, the spinning idle loop doesn't get reported as CPU usage, even though it is. Ryan _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
