On 09/07/15 08:43, Jonathan Barber wrote: > I think you're process is being moved between NUMA nodes and you're > losing locality to the data. Try confining the process and data to the > same NUMA node with the numactl command.
A friend of mine who hacks on MySQL (at a variety of companies, now at IBM) wrote yesterday about a long running "swap insanity" issue that sounds similar - the swap they are talking about is not paging to disk but memory getting moved between NUMA nodes. https://www.flamingspork.com/blog/2015/07/08/the-sad-state-of-mysql-and-numa/ It would be interesting for David to try his program with one of the suggestions in the blog to see if that makes any difference: numactl –interleave all The other thing that may shed some light is to run "perf top" whilst this is going on (as root) to see if there are any obvious culprits. Best of luck! Chris -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
