pmap and mtx scalability problem
I treid make -j 30 build{world,kernel} (latest -CURRENT) on 24-core machine and
see poor
scalability of pmap/mtx -- more then 50% cpu spend on system time.
pmcstat:
@ CPU_CLK_UNHALTED_CORE [194841 samples]
42.65% [83102]_mtx_lock_sleep @ /boot/kernel/kernel
40.97% [34051] pmap_enter
100.0% [34051] vm_fault_hold
100.0% [34051] trap_pfault
30.40% [25262] vm_page_activate
100.0% [25262] vm_fault_hold
100.0% [25262] trap_pfault
18.41% [15300] vm_pageq_remove
73.63% [11266] vm_page_free_toq
70.69% [7964]vm_object_terminate
29.31% [3302]vm_object_page_remove
26.37% [4034] vm_fault_hold
100.0% [4034]trap_pfault
make -j 8:
15.44% [10740]_mtx_lock_sleep @ /boot/kernel/kernel
38.77% [4164] pmap_enter
99.93% [4161] vm_fault_hold
100.0% [4161]trap_pfault
00.07% [3] kmem_back
100.0% [3] kmem_malloc
27.98% [3005] vm_page_activate
100.0% [3005] vm_fault_hold
100.0% [3005]trap_pfault
20.64% [2217] vm_pageq_remove
64.73% [1435] vm_page_free_toq
63.41% [910] vm_object_terminate
make -j 4
06.86% [4222] pagezero @ /boot/kernel/kernel
98.39% [4154] trap_pfault
100.0% [4154] trap
01.11% [47]vm_fault_hold
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
Known problem. There is an open disagreement about how to improve the
granularity of locking in pmap.
-Kip
On Tue, Apr 24, 2012 at 9:14 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote:
I treid make -j 30 build{world,kernel} (latest -CURRENT) on 24-core machine
and see poor
scalability of pmap/mtx -- more then 50% cpu spend on system time.
pmcstat:
@ CPU_CLK_UNHALTED_CORE [194841 samples]
42.65% [83102] _mtx_lock_sleep @ /boot/kernel/kernel
40.97% [34051] pmap_enter
100.0% [34051] vm_fault_hold
100.0% [34051] trap_pfault
30.40% [25262] vm_page_activate
100.0% [25262] vm_fault_hold
100.0% [25262] trap_pfault
18.41% [15300] vm_pageq_remove
73.63% [11266] vm_page_free_toq
70.69% [7964] vm_object_terminate
29.31% [3302] vm_object_page_remove
26.37% [4034] vm_fault_hold
100.0% [4034] trap_pfault
make -j 8:
15.44% [10740] _mtx_lock_sleep @ /boot/kernel/kernel
38.77% [4164] pmap_enter
99.93% [4161] vm_fault_hold
100.0% [4161] trap_pfault
00.07% [3] kmem_back
100.0% [3] kmem_malloc
27.98% [3005] vm_page_activate
100.0% [3005] vm_fault_hold
100.0% [3005] trap_pfault
20.64% [2217] vm_pageq_remove
64.73% [1435] vm_page_free_toq
63.41% [910] vm_object_terminate
make -j 4
06.86% [4222] pagezero @ /boot/kernel/kernel
98.39% [4154] trap_pfault
100.0% [4154] trap
01.11% [47] vm_fault_hold
___
freebsd-performa...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to freebsd-performance-unsubscr...@freebsd.org
--
“The real damage is done by those millions who want to 'get by.'
The ordinary men who just want to be left in peace. Those who don’t
want their little lives disturbed by anything bigger than themselves.
Those with no sides and no causes. Those who won’t take measure of
their own strength, for fear of antagonizing their own weakness. Those
who don’t like to make waves—or enemies.
Those for whom freedom, honour, truth, and principles are only
literature. Those who live small, love small, die small. It’s the
reductionist approach to life: if you keep it small, you’ll keep it
under control. If you don’t make any noise, the bogeyman won’t find
you.
But it’s all an illusion, because they die too, those people who
roll up their spirits into tiny little balls so as to be safe. Safe?!
From what? Life is always on the edge of death; narrow streets lead to
the same place as wide avenues, and a little candle burns itself out
just like a flaming torch does.
I choose my own way to burn.”
Sophie Scholl
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
On Tue, Apr 24, 2012 at 09:27:30PM +0200, K. Macy wrote: Known problem. There is an open disagreement about how to improve the granularity of locking in pmap. split locking to process-specific information and global information? use lock-free lists (i see TAILQ_INSERT_TAIL in pmap_enter)? sorry for stupidity, if any. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
No. I developed a patch from Jeffr that pushed the vm_page_lock array down in to the machine dependent code, replacing most of the uses of the single vm_page_queue_lock. However, alc doesn't like the design and has not proposed an alternative. -Kip On Tue, Apr 24, 2012 at 10:36 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 09:27:30PM +0200, K. Macy wrote: Known problem. There is an open disagreement about how to improve the granularity of locking in pmap. split locking to process-specific information and global information? use lock-free lists (i see TAILQ_INSERT_TAIL in pmap_enter)? sorry for stupidity, if any. -- “The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don’t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won’t take measure of their own strength, for fear of antagonizing their own weakness. Those who don’t like to make waves—or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It’s the reductionist approach to life: if you keep it small, you’ll keep it under control. If you don’t make any noise, the bogeyman won’t find you. But it’s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.” Sophie Scholl ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
On Tue, Apr 24, 2012 at 10:43:08PM +0200, K. Macy wrote: No. I developed a patch from Jeffr that pushed the vm_page_lock array down in to the machine dependent code, replacing most of the uses of the single vm_page_queue_lock. However, alc doesn't like the design and has not proposed an alternative. can i test this? -Kip On Tue, Apr 24, 2012 at 10:36 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 09:27:30PM +0200, K. Macy wrote: Known problem. There is an open disagreement about how to improve the granularity of locking in pmap. split locking to process-specific information and global information? use lock-free lists (i see TAILQ_INSERT_TAIL in pmap_enter)? sorry for stupidity, if any. -- ?The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don?t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won?t take measure of their own strength, for fear of antagonizing their own weakness. Those who don?t like to make waves?or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It?s the reductionist approach to life: if you keep it small, you?ll keep it under control. If you don?t make any noise, the bogeyman won?t find you. But it?s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.? Sophie Scholl ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
It's a bit dated at this point. Nonetheless, when gitorious is able to give something other than 503 to my search queries I'll post it. On Tue, Apr 24, 2012 at 10:45 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 10:43:08PM +0200, K. Macy wrote: No. I developed a patch from Jeffr that pushed the vm_page_lock array down in to the machine dependent code, replacing most of the uses of the single vm_page_queue_lock. However, alc doesn't like the design and has not proposed an alternative. can i test this? -Kip On Tue, Apr 24, 2012 at 10:36 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 09:27:30PM +0200, K. Macy wrote: Known problem. There is an open disagreement about how to improve the granularity of locking in pmap. split locking to process-specific information and global information? use lock-free lists (i see TAILQ_INSERT_TAIL in pmap_enter)? sorry for stupidity, if any. -- ?The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don?t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won?t take measure of their own strength, for fear of antagonizing their own weakness. Those who don?t like to make waves?or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It?s the reductionist approach to life: if you keep it small, you?ll keep it under control. If you don?t make any noise, the bogeyman won?t find you. But it?s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.? Sophie Scholl -- “The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don’t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won’t take measure of their own strength, for fear of antagonizing their own weakness. Those who don’t like to make waves—or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It’s the reductionist approach to life: if you keep it small, you’ll keep it under control. If you don’t make any noise, the bogeyman won’t find you. But it’s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.” Sophie Scholl ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: pmap and mtx scalability problem
You can try these. Your mileage *will* vary. https://gitorious.org/~kmm/freebsd/kmm-sandbox/commits/work/svn_release_8_1_0_page_lock https://gitorious.org/~kmm/freebsd/kmm-sandbox/commits/work/svn_trunk_page_lock On Tue, Apr 24, 2012 at 10:51 PM, K. Macy km...@freebsd.org wrote: It's a bit dated at this point. Nonetheless, when gitorious is able to give something other than 503 to my search queries I'll post it. On Tue, Apr 24, 2012 at 10:45 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 10:43:08PM +0200, K. Macy wrote: No. I developed a patch from Jeffr that pushed the vm_page_lock array down in to the machine dependent code, replacing most of the uses of the single vm_page_queue_lock. However, alc doesn't like the design and has not proposed an alternative. can i test this? -Kip On Tue, Apr 24, 2012 at 10:36 PM, Slawa Olhovchenkov s...@zxy.spb.ru wrote: On Tue, Apr 24, 2012 at 09:27:30PM +0200, K. Macy wrote: Known problem. There is an open disagreement about how to improve the granularity of locking in pmap. split locking to process-specific information and global information? use lock-free lists (i see TAILQ_INSERT_TAIL in pmap_enter)? sorry for stupidity, if any. -- ?The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don?t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won?t take measure of their own strength, for fear of antagonizing their own weakness. Those who don?t like to make waves?or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It?s the reductionist approach to life: if you keep it small, you?ll keep it under control. If you don?t make any noise, the bogeyman won?t find you. But it?s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.? Sophie Scholl -- “The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don’t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won’t take measure of their own strength, for fear of antagonizing their own weakness. Those who don’t like to make waves—or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It’s the reductionist approach to life: if you keep it small, you’ll keep it under control. If you don’t make any noise, the bogeyman won’t find you. But it’s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.” Sophie Scholl -- “The real damage is done by those millions who want to 'get by.' The ordinary men who just want to be left in peace. Those who don’t want their little lives disturbed by anything bigger than themselves. Those with no sides and no causes. Those who won’t take measure of their own strength, for fear of antagonizing their own weakness. Those who don’t like to make waves—or enemies. Those for whom freedom, honour, truth, and principles are only literature. Those who live small, love small, die small. It’s the reductionist approach to life: if you keep it small, you’ll keep it under control. If you don’t make any noise, the bogeyman won’t find you. But it’s all an illusion, because they die too, those people who roll up their spirits into tiny little balls so as to be safe. Safe?! From what? Life is always on the edge of death; narrow streets lead to the same place as wide avenues, and a little candle burns itself out just like a flaming torch does. I choose my own way to burn.” Sophie Scholl ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
