On Fri, Mar 23, 2012 at 9:13 AM, Alex Peshkoff <[email protected]> wrote: > On 03/23/12 00:24, Nikolay Samofatov wrote: >> Hello, All! >> >> We completed (conditional) implementation of lock manager buffer > 2 GB and >> put it into production. >> >> The production system tends to be 100% CPU bound now (as there is a few days >> back-log of work). >> All 24 cores are busy, 450 GB RAM utilized for one database. >> >> Roughly ~50% of CPU time is spent in Linux kernel > > Interesting, where is the time spent when changing rings is counted - > user or kernel time? > >> on average and lock manager mutex wait hovers >> around 40-50%. >> > > Afraid it's as expected for 24 cores system. > >> Live kernel profiler output of loaded system is as follows (generated with >> OProfile): >> >> 7073900 3.6298 vmlinux vmlinux >> copy_user_generic_string >> 5554648 2.8502 vmlinux vmlinux __wake_up >> 4695015 2.4091 vmlinux vmlinux >> prepare_to_wait_exclusive >> 4670651 2.3966 vmlinux vmlinux >> futex_wait_setup >> 3988807 2.0468 vmlinux vmlinux >> futex_wake >> 2006843 1.0298 vmlinux vmlinux >> intel_idle >> 1321253 0.6780 vmlinux vmlinux schedule >> 1022927 0.5249 vmlinux vmlinux put_page >> 1001815 0.5141 vmlinux vmlinux >> finish_wait >> 735902 0.3776 vmlinux vmlinux >> try_to_wake_up >> 716559 0.3677 vmlinux vmlinux >> _atomic_dec_and_lock >> 674752 0.3462 vmlinux vmlinux get_page >> 646537 0.3318 vmlinux vmlinux >> tick_nohz_stop_sched_tick >> 628516 0.3225 vmlinux vmlinux >> task_rq_lock >> 594621 0.3051 vmlinux vmlinux >> update_curr >> 583779 0.2996 vmlinux vmlinux >> native_write_msr_safe >> 527673 0.2708 vmlinux vmlinux >> select_nohz_load_balancer >> 504132 0.2587 vmlinux vmlinux >> get_futex_key >> 460908 0.2365 vmlinux vmlinux >> __audit_syscall_exit >> 447328 0.2295 vmlinux vmlinux >> enqueue_hrtimer >> 443841 0.2277 vmlinux vmlinux >> __wake_up_bit >> 429380 0.2203 vmlinux vmlinux >> thread_return >> 401287 0.2059 vmlinux vmlinux >> __switch_to >> 397639 0.2040 vmlinux vmlinux >> update_cfs_shares >> 381845 0.1959 vmlinux vmlinux >> rb_get_reader_page >> 377837 0.1939 vmlinux vmlinux ktime_get >> 371547 0.1907 vmlinux vmlinux >> native_read_tsc >> 369331 0.1895 vmlinux vmlinux >> radix_tree_lookup_slot >> >> >> This demonstrates that to scale effectively past 24-40 cores per node, >> Firebird needs to implement >> shared buffer cache. > > Luckily it's implemented in trunk. > And it's superserver - i.e. latches instead locks in lock manager. > >> Lock manager mutex operations and copying from file system cache to >> userspace emerge as hard bottleneck. >> >> With lock manager change to 64-bit one 24 core node handles ~10 million user >> transactions daily. > > Is it going to reach 2**31 limit soon? 200 days if I'm not mistaken. > >> reliably, at times having >1500 concurrent INSERT/UPDATE requests. >> >> Shall we merge lock manager changes? > > In trunk - definitely. What about 2.5 - not sure. May be having patch > for high-end systems is better approach than commit in svn? Another option is On github you can fork the firebird 2.5 branch https://github.com/asfernandes/firebird/tree/B2_5_Release and keep the set of patches for high-end systems in your fork if you want
------------------------------------------------------------------------------ This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
