On 03/23/12 00:24, Nikolay Samofatov wrote: > Hello, All! > > We completed (conditional) implementation of lock manager buffer > 2 GB and > put it into production. > > The production system tends to be 100% CPU bound now (as there is a few days > back-log of work). > All 24 cores are busy, 450 GB RAM utilized for one database. > > Roughly ~50% of CPU time is spent in Linux kernel
Interesting, where is the time spent when changing rings is counted - user or kernel time? > on average and lock manager mutex wait hovers > around 40-50%. > Afraid it's as expected for 24 cores system. > Live kernel profiler output of loaded system is as follows (generated with > OProfile): > > 7073900 3.6298 vmlinux vmlinux > copy_user_generic_string > 5554648 2.8502 vmlinux vmlinux __wake_up > 4695015 2.4091 vmlinux vmlinux > prepare_to_wait_exclusive > 4670651 2.3966 vmlinux vmlinux > futex_wait_setup > 3988807 2.0468 vmlinux vmlinux futex_wake > 2006843 1.0298 vmlinux vmlinux intel_idle > 1321253 0.6780 vmlinux vmlinux schedule > 1022927 0.5249 vmlinux vmlinux put_page > 1001815 0.5141 vmlinux vmlinux > finish_wait > 735902 0.3776 vmlinux vmlinux > try_to_wake_up > 716559 0.3677 vmlinux vmlinux > _atomic_dec_and_lock > 674752 0.3462 vmlinux vmlinux get_page > 646537 0.3318 vmlinux vmlinux > tick_nohz_stop_sched_tick > 628516 0.3225 vmlinux vmlinux > task_rq_lock > 594621 0.3051 vmlinux vmlinux > update_curr > 583779 0.2996 vmlinux vmlinux > native_write_msr_safe > 527673 0.2708 vmlinux vmlinux > select_nohz_load_balancer > 504132 0.2587 vmlinux vmlinux > get_futex_key > 460908 0.2365 vmlinux vmlinux > __audit_syscall_exit > 447328 0.2295 vmlinux vmlinux > enqueue_hrtimer > 443841 0.2277 vmlinux vmlinux > __wake_up_bit > 429380 0.2203 vmlinux vmlinux > thread_return > 401287 0.2059 vmlinux vmlinux > __switch_to > 397639 0.2040 vmlinux vmlinux > update_cfs_shares > 381845 0.1959 vmlinux vmlinux > rb_get_reader_page > 377837 0.1939 vmlinux vmlinux ktime_get > 371547 0.1907 vmlinux vmlinux > native_read_tsc > 369331 0.1895 vmlinux vmlinux > radix_tree_lookup_slot > > > This demonstrates that to scale effectively past 24-40 cores per node, > Firebird needs to implement > shared buffer cache. Luckily it's implemented in trunk. And it's superserver - i.e. latches instead locks in lock manager. > Lock manager mutex operations and copying from file system cache to userspace > emerge as hard bottleneck. > > With lock manager change to 64-bit one 24 core node handles ~10 million user > transactions daily. Is it going to reach 2**31 limit soon? 200 days if I'm not mistaken. > reliably, at times having >1500 concurrent INSERT/UPDATE requests. > > Shall we merge lock manager changes? In trunk - definitely. What about 2.5 - not sure. May be having patch for high-end systems is better approach than commit in svn? ------------------------------------------------------------------------------ This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
