Hi I did some further profiling winbindd and found that there was no network traffic but internally string functions were consuming most of the cpu. Here is the oprofile output of the winbindd which was consuming 99% of cpu [EMAIL PROTECTED] new]# opreport -l --demangle=smart "/mnt/RAIDB/vol/new/winbindd" | more CPU: PMC-Sierra RM9000, speed 0 MHz (estimated) Counted INSTRUCTIONS_ISSUED events (Instructions issued) with a unit mask of 0x00 (No unit mask) count 100000 samples % symbol name 77 20.5882 convert_string 57 15.2406 convert_string_internal 43 11.4973 check_dos_char_slowly 37 9.8930 ascii_push 27 7.2193 smb_iconv 26 6.9519 strwicmp 18 4.8128 toupper_w 15 4.0107 toupper_ascii 11 2.9412 lazy_initialize_conv 10 2.6738 dopr 9 2.4064 format_debug_text 9 2.4064 init_doschar_table 7 1.8717 map_parameter 4 1.0695 dopr_outch 4 1.0695 lp_do_parameter 3 0.8021 fmtstr 2 0.5348 Parameter 2 0.5348 copy_service 2 0.5348 mygetc 1 0.2674 EatComment 1 0.2674 EatWhitespace 1 0.2674 charset_name 1 0.2674 iface_comp 1 0.2674 lp_save_defaults 1 0.2674 lp_string 1 0.2674 set_server_role 1 0.2674 smb_probe_module 1 0.2674 smb_xstrdup 1 0.2674 string_free 1 0.2674 tdb_read
When I backtraced the winbindd through oprofile, here is the call graph which I get, Counted INSTRUCTIONS_ISSUED events (Instructions issued) with a unit mask of 0x00 (No unit mask) count 100000 samples % image name symbol name ------------------------------------------------------------------------------- 2 7.1429 winbindd convert_string 26 92.8571 winbindd check_dos_char_slowly 77 20.5882 winbindd convert_string 77 65.8120 winbindd convert_string [self] 38 32.4786 winbindd convert_string_internal 2 1.7094 winbindd convert_string ------------------------------------------------------------------------------- 38 100.000 winbindd convert_string 57 15.2406 winbindd convert_string_internal 57 69.5122 winbindd convert_string_internal [self] 15 18.2927 winbindd smb_iconv 10 12.1951 winbindd lazy_initialize_conv ------------------------------------------------------------------------------- 16 32.0000 winbindd check_dos_char_slowly 34 68.0000 winbindd init_doschar_table 43 11.4973 winbindd check_dos_char_slowly 43 50.5882 winbindd check_dos_char_slowly [self] 26 30.5882 winbindd convert_string 16 18.8235 winbindd check_dos_char_slowly ------------------------------------------------------------------------------- 28 100.000 winbindd smb_iconv 37 9.8930 winbindd ascii_push 37 100.000 winbindd ascii_push [self] ------------------------------------------------------------------------------- 15 100.000 winbindd convert_string_internal 27 7.2193 winbindd smb_iconv 28 50.9091 winbindd ascii_push 27 49.0909 winbindd smb_iconv [self] ------------------------------------------------------------------------------- 3 25.0000 winbindd strwicmp 9 75.0000 winbindd map_parameter 26 6.9519 winbindd strwicmp 30 43.4783 winbindd toupper_ascii 26 37.6812 winbindd strwicmp [self] 10 14.4928 libc-2.3.5.so (no symbols) 3 4.3478 winbindd strwicmp Also pidstat output says that winbindd is consuming more in kernel as well. Linux 2.6.18 (NAS0018f80551d3) 11/09/08 _mips_ 00:27:50 PID %usr %system %guest %CPU CPU Command 00:27:51 31001 45.45 47.47 0.00 92.93 0 winbindd 00:27:52 31001 41.58 56.44 0.00 98.02 0 winbindd 00:27:53 31001 29.00 61.00 0.00 90.00 0 winbindd 00:27:54 31001 33.66 65.35 0.00 99.01 0 winbindd 00:27:55 31001 36.00 54.00 0.00 90.00 0 winbindd 00:27:56 31001 40.40 52.53 0.00 92.93 0 winbindd 00:27:57 31001 34.65 63.37 0.00 98.02 0 winbindd 00:27:58 31001 36.27 52.94 0.00 89.22 0 winbindd 00:27:59 31001 38.00 61.00 0.00 99.00 0 winbindd Has anybody ever faced similar issue before? Thanks in advance Sagar --- On Wed, 11/5/08, Gerald (Jerry) Carter <[EMAIL PROTECTED]> wrote: > From: Gerald (Jerry) Carter <[EMAIL PROTECTED]> > Subject: Re: [Samba] winbindd consuming 99% CPU > To: [EMAIL PROTECTED] > Cc: samba@lists.samba.org, [EMAIL PROTECTED] > Date: Wednesday, November 5, 2008, 7:48 PM > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Sagar Borikar wrote: > > Hi All, > > > > We are currently caught up in a weird situation while > using samba > > 3.0.28. There is a requirement from the customer that > we should support > > ADS with 64000 objects. When we implemented the > feature, we found > > couple of things mentioned below: > > > > 1. After joining ADS with 64k users, samba takes > around 20 - 25 min > > to populate idmap_cache.tdb and winbindd_cache.tdb > This is because > > we haven't turned of enumeration and we would be > fixing it for > > next maintenance release of the NAS product. > > > > 2. We periodically check the quota state of all the > users > > using repquota command. It refers to > /etc/nsswitch.conf for mapping > > user ids to names. And winbindd inherently should look > for its > > cache as it is populated completely to do the > conversion. > > > > But we observed that every time the repquota command > is invoked, > > winbindd starts consuming 98% of CPU. It fetches the > info from > > ADS and not from local cache. So we are wondering why > it doesn't > > make use of local cache for mapping the ids to names? > > Sagar, > > The enumeration is killing you performance wise. As a > workaround > you could try in creasing the "winbind cache > time" to something like > 1 or 2 hours. But there is a constant struggle between > cache times > and accuracy. > > > > > cheers, jerry > - -- > ===================================================================== > Samba ------- > http://www.samba.org > Likewise Software --------- > http://www.likewisesoftware.com > "What man is a man who does not make the world > better?" --Balian > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > Comment: Using GnuPG with Mozilla - > http://enigmail.mozdev.org > > iD8DBQFJEatJIR7qMdg1EfYRAhmyAJ9CcrZLclYZ+P8q0hHI7EsUt7kRKwCg0km4 > LWFcHLKbEfl8UMYnd4JWqpA= > =3KW6 > -----END PGP SIGNATURE----- -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba