Title: LDAP performance
The client OS is BlueCoat.  Are you saying that 42217 is too high a port for windows to accept?  I thought it could go all the way up to 65535?


From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of joe
Sent: Tuesday, June 14, 2005 6:04 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] LDAP performance

From port 42217? What was the client OS again? That doesn't sound like Windows. Windows client I would expect port down in the range specified by the KB article. That modification they specify is for the client machine.
 
For instance, I fired up several queries to one of my DCs and let them complete, now I do a NETSTAT -A on my client and I see
 
  TCP    fastmofo:2497          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2526          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2535          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2552          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2575          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2597          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2602          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2609          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2665          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2675          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2686          2k3dc10.child1.joe.com:ldap  TIME_WAIT
  TCP    fastmofo:2697          2k3dc10.child1.joe.com:ldap  TIME_WAIT
 
These connections are all closed, but waiting on final cleanup. You can do a google on time_wait and get a better explanation than I can give. According to that article, if I get enough of these to eat up the pre-specified range on the client, the client will not be able to make any more connections to the DC. The KB tells you how to open up more ports for use on the client.
 
 
 
The trace should obviously go
 
Client:x -> Server:389   SYN
Server:389 -> Client:x   SYN ACK
Client:x -> Server:389   ACK
 
and then go into an LDAP conversation starting most likely with a rootdse search or a bind.
 
and then at the end you should see
 
Client:x -> Server:389   FIN ACK
Server:389 -> Client:x   ACK
Server:389 -> Client:x   FIN ACK
Client:x -> Server:389   ACK
 
assuming they are closing the connections down properly.
 
 
The trace below doesn't show this occurring. The trace is already filtered though with hundreds of packets missing so who knows what got screened, it could be a misrepresentation of what is going on if someone didn't do the trace or the filter quite right. If you get MS involved, you will almost certainly need to send them the whole trace so they can see everything going on. Especially some queries working and some not. I understand why you may not want to post a full trace to a group like this. If you want, I would be willing to look at a full trace as well, just zip and send to me offline and I will look at it in the evening when I get a chance. Please send a format that can be opened in Ethereal. Digging through text traces is a pain in the butt. It doesn't allow us to use the computer tools that do this work so much better than we do.
 
  joe
 
 
 


From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph
Sent: Tuesday, June 14, 2005 12:06 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] LDAP performance

The application owner says that they are not seeing any extended error info.  The connections are simply being disconnected.  Here is part of the network trace the network guys sent me.  This basically shows the same connection attempting to connect to 389 from port 42217.  as you can see it trys a syn, waits a couple minutes, then trys again.  It never gets acked.
 
I have the LDAP calls as well however; (CISSPs close your ears), they are simple binds so I'll need to do some cleaning before sending them out ;-)
 
In a nutshell here is the sequence that the application goes through every time it auths a user:
 
1.  Use  a service account to bind to the directory
2.  Search for the user account using filter "(samaccountname=x)" retrieve the DN.
2.  Now that it has the DN, bind as the user.
 
It does this for every single user auth.  Terribly inefficient I know.  The newer version of the product does not bind with the service account every single time and actually we do have the newer version implemented in one location.  The newer version has not seen this problem to date.
 
I'll go ahead and check out these articles,  Thanks
 
*******************************************************************************************************
No.     Time        Source                Destination           Protocol Info
   6827 32.129301   **.**.**.**           **.**.**.**        TCP      42217 > ldap [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0 TSV=5999338 TSER=0
 
Frame 6827 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 00:01:d7:14:d2:c1, Dst: 00:00:0c:07:ac:0e
802.1q Virtual LAN
Internet Protocol, Src Addr: **.**.**.** (**.**.**.**), Dst Addr: **.**.**.** (**.**.**.**)
Transmission Control Protocol, Src Port: 42217 (42217), Dst Port: ldap (389), Seq: 0, Ack: 0, Len: 0
 
No.     Time        Source                Destination           Protocol Info
   6943 33.121101   **.**.**.**           **.**.**.**        TCP      42217 > ldap [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0 TSV=5999340 TSER=0
 
Frame 6943 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 00:01:d7:14:d2:c1, Dst: 00:00:0c:07:ac:0e
802.1q Virtual LAN
Internet Protocol, Src Addr: **.**.**.** (**.**.**.**), Dst Addr: **.**.**.** (**.**.**.**)
Transmission Control Protocol, Src Port: 42217 (42217), Dst Port: ldap (389), Seq: 0, Ack: 0, Len: 0
 
No.     Time        Source                Destination           Protocol Info
   7692 36.132503   **.**.**.**           **.**.**.**        TCP      42217 > ldap [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0 TSV=5999346 TSER=0
 
Frame 7692 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 00:01:d7:14:d2:c1, Dst: 00:00:0c:07:ac:0e
802.1q Virtual LAN
Internet Protocol, Src Addr: **.**.**.** (**.**.**.**), Dst Addr: **.**.**.** (**.**.**.**)
Transmission Control Protocol, Src Port: 42217 (42217), Dst Port: ldap (389), Seq: 0, Ack: 0, Len: 0
 
No.     Time        Source                Destination           Protocol Info
   8267 39.142852   **.**.**.**           **.**.**.**        TCP      42217 > ldap [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0 TSV=5999352 TSER=0
 
Frame 8267 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 00:01:d7:14:d2:c1, Dst: 00:00:0c:07:ac:0e
802.1q Virtual LAN
Internet Protocol, Src Addr: **.**.**.** (**.**.**.**), Dst Addr: **.**.**.** (**.**.**.**)
Transmission Control Protocol, Src Port: 42217 (42217), Dst Port: ldap (389), Seq: 0, Ack: 0, Len: 0
 
No.     Time        Source                Destination           Protocol Info
   9160 43.155866   **.**.**.**           **.**.**.**        TCP      42217 > ldap [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460 WS=0 TSV=5999360 TSER=0
 
Frame 9160 (78 bytes on wire, 78 bytes captured)
Ethernet II, Src: 00:01:d7:14:d2:c1, Dst: 00:00:0c:07:ac:0e
802.1q Virtual LAN
Internet Protocol, Src Addr: **.**.**.** (**.**.**.**), Dst Addr: **.**.**.** (**.**.**.**)
Transmission Control Protocol, Src Port: 42217 (42217), Dst Port: ldap (389), Seq: 0, Ack: 0, Len: 0

************************************************************************************************************************************************************


From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of joe
Sent: Monday, June 13, 2005 10:09 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] LDAP performance

What errors specifically are the clients seeing? Is the server returning any extended information or are the connections just dying on the vine? And if so are you sure? As Eric indicated, running through a trace would probably be mucho helpful.
 
What type of client? If Windows, this KB may seem odd, but check out http://support.microsoft.com/?id=836429
 
What you are describing sounds like something I heard from another friend of mine doing some auth testing and the KB above ended up being what the issue was related to.
 
 
I am assuming they are most likely doing simple binds? If so, possibly the app developers may want to look at LDAP_OPT_FAST_CONCURRENT_BIND available in Windows Server 2003 AD which allows multiple binds over a single connection and should be faster overall. Read more here
 
http://msdn.microsoft.com/library/default.asp?url="">
 
 
 
 
 
 


From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Isenhour, Joseph
Sent: Monday, June 13, 2005 7:55 PM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] LDAP performance

We're running into what appears to be some performance issues.  We have several AD servers that we dedicate to doing LDAP authentications for various applications.  We recently added a new application that performs a large number of binds.  The day we cut the application over to AD LDAP the application owners began complaining that an average of 1 to 2 LDAP requests are being dropped every minute.  Here are the details:

Application:  Issues an average of 100 binds per second.  Average of 50 queries per second using filter "(samaccountname=X)" and requesting the DN as the return.

HW:  2 Domain Controllers.  Each is quad proc 2.4GHZ.  Each has 4GB of RAM with the 3GB switch set.

I ran this through ADSizer and it recommended one server with about half the capacity that is built into each of these servers.

I've run several performance checks on these machines and it appears that they are barely breaking a sweat in terms of available resources.  I've tweaked our default LDAP policies to add additional queries per proc and allowed larger buffers.  But the app owner is still complaining.

The network team has recommended that I increase the TCP listening queue on the servers.  They suspect this because they are seeing a few syns that never get acked.  I'm not familiar with how to do this in Windows and am not sure if that is really something I should be concerned with.  Can anyone out there vouch for this theory?  Or perhaps offer another theory as to why the DCs seem to not keep up with the load?

Thanks

One other thing,  I set the LDAP diags to two and found the following warning poping up from time to time:

**************************************************************************************************
Event Type:     Warning
Event Source:   NTDS LDAP
Event Category: LDAP Interface
Event ID:       1216
Date:           6/13/2005
Time:           6:34:37 PM
User:           N/A
Computer:       ******************
Description:
Internal event: An LDAP client connection was closed because of an error.
 
Client ID:
427107
 
Additional Data
Error value:
995 The I/O operation has been aborted because of either a thread exit or an application request.
Internal ID:
c0602ec

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

**************************************************************************************************

Reply via email to