If I had to guess, I'd say you were runing out of BBCredits on the ISL links between the two datacenters due to the distance. BBCredits are basically buffers on the link. It takes time for a packet to travel the 30km. If you don't have enought credits to allow enought in-transit packets you loose thoughput. This is similar to the tcp window processing to keep a WAN link full.
Below is my writeup I did for our docs. NOTE: For Brocade switches, the Extended Fabric feature which allows allocating more credits to a link is a extra charge feature, although I believe it's standard on DCX director switches. Brocade Extended Fabric Feature 1) What is it? Fibre Channel switches use Buffer-to-Buffer-Credits (BBC's) to control flow control of packets. In different places you will see this refered to as Buffer Credit, Frame Buffers, BB_Credit, and even some other names. 2) What does it do? In a fibre channel switch, a reply packet is required for every sent packet (I think). BBC's represent memory buffers. The number of them assigned to a port dictates how many packets a switch can send that without receiving a reply. Once this number of packets have been sent, the switch will NOT send any more packets until it receives a reply. This prevents a transmitting switch from overrunning the buffers of a receiving switch. 3) Why does it need changed for long distance ISL links? Think of a packet as requiring a finite amount of time to travel the fibre cable. A fibre channel frame/packet is approximately 2k in size. At 1Gbps a frame is about 4km long. At 2Gbps the same frame is aboutt 2km long. A 10km ISL link running at 2Gbps would be able to hold 5 frames in flight. A 50km ISL link at 2Gbps would be able to hold 25 frames. To NOT have enough BBC's available to fill the cable results in decreased throughput. Unfortunely, to be able to adjust the BBC's of a port you must PURCHASE an extra feature for Brocade swithes: Extended Fabric. It comes as a license which you enter to enable this feature. 4) How do you implement it? Extended Fabric works on a PER PORT basis. For any switch port for which you want to enable Extended Fabric, you enter the cmd: portcfglongdistance <port> <distance_level> <init_flag> <distance> <port> = switch port # <distance_level> = a MODE for the port that sets the distance of the ISL link <init_flag> = port initialization setting. Should be "1" for FE. <distance> = for MODE "LD", this is the max distance to configure for. Command used to configure Extended Fabric on Fab5: portcfglongdistance 13 LD 1 50 13 = switch port 13 (or whichever one you are working on) LD = Mode "LD" is AUTO - it COMPUTES the distance of the ISL link. 1 = just use it! 50 = For AUTO mode (LD mode), this sets the MAX distance it will use. For the GO-to-ISOC links on the new DWDM, the LD mode set the distance to 17km. To un-set a port from Extended Fabric (ie: turn it off), issue the cmd: portcfglongdistance 13 L0 13 = switch port L0 = default distance (no extended fabric) 5) Requirements For Extended Fabric to work, you must . . . . - Ports MUST be running at the same speed. - Ports MUST be set to the same MODE, init flag, and max distance. In other words, the ports MUST be setup and operating IDENTICALLY. 6) IMPORTANT CONSIDERATION The extra credits that Extended Fabric allows you to allocate to a port ARE NOT FREE. The are STOLEN from another port. IF a certain distance on a Extended Fabric setup for a port requires enought credits that some other ports is starved, the swith can/will DISABLE a the starved port. This behavior is different between 3900 and 4100 class switches, but this is a situation you MUST understand. 7) More Info Brocase Fabric OS Reference Manual (command reference manual) see cmds: portcfglongdistance portcfgshow portshow Brocade Fabric OS Procedures Guide see section: Extended Fabrics Procedures See attached documents 8) Example fab5_103:admin> switchshow switchName: fab5_103 switchType: 32.0 switchState: Online switchMode: Native switchRole: Principal switchDomain: 103 switchId: fffc67 switchWwn: 10:00:00:05:1e:35:bc:8e zoning: ON (fabric5) switchBeacon: OFF Area Port Media Speed State ============================== 0 0 id N4 Online E-Port 10:00:00:05:1e:35:c0:7e "fab5_102" (downstream)(Trunk master) 1 1 id N4 Online E-Port (Trunk port, master is Port 0 ) 2 2 id N2 Online F-Port 10:00:00:00:c9:4e:50:03 3 3 id N2 Online F-Port 10:00:00:00:c9:4e:4d:ec 4 4 id N4 Online F-Port 50:05:07:63:0f:4d:8b:01 5 5 id N4 Online F-Port 50:05:07:63:0f:4d:8b:03 6 6 id N4 Online F-Port 50:05:07:63:0f:4d:8b:05 7 7 id N4 Online F-Port 50:05:07:63:0f:4d:8b:07 8 8 id N2 Online F-Port 10:00:00:00:c9:4e:4f:d1 9 9 id N4 Online F-Port 50:05:07:63:0f:4d:8b:0c 10 10 id N4 No_Light 11 11 id N4 No_Light 12 12 id N4 Online F-Port 50:05:07:63:0f:4d:8b:09 13 13 id N4 No_Light 14 14 id N4 No_Light 15 15 -- N4 No_Module 16 16 id N1 Online L-Port 1 public 17 17 id N1 Online L-Port 1 public 18 18 id N1 Online L-Port 1 public 19 19 id N1 Online L-Port 1 public 20 20 id N1 Online L-Port 1 public 21 21 id N1 Online L-Port 1 public 22 22 id N1 Online L-Port 1 public 23 23 id N1 Online L-Port 1 public 24 24 id N2 Online LD E-Port (Trunk port, master is Port 28 ) 25 25 id N4 No_Light 26 26 id N4 No_Light 27 27 id N2 Online F-Port 10:00:00:00:c9:3d:a1:3a 28 28 id N2 Online LD E-Port 10:00:00:60:69:90:04:8f "fab5_101" (downstream)(Trunk master) 29 29 id N4 No_Light 30 30 id N4 No_Light 31 31 id N2 Online F-Port 10:00:00:00:c9:3d:a1:49 fab5_103:admin> portshow 28 portName: portHealth: No License Authentication: None portDisableReason: None portCFlags: 0x1 portFlags: 0x490b PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED portType: 10.0 POD Port: License available to enable the port portState: 1 Online portPhys: 6 In_Sync portScn: 16 E_Port Trunk master port port generation number: 126 portId: 671c00 portIfId: 43020018 portWwn: 20:1c:00:05:1e:35:bc:8e portWwn of device(s) connected: 20:19:00:60:69:90:04:8f Distance: auto (desired = 50 Km, actual = 17 Km) portSpeed: N2Gbps Interrupts: 0 Link_failure: 5 Frjt: 0 Unknown: 0 Loss_of_sync: 1922 Fbsy: 0 Lli: 928406 Loss_of_sig: 3822 Proc_rqrd: 445239 Protocol_err: 0 Timed_out: 0 Invalid_word: 0 Rx_flushed: 0 Invalid_crc: 0 Tx_unavail: 0 Delim_err: 0 Free_buffer: 0 Address_err: 2 Overrun: 0 Lr_in: 32 Suspended: 0 Lr_out: 5 Parity_err: 0 Ols_in: 4 2_parity_err: 0 Ols_out: 16 CMI_bus_err: 0 fab5_103:admin> portcfgshow 28 Area Number: 28 Speed Level: AUTO Trunk Port ON Long Distance LD VC Link Init ON Desired Distance 50 Km Locked L_Port OFF Locked G_Port OFF Disabled E_Port OFF ISL R_RDY Mode OFF RSCN Suppressed OFF Persistent Disable OFF NPIV capability ON "Meuleman, Ruud" <ruud.meule...@co RUSGROUP.COM> To Sent by: "ADSM: ADSM-L@VM.MARIST.EDU Dist Stor cc Manager" <ads...@vm.marist Subject .EDU> Tapedrive 3592 E06 speed of 160MB/sec 02/10/2010 10:43 AM Please respond to "ADSM: Dist Stor Manager" <ads...@vm.marist .EDU> Hi, In our TSM environment we use two locations that are about 30 km away from each other. On both sides we use TSM servers and 3592 tapedrives. We use SAN and DWDM between the two locations. Writing data from a TSM server to a tapelibrary that are on the same location have the speed according the technical specifications. A TSM server writing to the other location has not half of the speed. For example: On one side we have 3592 E06 tapedrives, that can write with a speed of 160MB/sec, when the TSM server is on the same location. There is 1 SAN switch between the server and tapelibrary. A TSM server on the other location, that is also connected with that library, get a speed of 60MB/sec. There is a SAN switch and a DWDM switch on one location and also a DWDM switch and a SAN switch on the other location between the server and tapelibrary. The speed of the HBA's in the TSM servers is 2Gb/sec, the speed of the ports on the SAN switches are 2 Gb/sec and 4 Gb/sec and the speed of the DWDM swiches is 2,5 Gb/sec. 2Gb/sec is about 320MB/sec. So the bottleneck is should be 320MB/sec. When there is no other traffic on this network, the speed should be more than 60MB/sec. The speed is of the tests is determined of data of the summary table of the TSM datebase. Can some explain why the speed is very slow? (only 60MB/sec instead of 160MB/sec). Does anyone know how one can determine the speed of DWDM switches? Thanks, Ruud Meuleman ********************************************************************** This transmission is confidential and must not be used or disclosed by anyone other than the intended recipient. Neither Tata Steel Europe Limited nor any of its subsidiaries can accept any responsibility for any use or misuse of the transmission by anyone. For address and company registration details of certain entities within the Corus group of companies, please visit http://www.corusgroup.com/entities ********************************************************************** ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.