Re: TCP and WAN issue
A lot of different theoretical things have been discussed, but basically, if you are running Windows XP, 2000, or 2003 over a WAN with anything more than 10-20ms of latency, make the following change to the registry and you will find a world of difference. Ideally, you would make the change to both sides, but as long as one side has this, it will auto-negotiate with the other side to adjust the Start, Run, regedt32 Go to: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters Edit, New -> DWORD Value Then name it "Tcp1323Opts" (without the quotes) and change the value to 3 That will give you far better performance by enabling automatic TCP window size scaling as per RFC 1323 over your WAN links (hence the key name), but only to other computers which support RFC1323. XP and 2003 default to allowing scalable windows if the far side intiates the negotiation, but that will never happen unless one side has this key set. Values are: 0 (disable RFC 1323 options) - default before creating the key 1 (window scale enabled only) 2 (timestamps enabled only) 3 (both options enabled) Try this and let us know what you see. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
Re: TCP and WAN issue
Andre Oppermann gave the best advice so far IMHO. I'll add a few points. > To quickly sum up the facts and to dispell some misinformation: > - TCP is limited the delay bandwidth product and the socket buffer >sizes. Hm... what about: The TCP socket buffer size limits the achievable throughput-RTT product? :-) > - for a T3 with 70ms your socket buffer on both endss should be >450-512KB. Right. (Victor Reijs' "goodput calculator" says 378kB.) > - TCP is also limited by the round trip time (RTT). This was stated before, wasn't it? > - if your application is working in a request/reply model no amount >of bandwidth will make a difference. The performance is then >entirely dominated by the RTT. The only solution would be to run >multiple sessions in parallel to fill the available bandwidth. Very good point. Also, some applications have internal window limitations. Notably SSH, which has become quite popular as a bulk data transfer method. See http://kb.pert.geant2.net/PERTKB/SecureShell > - Jumbo Frames have definately zero impact on your case as they >don't change any of the limiting parameters and don't make TCP go >faster. Right. Jumbo frames have these potential benefits for bulk transfer: (1) They reduce the forwarding/interrupt overhead in routers and hosts by reducing the number of packets. But in your situation it is quite unlikely that the packet rate is a bottleneck. Modern routers typically forward even small packets at line rate, and modern hosts/OSes/Ethernet adapters have mechanisms such as "interrupt coalescence" and "large send offload" that make the packet size largely irrelevant. But even without these mechanisms and with 1500-byte packets, 45 Mb/s shouldn't be a problem for hosts built in the last ten years, provided they aren't (very) busy with other processing. (2) As Perry Lorier pointed out, jumbo frames accelerate the "additive increase" phases of TCP, so you reach full speed faster both at startup and when recovering from congestion. This may be noticeable when there is competition on the path, or when you have many smaller transfers such that ramp-up time is an issue. (3) Large frames reduce header overhead somewhat. But the improvement going from 1500-byte to 9000-bytes packets is only 2-3%, from ~97% efficiency to ~99.5%. No orders of magnitude here. >There are certain very high-speed and LAN (<5ms) case where it >may make a difference but not here. Cases where jumbo frames might make a difference: When the network path or the hosts are pps-limited (in the >Gb/s range with modern hosts); when you compete with other traffic. I don't see a relation with RTTs - why do you think this is more important on <5ms LANs? > - Your problem is not machine or network speed, only tuning. Probably yes, but it's not clear what is actually happening. As it often happens, the problem is described with very little detail, so experts (and "experts" :-) have a lot of room to speculate. This was the original problem description from Philip Lavine: I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP In the meantime, Philip gave more information, about the throughput he is seeing (no mention how this is measured, whether it is total load on the DS3, throughput for an application/transaction or whatever): This is the exact issue. I can only get between 5-7 Mbps. And about the protocols he is using: I have 2 data transmission scenarios: 1. Microsoft MSMQ data using TCP 2. "Streaming" market data stock quotes transmitted via a TCP sockets It seems quite likely that these applications have their own performance limits in high-RTT situations. Philip, you could try a memory-to-memory-test first, to check whether TCP is really the limiting factor. You could use the TCP tests of iperf, ttcp or netperf, or simply FTP a large-but-not-too-large file to /dev/null multiple times (so that it is cached and you don't measure the speed of your disks). If you find that this, too, gives you only 5-7 Mb/s, then you should look at tuning TCP according to Andre's excellent suggestions quoted below, and check for duplex mismatches and other sources of transmission errors. If you find that the TCP memory-to-memory-test gives you close to DS3 throughput (modulo overhead), then maybe your applications limit throughput over long-RTT paths, and you have to look for tuning opportunities on that level. > Change these settings on both ends and reboot once to get better throughput: > [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] > "SackOpts"=dword:0x1 (enable SACK) > "TcpWindowSize"=dword:0x7D000 (512000 Bytes) > "Tcp1323Opts"=dword:0x3 (enable window scaling and timestamps) > "GlobalMaxTcpWindowSize"=dword:0x7D000 (512000 Bytes) > http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx -- Simon.
Re: TCP and WAN issue
On Mar 28, 2007, at 5:12 AM, Andre Oppermann wrote: Marshall Eubanks wrote: You might want to look at this classic by Stanislav Shalunov http://shlang.com/writing/tcp-perf.html The description on this website is very good. Disclaimer: I'm a FreeBSD TCP/IP network stack kernel hacker. To quickly sum up the facts and to dispell some misinformation: - TCP is limited the delay bandwidth product and the socket buffer sizes. - for a T3 with 70ms your socket buffer on both endss should be 450-512KB. - TCP is also limited by the round trip time (RTT). - if your application is working in a request/reply model no amount of bandwidth will make a difference. The performance is then entirely dominated by the RTT. The only solution would be to run multiple sessions in parallel to fill the available bandwidth. - Jumbo Frames have definately zero impact on your case as they don't change any of the limiting parameters and don't make TCP go faster. There are certain very high-speed and LAN (<5ms) case where it may make a difference but not here. - Your problem is not machine or network speed, only tuning. Change these settings on both ends and reboot once to get better throughput: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip \Parameters] "SackOpts"=dword:0x1 (enable SACK) "TcpWindowSize"=dword:0x7D000 (512000 Bytes) "Tcp1323Opts"=dword:0x3 (enable window scaling and timestamps) "GlobalMaxTcpWindowSize"=dword:0x7D000 (512000 Bytes) http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx And, of course, if you have Ethernet duplex or other mismatch issues anywhere along the path, performance will be bad. Regards Marshall -- Andre Marshall On Mar 27, 2007, at 4:26 PM, Philip Lavine wrote: To all, I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? Philip __ __ Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091
Re: TCP and WAN issue
Marshall Eubanks wrote: You might want to look at this classic by Stanislav Shalunov http://shlang.com/writing/tcp-perf.html The description on this website is very good. Disclaimer: I'm a FreeBSD TCP/IP network stack kernel hacker. To quickly sum up the facts and to dispell some misinformation: - TCP is limited the delay bandwidth product and the socket buffer sizes. - for a T3 with 70ms your socket buffer on both endss should be 450-512KB. - TCP is also limited by the round trip time (RTT). - if your application is working in a request/reply model no amount of bandwidth will make a difference. The performance is then entirely dominated by the RTT. The only solution would be to run multiple sessions in parallel to fill the available bandwidth. - Jumbo Frames have definately zero impact on your case as they don't change any of the limiting parameters and don't make TCP go faster. There are certain very high-speed and LAN (<5ms) case where it may make a difference but not here. - Your problem is not machine or network speed, only tuning. Change these settings on both ends and reboot once to get better throughput: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] "SackOpts"=dword:0x1 (enable SACK) "TcpWindowSize"=dword:0x7D000 (512000 Bytes) "Tcp1323Opts"=dword:0x3 (enable window scaling and timestamps) "GlobalMaxTcpWindowSize"=dword:0x7D000 (512000 Bytes) http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx -- Andre Marshall On Mar 27, 2007, at 4:26 PM, Philip Lavine wrote: To all, I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? Philip __ __ Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091
Re: TCP and WAN issue
On 3/27/2007 8:44 PM, Leo Bicknell wrote: > Jumbo frames are not [required], and in fact make little difference. They are actually worse in some applications, since they have such a high propagation delay. Instead of streaming 6 segments spread across a 300 usec time window, you send a single segment that still requires 300 usec... not always with happy results. -- Eric A. Hallhttp://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/
Re: TCP and WAN issue
In a message written on Tue, Mar 27, 2007 at 02:41:13PM -0700, Philip Lavine wrote: > This is the exact issue. I can only get between 5-7 Mbps. So the > question is really what incremental performance gain do WAN > accelerators/optimizers offer? Can registry/OS tweaks really make > a significant difference because so far with all the "speed > enhancements" I have deployed to the registry based on the some of > the aforementioned sites I have seen no improvement. In short, yes. Working for an ISP with colos on both coasts I helped customers on numerous occasions "tune" their operating system. A plain jane desktop from choose your favorite vendor today will do ~300-~500Mbps/sec of FTP with only minor tuning assuming you have good disks (e.g. not a laptop). Most servers can be easily tuned to chew a full gigabit. TCP Window Scale, Selective Acknowledgement, and a "TCP Send" and "TCP Receive" buffers that are big enough to handle your bandwidth*delay product (*2 for a good margin) are all that's required. Jumbo frames are not, and in fact make little difference. All of the TCP limits are 100% the same with Jumbo frames, and most NIC's generate roughly the same number of Interrupts with Jumbo frames enabled. You're saving a little bit of IP processing overhead on the end hosts, but I doubt you'll measure it. Wan optimizers are extremely complex points of failure that generally mess with the protocol in ways that are extremely dangerous, they should only be used if you don't have direct access to the end boxes to fix them. Google can turn up 50 write ups on how to tune your settings better than I could ever write here. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org pgpkpyd5jibVi.pgp Description: PGP signature
Re: TCP and WAN issue
On 3/27/07, Lincoln Dale <[EMAIL PROTECTED]> wrote: even on "default settings" on a modern TCP stack, getting close to path-line-rate on a 80msec RTT WAN @ DS3 speeds with a single TCP stream should not be that difficult. the Windows TCP stack as of Windows XP SP2 has some fairly decent defaults. it will do RFC1323 / large windows / SACK., but all of these can be tuned with registry settings if you wish. I was under the impression that XP's default window size was 17,520 bytes, rfc1323 options disabled. Assuming 80ms and 45Mb/s, I come up with a window size of 440Kbytes required to fill the pipe. At windows default I would only expect to see 220Kbs over that same path. I think even modern *nix OSs tend to have default window sizes in the 64kB region, still not enough for that Bandwidth/delay. -- -Steve
Re: TCP and WAN issue
You might want to look at this classic by Stanislav Shalunov http://shlang.com/writing/tcp-perf.html Marshall On Mar 27, 2007, at 4:26 PM, Philip Lavine wrote: To all, I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? Philip __ __ Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091
RE: TCP and WAN issue
> I have an east coast and west coast data center connected with a DS3. I am > running into issues with streaming data via TCP and was wondering besides > hardware acceleration, is there any options at increasing throughput and > maximizing the bandwidth? How can I overcome the TCP stack limitations > inherent in Windows (registry tweaks seem to not functions too well)? even on "default settings" on a modern TCP stack, getting close to path-line-rate on a 80msec RTT WAN @ DS3 speeds with a single TCP stream should not be that difficult. the Windows TCP stack as of Windows XP SP2 has some fairly decent defaults. it will do RFC1323 / large windows / SACK., but all of these can be tuned with registry settings if you wish. with a particular implementation of FCIP (Fibre Channel over IP) i worked on, we could pretty much sustain a single TCP stream from a single GbE port at wire-rate GbE with RTT up to 280msec with minimal enhancements to TCP. at that point it started to get difficult because you had close to 32MB of data in transit around at any given time, which is the current standard limit for how large you can grow a TCP window. i think the first thing you should do is ascertain there are no problems with your LAN or WAN. i.e. that there are no drops being recorded, no duplex mismatch anywhere, etc. i suggest you fire up "ttcp" on a host on each end and see what throughput you get. with both tcp & udp you should be able to get close to 5.5 to 5.6 MB/s. if you can't, i'd suggest looking into why & addressing the root cause. once you've done that, its then a case of ensuring the _applications_ you're using can actually "fill the pipe" and aren't latency-sensitive at that distance.
Re: TCP and WAN issue
On Tue, Mar 27, 2007, Philip Lavine wrote: > > This is the exact issue. I can only get between 5-7 Mbps. So the question is > really what incremental performance gain do WAN accelerators/optimizers > offer? Can registry/OS tweaks really make a significant difference because so > far with all the "speed enhancements" I have deployed to the registry based > on the some of the aforementioned sites I have seen no improvement. > > I guess I am looking for a product that as a wrapper can multiplex a single > socket connection. The "WAN optimisers" (yuk, I do have to deal with them in my current work) offer some kind of TCP acceleration and flow control - the most advanced I've seen here is Packeteer. What they (claim to) do is a few things, notably: * spoof the TCP connection at both ends over high-latency links so the TCP stacks at either end think they're talking to something with low RTT/loss and thus open up their window sizes right up w/ less touchy expoential backoff; * then massage the TCP delay and window sizes to make things play happy inside the given pipe size. * Then insert all kinds of protocol spoofing to deal with WAN link speed/RTT to fix performance of "LAN only" protocols (such as pretending to be SMB end-to-end I believe) - this isn't a new thing btw, I seem to remember seeing routers in the 90's spoof various bits of IPX/SPX over ISDN to get a speedup and prevent spurious re-dial just to see if the other end is there.. Now, this stuff seems to work fine in the situations I'm using them for (low bandwidth links, lots of users end-to-end, TCP normally chokes rather easily) but they do allow things like windows boxes to get full speed out of satellite links. This, of course, makes 0% difference to the throughput of my well-tuned Linux Squid web proxies at either end; but it does mean the proxies don't end up using -all- of the bandwidth of the link as they want to do. So! * if you're after "fill this pipe please" and you don't require the super-fine application throttling/shaping/acceleration, then do whatever QoS you require on your DS3 (if any!) to ensure you're not causing your high-priority traffic to get tail-dropped in the "dumb" fashion and causing backoff, and * you don't need jumbo frames to get full DS3 speed over 70ms; just bump up tx/rx buffer sizes and maximum window sizes and re-test. * realise that the WAN accelerators make their money doing TCP optimisation because its easier to buy a box and put it in then push out group policies to windows desktops to fix their default TCP settings. :) * and finally, make sure the rest of your network isn't introducing issues - stuff like bung cables/drivers/duplex settings can cause packetloss which becomes much nastier over higher latency links (backoff/retry over LAN happens much faster than your 70ms cross-country link.) Adrian
Re: TCP and WAN issue
On Mar 27, 2007, at 3:24 PM, <[EMAIL PROTECTED]> wrote: Personally, I would prefer to see more people fixing the infrastructure rather than accepting it as a limit. Concur - what I meant is, 'can support when fully optimized'. ;> Tweaking apps generally turns out to be heavy-duty stuff with lots of release control and testing. Also, the applications programmers generally have a poor understanding of network issues. If you can separate the applications stuff from the data transfer stuff, and tackle the network issues first, then you will have an easier time of it. Concur - app-tweaking should be the penultimate approach (and then, maybe, look at boxes, if there's an issue which can't be resolved the other ways; but my guess is that doing the BCPs should yield good results). --- Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice Words that come from a machine have no soul. -- Duong Van Ngo
Re: TCP and WAN issue
On Tue, 27 Mar 2007, Philip Lavine wrote: I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? You should talk to the vendor (microsoft) and ask them how to tweak their product to properly work over the WAN. Don't let them get away with substandard product when it comes to WAN optimization. If you can get microsoft to clean up their act, you'd have done ISPs a great service, because then we can stop trying to convince customers that it's not ISP fault that they get bad speeds with their windows PCs. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: TCP and WAN issue
Philip Lavine wrote: I have 2 data transmission scenarios: 1. Microsoft MSMQ data using TCP 2. "Streaming" market data stock quotes transmitted via a TCP sockets Philip TCP stack tuning works very well for applications with large sized network reads and writes. Applications that will only write X and then will wait to read Y do not gain from tweaks when Y or X are not large enough to fill a significant portion of the pipe end to end. IOW, the chattiness of the protocol multiplied by the rtt equals delay. Applications that open and close sockets to chat are even worse. The only real thing that can cure this other than a rewrite of the applications behavior, is a application layer proxy that "chats on behalf of" This is part of the wan optimization space from what I understand. - Original Message From: William F. Maton Sotomayor <[EMAIL PROTECTED]> To: Philip Lavine <[EMAIL PROTECTED]> Sent: Tuesday, March 27, 2007 2:50:44 PM Subject: Re: TCP and WAN issue Hi Philip, We've run across this problem from one end of Canada to the other. As others have alluded to, the best you can hope for is to tweak the TCP window sizes up and increase the socket buffers to suit. However, if you're doing something over NetBIOS, it's turned out to be a total lost cause. Right now we're looking at the Steelheads from Riverbed to solve the NetBIOS problem. (The Steelheads do the dirty work of TCP tweaking on your behalf, so you don't have to, besides cacheing.) BTW, when you say stream, you're saying transfer of data other than a webcast, correct? On Tue, 27 Mar 2007, Philip Lavine wrote: This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement. I guess I am looking for a product that as a wrapper can multiplex a single socket connection. Philip - Original Message From: Robert Boyle <[EMAIL PROTECTED]> To: Philip Lavine <[EMAIL PROTECTED]>; nanog Sent: Tuesday, March 27, 2007 2:04:20 PM Subject: Re: TCP and WAN issue At 04:26 PM 3/27/2007, Philip Lavine wrote: I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? You will have problems obtaining anything more than 5-7Mbit/s based on 1500 byte Ethernet packets and a RTT latency of 70-90ms. You can increase your window size or use Jumbo Ethernet frames. Almost all GigE gear supports jumbo frames. I'm not sure of your application, but without OS tweaks, each stream is limited to 5-7Mbit/s. You can open multiple streams between the same two hosts or you can use multiple hosts to transfer your data. You can utilize the entire DS3, but not without OS TCP stack tweaks or a move to jumbo frames. You can also use UDP or another connectionless packet method to move the data between sites. Good luck. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097 wfms 8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news
RE: TCP and WAN issue
> What you want to see is large packets, as large as your end-to-end > infrastructure can support. Personally, I would prefer to see more people fixing the infrastructure rather than accepting it as a limit. Install some Linux servers even if all they do is run an application layer proxy to turn the TCP stream from the LAN into something better suited for the WAN. They could use UDP, compress the stream, consolidate small MTU packets into big MTU packets, etc. I remember many years ago when a dentist asked me to help with an urgent problem on his billing system. The system worked fine but he split from his wife and had changed the corporate name of his dental practice. He needed to produce bills under the new name, but the software provider had gone out of business a couple of years earlier and there was no source code. Fortunately, it was a UNIX based application that used the print spooler for billing reports so I just inserted a shell script to change "Pleasant Valley Dental Inc." to "Dr. Smith Dental (1987) Inc." or some such. Fixing the infrastructure may also mean switching IP network providers to get someone who supports bulk data transfers between data centers with jumbo frames end-to-end. > Your application should be able to do that, potentially, and > as other > folks mentioned, your app can potentially be tweaked to open up > multiple connections. I think there are also NICs which do > something > of this sort, but it's not something I've personally used (maybe > others have experiences they can relate?). Tweaking apps generally turns out to be heavy-duty stuff with lots of release control and testing. Also, the applications programmers generally have a poor understanding of network issues. If you can separate the applications stuff from the data transfer stuff, and tackle the network issues first, then you will have an easier time of it. --Michael Dillon
RE: TCP and WAN issue
> I have an east coast and west coast data center connected > with a DS3. I am running into issues with streaming data via > TCP and was wondering besides hardware acceleration, is there > any options at increasing throughput and maximizing the > bandwidth? Use GigE cards on the servers with a jumbo MTU and only buy IP network access from a service provider who supports jumbo MTUs end-to-end through their network. > How can I overcome the TCP stack limitations > inherent in Windows (registry tweaks seem to not functions too well)? Install a pair of Linux servers and use them to send/receive the data over the WAN. Also, do some googling for Internet Speed Record and read pages like this one: http://www.internet2.edu/lsr/history.html And read up on scaling MTI size with bandwidth here: http://www.psc.edu/~mathis/MTU/arguments.html --Michael Dillon
Re: TCP and WAN issue
On Mar 27, 2007, at 2:41 PM, Philip Lavine wrote: This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? I don't know if you'd get much of a performance benefit from this approach. Bandwidth savings, possibly, depending upon your application. We have a box called a WAAS which is a WAN optimizer, so do several other vendors (search online for 'wan optimizer' or 'wan optimization', you should get a lot of hits), but I have no experience with these types of boxes. Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement. I'm not a Windows person, so I don't know the answer to this; I know you can do a fair amount of optimization with other OSes, depending upon the OS and your NICs. The MTU, MSS, window-size stuff mentioned previously all applies, as do jumbo frames, if your end-stations and network infrastructure support them. What you want to see is large packets, as large as your end-to-end infrastructure can support. I guess I am looking for a product that as a wrapper can multiplex a single socket connection. Your application should be able to do that, potentially, and as other folks mentioned, your app can potentially be tweaked to open up multiple connections. I think there are also NICs which do something of this sort, but it's not something I've personally used (maybe others have experiences they can relate?). My general advice would be to look at all the things mentioned previously you can potentially do with your existing OSes, network infrastructure, and apps, and do them prior to looking at specialized boxes. --- Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice Words that come from a machine have no soul. -- Duong Van Ngo
Re: TCP and WAN issue
I have 2 data transmission scenarios: 1. Microsoft MSMQ data using TCP 2. "Streaming" market data stock quotes transmitted via a TCP sockets Philip - Original Message From: William F. Maton Sotomayor <[EMAIL PROTECTED]> To: Philip Lavine <[EMAIL PROTECTED]> Sent: Tuesday, March 27, 2007 2:50:44 PM Subject: Re: TCP and WAN issue Hi Philip, We've run across this problem from one end of Canada to the other. As others have alluded to, the best you can hope for is to tweak the TCP window sizes up and increase the socket buffers to suit. However, if you're doing something over NetBIOS, it's turned out to be a total lost cause. Right now we're looking at the Steelheads from Riverbed to solve the NetBIOS problem. (The Steelheads do the dirty work of TCP tweaking on your behalf, so you don't have to, besides cacheing.) BTW, when you say stream, you're saying transfer of data other than a webcast, correct? On Tue, 27 Mar 2007, Philip Lavine wrote: > > This is the exact issue. I can only get between 5-7 Mbps. So the question is > really what incremental performance gain do WAN accelerators/optimizers > offer? Can registry/OS tweaks really make a significant difference because so > far with all the "speed enhancements" I have deployed to the registry based > on the some of the aforementioned sites I have seen no improvement. > > I guess I am looking for a product that as a wrapper can multiplex a single > socket connection. > > Philip > > - Original Message > From: Robert Boyle <[EMAIL PROTECTED]> > To: Philip Lavine <[EMAIL PROTECTED]>; nanog > Sent: Tuesday, March 27, 2007 2:04:20 PM > Subject: Re: TCP and WAN issue > > At 04:26 PM 3/27/2007, Philip Lavine wrote: >> I have an east coast and west coast data center connected with a >> DS3. I am running into issues with streaming data via TCP and was >> wondering besides hardware acceleration, is there any options at >> increasing throughput and maximizing the bandwidth? How can I >> overcome the TCP stack limitations inherent in Windows (registry >> tweaks seem to not functions too well)? > > You will have problems obtaining anything more than 5-7Mbit/s based > on 1500 byte Ethernet packets and a RTT latency of 70-90ms. You can > increase your window size or use Jumbo Ethernet frames. Almost all > GigE gear supports jumbo frames. I'm not sure of your application, > but without OS tweaks, each stream is limited to 5-7Mbit/s. You can > open multiple streams between the same two hosts or you can use > multiple hosts to transfer your data. You can utilize the entire DS3, > but not without OS TCP stack tweaks or a move to jumbo frames. You > can also use UDP or another connectionless packet method to move the > data between sites. Good luck. > > -Robert > > > > Tellurian Networks - Global Hosting Solutions Since 1995 > http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 > "Well done is better than well said." - Benjamin Franklin > > > > > > > > > Finding fabulous fares is fun. > Let Yahoo! FareChase search your favorite travel sites to find flight and > hotel bargains. > http://farechase.yahoo.com/promo-generic-14795097 > wfms 8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news
Re: TCP and WAN issue
This is the exact issue. I can only get between 5-7 Mbps. So the question is really what incremental performance gain do WAN accelerators/optimizers offer? Can registry/OS tweaks really make a significant difference because so far with all the "speed enhancements" I have deployed to the registry based on the some of the aforementioned sites I have seen no improvement. I guess I am looking for a product that as a wrapper can multiplex a single socket connection. Philip - Original Message From: Robert Boyle <[EMAIL PROTECTED]> To: Philip Lavine <[EMAIL PROTECTED]>; nanog Sent: Tuesday, March 27, 2007 2:04:20 PM Subject: Re: TCP and WAN issue At 04:26 PM 3/27/2007, Philip Lavine wrote: >I have an east coast and west coast data center connected with a >DS3. I am running into issues with streaming data via TCP and was >wondering besides hardware acceleration, is there any options at >increasing throughput and maximizing the bandwidth? How can I >overcome the TCP stack limitations inherent in Windows (registry >tweaks seem to not functions too well)? You will have problems obtaining anything more than 5-7Mbit/s based on 1500 byte Ethernet packets and a RTT latency of 70-90ms. You can increase your window size or use Jumbo Ethernet frames. Almost all GigE gear supports jumbo frames. I'm not sure of your application, but without OS tweaks, each stream is limited to 5-7Mbit/s. You can open multiple streams between the same two hosts or you can use multiple hosts to transfer your data. You can utilize the entire DS3, but not without OS TCP stack tweaks or a move to jumbo frames. You can also use UDP or another connectionless packet method to move the data between sites. Good luck. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097
Re: TCP and WAN issue
Philip, > I have an east coast and west coast data center connected with a DS3. I > am running into issues with streaming data via TCP and was wondering > besides hardware acceleration, is there any options at increasing > throughput and maximizing the bandwidth? How can I overcome the TCP > stack limitations inherent in Windows (registry tweaks seem to not > functions too well)? I don't know the RTT, but you should have at least 300 kByte buffers on the end hosts for a 60 ms RTT path to reach 40 Mbps TCP throughput. (This requires window scaling as well.) Is this what you were trying to tune on your Windows hosts? Is your DS3 free of errors? Even a very low packet loss can degrade TCP performance badly. You'll find a lot of useful information about TCP performance in the GEANT2 PERT Knowledge Base at http://www.kb.pert.geant2.net/ Andras
Re: TCP and WAN issue
At 04:26 PM 3/27/2007, Philip Lavine wrote: I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? You will have problems obtaining anything more than 5-7Mbit/s based on 1500 byte Ethernet packets and a RTT latency of 70-90ms. You can increase your window size or use Jumbo Ethernet frames. Almost all GigE gear supports jumbo frames. I'm not sure of your application, but without OS tweaks, each stream is limited to 5-7Mbit/s. You can open multiple streams between the same two hosts or you can use multiple hosts to transfer your data. You can utilize the entire DS3, but not without OS TCP stack tweaks or a move to jumbo frames. You can also use UDP or another connectionless packet method to move the data between sites. Good luck. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
Re: TCP and WAN issue
On 27-Mar-2007, at 16:35, Joe Abley wrote: You might take a look through RFC 2488/BCP 28, if you haven't already. The circuit propagation delays in that scenarios painted by that document are far higher than yours, but the principles are the same. "... in *the* scenarios..." I am having trouble with words, today.
Re: TCP and WAN issue
On Mar 27, 2007, at 1:26 PM, Philip Lavine wrote: inherent in Windows (registry tweaks seem to not functions too well)? You should certainly look at your MTU and MSS values to ensure there are no difficulties of that sort. Is there any other factor such as perhaps slow DNS server (or other lookup-type services) responses which can be contributing to the perceived slowness? How about tuning the I/O buffers on the relevant routers? Can you tune the I/O buffers on the servers? And what about your link utilization? Is the DS3 sufficient? Take a look at pps and bps, and take a look at your average packet sizes (NetFlow can help with this). Are your apps sending lots of smaller packets, or are you getting nice, large packet-sizes? Finally, if none of the above help, you could look at something like SCTP, if your servers and apps support it. But I'd go through the above exercise, first. --- Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice Words that come from a machine have no soul. -- Duong Van Ngo
Re: TCP and WAN issue
On 27-Mar-2007, at 16:26, Philip Lavine wrote: I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? You might take a look through RFC 2488/BCP 28, if you haven't already. The circuit propagation delays in that scenarios painted by that document are far higher than yours, but the principles are the same. Joe
TCP and WAN issue
To all, I have an east coast and west coast data center connected with a DS3. I am running into issues with streaming data via TCP and was wondering besides hardware acceleration, is there any options at increasing throughput and maximizing the bandwidth? How can I overcome the TCP stack limitations inherent in Windows (registry tweaks seem to not functions too well)? Philip Need Mail bonding? Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users. http://answers.yahoo.com/dir/?link=list&sid=396546091
