Re: Baffling problem with OBSD-protected servers and Windows Vista...
On 2006/11/28 18:07, Michael Lockhart wrote: > Set net.inet.tcp.rfc1323=0 in /etc/sysctl.conf and that should resolve > the issue. that's not a fix though, it just avoids the conditions which cause the problem to occur. better to ensure the ruleset is completely sane. if so, then test cases need to be found to isolate the problem. if anyone wants an example of wonderful source code commenting, the pf_norm.c sections relating to rfc1323 are particularly good.
Re: Baffling problem with OBSD-protected servers and Windows Vista...
Set net.inet.tcp.rfc1323=0 in /etc/sysctl.conf and that should resolve
the issue. We've been testing in house with OpenBSD for Vista (we have
700+/- systems in the field) and this seems to resolve the issue.
Regards,
Mike Lockhart
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Mike Lockhart[Systems Engineering & Operations]
StayOnline, Inc
http://www.stayonline.net/
mailto: [EMAIL PROTECTED]
GPG: 8714 6F73 3FC8 E0A4 0663 3AFF 9F5C 888D 0767 1550
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf
Of Stuart Henderson
Sent: Tuesday, November 28, 2006 5:46 PM
To: Reverend Deuce
Cc: misc@openbsd.org
Subject: Re: Baffling problem with OBSD-protected servers and Windows
Vista...
On 2006/11/28 14:32, Reverend Deuce wrote:
> Okay guys, I posted that long message about Firefox/etc on Windows
> Vista a couple of days ago.
this would be easier if you just posted pf.conf rather than non-linear
snippets; however..
> a) there is a default block policy
I didn't notice you posting anything showing a default block for
outgoing
packets, check this and if not, add one.
> block in log from any to any
label "DefaultBlock"
> block in log on { $ext_if } all label
"DefaultBlock"
> block return-rst in log on { $ext_if } proto tcp all label
"DefaultBlock"
> block return-icmp in log on { $ext_if } proto udp all label
"DefaultBlock"
fwiw, you can simplify these if you like:
'block return in log on { $ext_if } label "DefaultBlock"'
> I have heard it said that it makes no sense to filter on two
> interfaces, best to pass on one and block on the other.
that advice is usually given in relation to filtering bridges.
Re: Baffling problem with OBSD-protected servers and Windows Vista...
On 2006/11/28 14:32, Reverend Deuce wrote:
> Okay guys, I posted that long message about Firefox/etc on Windows
> Vista a couple of days ago.
this would be easier if you just posted pf.conf rather than non-linear
snippets; however..
> a) there is a default block policy
I didn't notice you posting anything showing a default block for outgoing
packets, check this and if not, add one.
> block in log from any to any label
> "DefaultBlock"
> block in log on { $ext_if } all label "DefaultBlock"
> block return-rst in log on { $ext_if } proto tcp all label "DefaultBlock"
> block return-icmp in log on { $ext_if } proto udp all label "DefaultBlock"
fwiw, you can simplify these if you like:
'block return in log on { $ext_if } label "DefaultBlock"'
> I have heard it said that it makes no sense to filter on two
> interfaces, best to pass on one and block on the other.
that advice is usually given in relation to filtering bridges.
Re: Baffling problem with OBSD-protected servers and Windows Vista...
Okay guys, I posted that long message about Firefox/etc on Windows Vista a couple of days ago. After I re-read my post and looked at the tcpdump output, and chatting with a friend of mine who also runs several OBSD firewalls at his company which exhibited the same EXACT problem when my Vista installs attempted to connect... I think we've figured it out. I remember that Vista (and presumably Longhorn Server) have a completely re-written TCP stack by Microsoft. They've put in all kinds of new stuff. One of which is Receive Window Auto-Tuning. I noticed in my tcpdump the following lines: From Opera/Firefox: 20:40:45.824144 my.workstation.ip.49370 > remote.server.ip.80: S 1215871830:1215871830(0) win 8192 (DF) 20:38:25.198320 remote.server.ip.80 > my.workstation.ip.49357: S 852828096:852828096(0) ack 643900712 win 64240 From IE 7: 20:39:08.834465 my.workstation.ip.49358 > remote.server.ip.80: S 4155969795:4155969795(0) win 8192 (DF) 20:39:08.835095 remote.server.ip.80 > my.workstation.ip.49358: S 3294485308:3294485308(0) ack 4155969796 win 64240 Notice the window scale is 2 for IE, and 8 for Firefox/Opera. From the following MS blog @ http://www.microsoft.com/technet/community/columns/cableguy/cg1105.mspx : Note: Some Internet gateway devices and firewalls block packet flows because they do not correctly interpret the scaling factor used in TCP connections. Because of this, Internet Explorer in Windows Vista uses an initial scaling factor of 2. Other applications use a default initial scaling factor of 8. After doing the new Vista-equivilent of sudo and elevating my command shell to Administrator mode, I was able to use the following command to disable window scaling completely: C:\windows\system32> netsh interface tcp set global autotuninglevel=disabled Once I performed this command, Firefox/Opera/Remote Desktop Connection all function once more as expected. Now, as I am clearly looking at the window scale issue here, I had found a thread at http://archive.openbsd.nu/?ml=openbsd-pf&a=2006-07&t=2147873 where Daniel Hartmeier comments there are three things that need to be done to have state created correctly: a) there is a default block policy b) all 'pass' rules that can match TCP have 'flags S/SA' c) all 'pass' rules have 'keep state' Here is what I am seeing inside PF with the connections in question: Nov 26 23:09:36.970856 rule 80/(match) pass in on fxp1: my.workstation.ip.59970 > remote.server.ip.443: [|tcp] (DF) Then if I pull the 80th rule out: @80 pass in log quick on fxp1 inet proto tcp from any to remote.server.ip port = https flags S/SA keep state label "ExchangeIn" Now, I can easily see that I am matching B and C of Daniel's list, however A is a bit more in question from my point of view. The rules I do have are: @47 block drop in log on fxp1 all label "DefaultBlock" @48 block return-rst in log on fxp1 proto tcp all label "DefaultBlock" @49 block return-icmp(port-unr, port-unr) in log on fxp1 proto udp all label "DefaultBlock" So, it appears I have condition A matched as well. I do have a line regarding: @95 pass in quick on fxp0 inet proto tcp from to any flags S/SA keep state label "lanOUT" That should not come into play here at all, as again it is creating state on a Syn, not a Syn Ack. However, after testing on this system, I am thinking I am filtering wrong here. Here is what I have found as the full story of what is going on: Connection is open: Nov 27 12:09:07.978281 rule 80/(match) pass in on fxp1: my.workstation.ip.62658 > remote.server.ip.443: [|tcp] (DF) Two state entries are created: all tcp remote.server.ip:443 <- remote.server.ip:443 <- my.workstation.ip:62658 ESTABLISHED:ESTABLISHED [4265902579 + 65535] [1356591875 + 65535] age 00:01:08, expires in 119:59:09, 12:9 pkts, 2014:5401 bytes, rule 80 id: 453e890500b00c1e creatorid: 19ad04b2 all tcp my.workstation.ip:62658 <- remote.server.ip:443 ESTABLISHED:ESTABLISHED [1356591875 + 65535] [4265902579 + 65535] age 00:01:08, expires in 119:59:09, 9:11 pkts, 5401:1966 bytes, rule 96 id: 453e890500b00c1f creatorid: 19ad04b2 Rules that match the state entries: @80 pass in log quick on fxp1 inet proto tcp from any to remote.server.ip port = https flags S/SA keep state label "ExchangeIn" [ Evaluations: 5000 Packets: 204 Bytes: 50906 States: 5 ] [ Inserted: uid 0 pid 806 ] @95 pass in quick on fxp0 inet proto tcp from to any flags S/SA keep state label "lanOUT" [ Evaluations: 417330Packets: 60770372 Bytes: 36986041704 States: 771 ] [ Inserted: uid 0 pid 6307 ] @96 pass in quick on fxp0 from to any keep state label "lanOUT" [ Evaluations: 429312Packets: 9560597 Bytes: 5957780712 States: 135 ] [ Inserted: uid 0 pid 6307 ] Now, it is looking to me like the issue is a second state entry is created by that rule 96. When it is modified to be only protocol UDP, traffic through the FW stops due to the rules: block in log
Re: Baffling problem with OBSD-protected servers and Windows Vista...
[2006-11-27 10:43] Claudio Jeker <[EMAIL PROTECTED]> wrote: > Both Firefox and Opera use a wscale of 8 whereas IE uses a wscale of 2. > In my opinion this sounds like the typical problem where states are not > created on the initial SYN packet. sounds like a window scaling problem as described in: http://inodes.org/blog/2006/09/06/tcp-window-scaling-and-kernel-2617/ there have been some very opinionated mail on LKML about it (they had the same problem mit linux kernel 2.6.17+) (about striping the scaling factor .. .. or ignoring it when calculation windows) though i am not able to find it now christian bahls -- personal email reaches me at gmx.de [EMAIL PROTECTED]
Re: Baffling problem with OBSD-protected servers and Windows Vista...
Hi there I had the exact same strange (kind of) problem. All clients could connect to my (own OpenBSD) web server, only my main PC (sorry linux gentoo machine) could not. The packets match what you show below. It stops because the initial http) packet does't arrive at your VistaPC. [Fire up wireshark and load the tcpdump into that -> TCP previous segment lost, it told me] After 2 days and nights shuffling around all Hardware and software (Firewall, PC, Ethernetcards, Gateway, kernels, ...) this was the problem: On the OpenBSD www server (pf enabled), the one rule had no 'keep state' not good: pass in quick on $external inet proto tcp from any to $http_server port = $http_port label "l_http_in" good: pass in quick on $external inet proto tcp from any to $http_srv port = http keep statelabel "l_http_in" # queue q_rest_out pass out quick on $external inet proto tcp from any to anyport = http keep state (I added the pass out rule, not sure if needed) Why on earth this only produced problems with 1/100 of all clients, I cannot say... (It should always fail or never...) For example with linux kernel 2.6.16.x it worked, with kernel 2.6.17.x it didn't. Some IP/networking behavior must have been changed in defaults ...?) Hope that helps -cmb [ long part of email removed] > > > Opera 9: > > 20:40:45.824144 my.workstation.ip.49370 > remote.server.ip.80: S > 1215871830:1215871830(0) win 8192 8,nop,nop,sackOK> (DF) > 20:40:45.824646 207.218.64.33.80 > my.workstation.ip.49370: S > 2582857930:2582857930(0) ack 1215871831 win 64240 0,nop,nop,sackOK> > 20:40:45.878361 my.workstation.ip.49370 > 207.218.64.33.80: . ack 1 win > 260 (DF) > 20:40:45.904597 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.058234 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > 20:40:46.061253 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.061726 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > (at this point, the connection is hung -- the Vista workstation > receives no further communcations -- it's like it just drops the > replies) > > > > Firefox: > > 20:38:25.197691 my.workstation.ip.49357 > remote.server.ip.80: S > 643900711:643900711(0) win 8192 > (DF) > 20:38:25.198320 remote.server.ip.80 > my.workstation.ip.49357: S > 852828096:852828096(0) ack 643900712 win 64240 0,nop,nop,sackOK> > 20:38:25.244540 my.workstation.ip.49357 > remote.server.ip.80: . ack 1 > win 260 (DF) > 20:38:25.251037 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.567602 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.568042 remote.server.ip.80 > my.workstation.ip.49357: . ack > 403 win 63838 (DF) > (same deal -- it just seems to die right here) > [ removed ... ]
Re: Baffling problem with OBSD-protected servers and Windows Vista...
On Sun, Nov 26, 2006 at 09:19:25PM -0600, Reverend Deuce wrote: > (This is very long email because it's a very complicated problem... > I've included some tcpdump logs below to assist...) > > Here are some tcpdumps from the master FW during connection attempts > with a browser: > > > > Opera 9: > > 20:40:45.824144 my.workstation.ip.49370 > remote.server.ip.80: S > 1215871830:1215871830(0) win 8192 8,nop,nop,sackOK> (DF) > 20:40:45.824646 207.218.64.33.80 > my.workstation.ip.49370: S > 2582857930:2582857930(0) ack 1215871831 win 64240 0,nop,nop,sackOK> > 20:40:45.878361 my.workstation.ip.49370 > 207.218.64.33.80: . ack 1 win 260 > (DF) > 20:40:45.904597 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.058234 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > 20:40:46.061253 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.061726 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > (at this point, the connection is hung -- the Vista workstation > receives no further communcations -- it's like it just drops the > replies) > > > > Firefox: > > 20:38:25.197691 my.workstation.ip.49357 > remote.server.ip.80: S > 643900711:643900711(0) win 8192 > (DF) > 20:38:25.198320 remote.server.ip.80 > my.workstation.ip.49357: S > 852828096:852828096(0) ack 643900712 win 64240 0,nop,nop,sackOK> > 20:38:25.244540 my.workstation.ip.49357 > remote.server.ip.80: . ack 1 > win 260 (DF) > 20:38:25.251037 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.567602 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.568042 remote.server.ip.80 > my.workstation.ip.49357: . ack > 403 win 63838 (DF) > (same deal -- it just seems to die right here) > > > > IE 7: > > 20:39:08.834465 my.workstation.ip.49358 > remote.server.ip.80: S > 4155969795:4155969795(0) win 8192 2,nop,nop,sackOK> (DF) > 20:39:08.835095 remote.server.ip.80 > my.workstation.ip.49358: S > 3294485308:3294485308(0) ack 4155969796 win 64240 0,nop,nop,sackOK> > 20:39:08.892057 my.workstation.ip.49358 > remote.server.ip.80: . ack 1 > win 16685 (DF) > 20:39:08.904548 my.workstation.ip.49358 > remote.server.ip.80: P > 1:472(471) ack 1 win 16685 (DF) > 20:39:08.907010 remote.server.ip.80 > my.workstation.ip.49358: . > 1:1381(1380) ack 472 win 63769 (DF) > 20:39:08.907135 remote.server.ip.80 > my.workstation.ip.49358: . > 1381:2761(1380) ack 472 win 63769 (DF) > 20:39:08.959016 my.workstation.ip.49358 > remote.server.ip.80: . ack > 2761 win 16685 (DF) > 20:39:08.959740 remote.server.ip.80 > my.workstation.ip.49358: . > 2761:4141(1380) ack 472 win 63769 (DF) > 20:39:08.959750 remote.server.ip.80 > my.workstation.ip.49358: . > 4141:5521(1380) ack 472 win 63769 (DF) > 20:39:08.959911 remote.server.ip.80 > my.workstation.ip.49358: . > 5521:6901(1380) ack 472 win 63769 (DF) > 20:39:09.010614 my.workstation.ip.49358 > remote.server.ip.80: . ack > 6901 win 16685 (DF) > 20:39:09.011323 remote.server.ip.80 > my.workstation.ip.49358: . > 6901:8281(1380) ack 472 win 63769 (DF) > 20:39:09.011333 remote.server.ip.80 > my.workstation.ip.49358: . > 8281:9661(1380) ack 472 win 63769 (DF) > 20:39:09.011447 remote.server.ip.80 > my.workstation.ip.49358: . > 9661:11041(1380) ack 472 win 63769 (DF) > 20:39:09.011571 remote.server.ip.80 > my.workstation.ip.49358: . > 11041:12421(1380) ack 472 win 63769 (DF) > 20:39:09.058459 my.workstation.ip.49358 > remote.server.ip.80: . ack > 9661 win 16685 (DF) > 20:39:09.059165 remote.server.ip.80 > my.workstation.ip.49358: . > 12421:13801(1380) ack 472 win 63769 (DF) > 20:39:09.059289 remote.server.ip.80 > my.workstation.ip.49358: P > 13801:15131(1330) ack 472 win 63769 (DF) > 20:39:09.064831 my.workstation.ip.49358 > remote.server.ip.80: . ack > 12421 win 16685 (DF) > 20:39:09.097561 my.workstation.ip.49359 > remote.server.ip.80: S > 3924198291:3924198291(0) win 8192 2,nop,nop,sackOK> (DF) > 20:39:09.098564 remote.server.ip.80 > my.workstation.ip.49359: S > 4022470982:4022470982(0) ack 3924198292 win 64240 0,nop,nop,sackOK> > (But IE 7 is just dandy! It loads the whole page and just keeps on > tickin, goddamnit!) > > > I should note that the only connections that seem to have trouble are > TCP connections. It seems that OpenVPN clients can traverse the > firewall without issue (our OpenVPN server sites behind the firewall). > Inbound DNS queries have no issues either. ICMP is good also, for the > hosts that have permission to ping. > > Again, I need to stress this -- I've NOT overlooked the obvious here. > Remember, the problem **only** happens in Vista. It has happened on > several desktops/workstations, both off-site from our office, inside a > VM, directly on hardware, etc. I've tried about a dozen configs so > far. This is not a "try swapping the RAM" situation. > > There are no strange VPNs, static routes, oddball topologies, etc
Re: Baffling problem with OBSD-protected servers and Windows Vista...
I'm not sure if this will be of any help, but at least the Firefox issue sounds like FF is able to connect, but never receives any return traffic. I've had that with misconfigured netmasks I believe. Does Vista use some sort of net group or certificate based access scheme (e.g. "if it's not a Vista box talking to me, I won't talk back")? May sound stupid, but you never know. Who on earth knows what MS does with network traffic? Bill On Sun, 2006-11-26 at 21:19 -0600, Reverend Deuce wrote: > (This is very long email because it's a very complicated problem... > I've included some tcpdump logs below to assist...) > > The last week and days I've been working with the RTM version of Vista > obtained through my MSDN license. This is the "gold" version of > Windows Vista, BTW. It's done. It's been shipped to manufacturing > (hence RTM, "release to manufacturing"). > > Okay, so I've installed this thing and am testing out all the bells > and whistles. > > I install Firefox, OpenVPN, putty, the Java JRE from Sun, etc. > > I start to tool around and I notice that none of our company's web > sites will load in Firefox any longer. Firefox's status bar says > "Waiting for www.site.tld..." and eventually will time out. It does > this for every single web site we host. > > I fire up IE 7. No problems with any site. > > I go back to Firefox and it's still having issues -- but *only* with > sites hosted behind our OpenBSD firewalls. > > I fire up "telnet" (after enabling it through the control panel -- no > idea why MS did this). I can telnet to our web servers via port 80, > issue "GET" requests, receive responses. No trouble with Telnet.exe. > > Putty however, has trouble. Wont work period. Port 80 telnet, ssh port > 22, etc. None of them. > > So now I am thinking that it might just be a Firefox problem... but > it's not. Microsoft's own Remote Desktop Connection (terminal services > client, rdp client, etc) wont connect to our datacenter servers -- and > they are accessed via an openvpn point to point VPN that terminates on > the OpenBSD firewall which acts strictly as a routed tunnel between > our two networks. > > I turn off as much of the Vista "security" features as I can. This does > nothing. > > Since our OBSD firewalls were of the older variety (3.6), I figured I > might try an upgrade to 4.0 to see what happens. No dice. > > To summarize: > > This **only** is affecting Windows Vista (have not tried the latest > betas of Longhorn Server). Windows XP, Windows 2000, Free/OpenBSD, > CentOS, and our four Mac users with OSX have zero trouble. None. Nada. > They work flawlessly. > > Okay, so we can blame Vista -- that would be fine with me, but let's > face it -- this going to be big come January. I have a month to fix > this damn thing and I am really out of ideas. > > Our network: > > 100mbit dedicated inet connection through AT&T, terminates to a big > Cisco setup owned by our datacenter. > > Firewalls are now OBSD 4.0, single-proc Xeon 2.4gHz, 1GB RAM, etc. -- > they are decent systems with six gigabit NICs each. > > They are all configured with CARP and pfsync. This has worked very, > very well since day 1 in 2004! CARP rocks! > > They connect to an HP ProCurve 5400ZL modular switch, configured with > various port VLANs, etc. Everything is gigabit, 'cept for a few > databases using 10-gigabit CX4. > > Here are some tcpdumps from the master FW during connection attempts > with a browser: > > > > Opera 9: > > 20:40:45.824144 my.workstation.ip.49370 > remote.server.ip.80: S > 1215871830:1215871830(0) win 8192 8,nop,nop,sackOK> (DF) > 20:40:45.824646 207.218.64.33.80 > my.workstation.ip.49370: S > 2582857930:2582857930(0) ack 1215871831 win 64240 0,nop,nop,sackOK> > 20:40:45.878361 my.workstation.ip.49370 > 207.218.64.33.80: . ack 1 win 260 > (DF) > 20:40:45.904597 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.058234 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > 20:40:46.061253 my.workstation.ip.49370 > 207.218.64.33.80: P > 1:384(383) ack 1 win 260 (DF) > 20:40:46.061726 207.218.64.33.80 > my.workstation.ip.49370: . ack 384 > win 63857 (DF) > (at this point, the connection is hung -- the Vista workstation > receives no further communcations -- it's like it just drops the > replies) > > > > Firefox: > > 20:38:25.197691 my.workstation.ip.49357 > remote.server.ip.80: S > 643900711:643900711(0) win 8192 > (DF) > 20:38:25.198320 remote.server.ip.80 > my.workstation.ip.49357: S > 852828096:852828096(0) ack 643900712 win 64240 0,nop,nop,sackOK> > 20:38:25.244540 my.workstation.ip.49357 > remote.server.ip.80: . ack 1 > win 260 (DF) > 20:38:25.251037 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.567602 my.workstation.ip.49357 > remote.server.ip.80: P > 1:403(402) ack 1 win 260 (DF) > 20:38:25.568042 remote.server.ip.80 > my.workstation.ip.49357: . ack > 403 win 63838 (DF) > (s
