Trouble with httpGate and FreeBSD 9.1
Hi All,
When I tried to use httpGate on my NanoBSD box, I've got following error:
400 Bad Request
# uname -a
FreeBSD fs1.mtmlab.ru 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r245297: Sat Jan 12
08:23:47 YEKT 2013
root@proto.mtmlab.local:/usr/obj/nanobsd.NANOSERVER/usr/src/sys/NANOSERVER
amd64
My little investigation shows, that httpGate (or TCP stack?) sends "GET /" in
first TCP packet and the rest of request in second TCP packet.
And what I see in "http" function :
1) without httpGate
! L
-> ("G" "E" "T" " " "/" " " "H" "T" "T" "P" "/" "1" "." "1")
2) with httpGate
! L
-> ("G" "E" "T" " " "/")
If I understand correctly, (line) immediatly returns first TCP packet value.
I'm not expert in TCP/IP programming, maybe reason is FreeBSD itself?
I tried both 32 bit and 64 pil and httpGate with the same results.
tcpdump listing:
... skipped
19:32:16.579988 IP6 ::1.22649 > ::1.8080: Flags [P.], seq 1:6, ack 1, win 1120,
options [nop,nop,TS val 42586113 ecr 2588126972], length 5
0x: 600b 4bd5 0025 0640 `.K..%.@
0x0010: 0001
0x0020: 0001 5879 1f90 07b4 e127 Xy.'
0x0030: 8b75 e17c 8018 0460 002d 0101 080a .u.|...`.-..
0x0040: 0289 d001 9a43 aefc 4745 5420 2f .C..GET./
19:32:16.581157 IP6 ::1.8080 > ::1.22649: Flags [P.], seq 1:185, ack 6, win
1120,
options [nop,nop,TS val 2588126974 ecr 42586113], length 184
0x: 6008 9f78 00d8 0640 `..x...@
0x0010: 0001
0x0020: 0001 1f90 5879 8b75 e17c ..Xy.u.|
0x0030: 07b4 e12c 8018 0460 00e0 0101 080a ...,...`
0x0040: 9a43 aefe 0289 d001 4854 5450 2f31 2e30 .C..HTTP/1.0
0x0050: 2034 3030 2042 6164 2052 6571 7565 7374 .400.Bad.Request
0x0060: 0d0a 5365 7276 6572 3a20 5069 636f 4c69 ..Server:.PicoLi
0x0070: 7370 0d0a 436f 6e74 656e 742d 5479 7065 sp..Content-Type
0x0080: 3a20 7465 7874 2f68 746d 6c0d 0a43 6f6e :.text/html..Con
0x0090: 7465 6e74 2d4c 656e 6774 683a 2039 330d tent-Length:.93.
0x00a0: 0a0d 0a3c 4854 4d4c 3e0a 3c48 4541 443e
0x00b0: 3c54 4954 4c45 3e34 3030 2042 6164 2052 400.Bad.R
0x00c0: 6571 7565 7374 3c2f 5449 544c 453e 3c2f equest.
0x00e0: 4261 6420 5265 7175 6573 743c 2f48 313e Bad.Request
0x00f0: 3c2f 424f 4459 3e0a 3c2f 4854 4d4c 3e0a ..
19:32:16.581218 IP6 ::1.22649 > ::1.8080: Flags [P.], seq 6:369, ack 185, win
1117,
options [nop,nop,TS val 42586115 ecr 2588126974], length 363
0x: 600b 4bd5 018b 0640 `.K@
0x0010: 0001
0x0020: 0001 5879 1f90 07b4 e12c Xy.,
0x0030: 8b75 e234 8018 045d 0193 0101 080a .u.4...]
0x0040: 0289 d003 9a43 aefe 2048 5454 502f 312e .C...HTTP/1.
0x0050: 310d 0a58 2d50 696c 3a20 2a47 6174 653d 1..X-Pil:.*Gate=
0x0060: 6874 7470 0d0a 582d 5069 6c3a 202a 4164 http..X-Pil:.*Ad
0x0070: 723d 3a3a 3a31 3932 2e31 3638 r=:::192.168
0x0080: 2e31 302e 320d 0a48 6f73 743a 2031 3932 .10.2..Host:.192
0x0090: 2e31 3638 2e31 302e 350d 0a55 7365 722d .168.10.5..User-
0x00a0: 4167 656e 743a 204d 6f7a 696c 6c61 2f35 Agent:.Mozilla/5
0x00b0: 2e30 2028 5831 313b 2055 6275 6e74 753b .0.(X11;.Ubuntu;
0x00c0: 204c 696e 7578 2078 3836 5f36 343b 2072 .Linux.x86_64;.r
0x00d0: 763a 3138 2e30 2920 4765 636b 6f2f 3230 v:18.0).Gecko/20
0x00e0: 3130 3031 3031 2046 6972 6566 6f78 2f31 100101.Firefox/1
0x00f0: 382e 300d 0a41 6363 6570 743a 2074 6578 8.0..Accept:.tex
0x0100: 742f 6874 6d6c 2c61 7070 6c69 6361 7469 t/html,applicati
0x0110: 6f6e 2f78 6874 6d6c 2b78 6d6c 2c61 7070 on/xhtml+xml,app
0x0120: 6c69 6361 7469 6f6e 2f78 6d6c 3b71 3d30 lication/xml;q=0
0x0130: 2e39 2c2a 2f2a 3b71 3d30 2e38 0d0a 4163 .9,*/*;q=0.8..Ac
0x0140: 6365 7074 2d4c 616e 6775 6167 653a 2065 cept-Language:.e
0x0150: 6e2d 5553 2c65 6e3b 713d 302e 350d 0a41 n-US,en;q=0.5..A
0x0160: 6363 6570 742d 456e 636f 6469 6e67 3a20 ccept-Encoding:.
0x0170: 677a 6970 2c20 6465 666c 6174 650d 0a43 gzip,.deflate..C
0x0180: 6f6e 6e65 6374 696f 6e3a 206b 6565 702d onnection:.keep-
0x0190: 616c 6976 650d 0a43 6163 6865 2d43 6f6e alive..Cache-Con
0x01a0: 7472 6f6c 3a20 6d61 782d 6167 653d 300d trol:.max-age=0.
0x01b0: 0a0d 0a ...
... skipped
Best regards,
Mansur
Re: Trouble with httpGate and FreeBSD 9.1
Hi Mansur, thanks for the bug report! First of all, I'd like to ask you which version of 'httpGate' you are using. Is it the latest one, with the source date "20mar12" (i.e. picoLisp-3.1.0 or newer)? > My little investigation shows, that httpGate (or TCP stack?) sends > "GET /" in first TCP packet and the rest of request in second TCP > packet. I'm asking the above, because this behavior was observed last year too, and resulted in a rewrite of some parts of 'httpGate'. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re[2]: Trouble with httpGate and FreeBSD 9.1
Hi Alex, > First of all, I'd like to ask you which version of 'httpGate' you are > using. Is it the latest one, with the source date "20mar12" (i.e. > picoLisp-3.1.0 or newer)? Yes, the source date is: 20mar12abu Best regards, Mansur PԔ � &j)mX��� ��zV�u�.n7�
Re[3]: Trouble with httpGate and FreeBSD 9.1
Hi again, Maybe it's possible to use SO_RCVLOWAT/SO_SNDLOWAT socket options? Or maybe server side should do some "reassembly" (with a little timeouts to prevent DoS attacks) to get whole line of the request Just IMHO.
Re: Trouble with httpGate and FreeBSD 9.1
Hi Mansur,
> Maybe it's possible to use SO_RCVLOWAT/SO_SNDLOWAT socket options?
As far as I understand it, this should not be necessary (see below).
> Or maybe server side should do some "reassembly" (with a little
> timeouts to prevent DoS attacks) to get whole line of the request
Yes, this is what the change from Mar 20th does.
It is the new function rdLine(), which reads from the socket until at
least a full line is received. It does so on the _receiving_ side of
httpGate, i.e. on the way from the browser to httpGate.
But your description seems to refer to the _sending_ side of httpGate.
There is no special handling on the sending side, i.e. from httpGate to
the PicoLisp process, because here plain TCP (without SSL encryption) is
used and thus the 'line' function should be safe to receive the complete
line.
> 2) with httpGate
> ! L
> -> ("G" "E" "T" " " "/")
>
> If I understand correctly, (line) immediatly returns first TCP packet
> value. I'm not expert in TCP/IP programming, maybe reason is FreeBSD
> itself?
As I understand it, 'line' should not be concerned about TCP packets.
These are on a lower level. PicoLisp reads from a socket, and should
receive either correct data or EOF, but never partial data. It is a
principle of TCP to handle this.
So I must say I have no clue at the moment ... Do you have an idea how
to further debug this?
♪♫ Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re[2]: Trouble with httpGate and FreeBSD 9.1
> Hi Mansur,
>
> > Maybe it's possible to use SO_RCVLOWAT/SO_SNDLOWAT socket options?
>
> As far as I understand it, this should not be necessary (see below).
>
>
> > Or maybe server side should do some "reassembly" (with a little
> > timeouts to prevent DoS attacks) to get whole line of the request
>
> Yes, this is what the change from Mar 20th does.
>
> It is the new function rdLine(), which reads from the socket until at
> least a full line is received. It does so on the _receiving_ side of
> httpGate, i.e. on the way from the browser to httpGate.
>
>
> But your description seems to refer to the _sending_ side of httpGate.
Right.
>
> There is no special handling on the sending side, i.e. from httpGate to
> the PicoLisp process, because here plain TCP (without SSL encryption) is
> used and thus the 'line' function should be safe to receive the complete
> line.
>
>
> > 2) with httpGate
> > ! L
> > -> ("G" "E" "T" " " "/")
> >
> > If I understand correctly, (line) immediatly returns first TCP packet
> > value. I'm not expert in TCP/IP programming, maybe reason is FreeBSD
> > itself?
>
> As I understand it, 'line' should not be concerned about TCP packets.
> These are on a lower level. PicoLisp reads from a socket, and should
> receive either correct data or EOF, but never partial data. It is a
> principle of TCP to handle this.
I can not agree with this. TCP guarantees only :
- data arrives in-order
- data has minimal error (i.e. correctness)
- duplicate data is discarded
- lost/discarded packets are resent
At the moment of calling (line) it's highly possible that there is no whole
line in receive TCP buffer,
and then application will get only arrived part of the request line and eof.
See also tcpdump listing in some of my prevous letters, which shows that pil
server
gets "GET /" piece, responds with "400 Bad request" and after that arrives the
large rest of the request.
So taking into account the principles of TCP stream that's perfectly legal
situation.
>
> So I must say I have no clue at the moment ... Do you have an idea how
> to further debug this?
Currently I think that pil server needs to do something to wait until more data
arrived.
Maybe at the lisp level (in http function) or with socket options.
Best regards,
Mansur
