Re: JK connector and extra characters showing up
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 6/30/2010 11:18 PM, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: JK connector and extra characters showing up Those 4 extra characters are likely to be the chunk size. 31 66 66 38 is, well, 1ff8, which is 792 in decimal. Not on my calculator; would you believe 8184 in decimal? There's an extremely low probability of having a decimal value containing fewer digits than its hex equivalent... Hmm. It appears I was sleepy and lazy altogether. 8 + 16*15 + 32 * 15 + 64 * 1 = 792 but the LHS is complete malarkey. At least my arithmetic was correct ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwuiDcACgkQ9CaO5/Lv0PA50gCfTIFZAozmYPnf2mfIPjc7c9GE e+MAn0rDAb6XYpKsf4eKdDbJlh3iZ2lT =O2b0 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JK connector and extra characters showing up
Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: JK connector and extra characters showing up Those 4 extra characters are likely to be the chunk size. 31 66 66 38 is, well, 1ff8, which is 792 in decimal. Not on my calculator; would you believe 8184 in decimal? There's an extremely low probability of having a decimal value containing fewer digits than its hex equivalent... Guys, is it me, or you, that is getting a bit confused here ? First of all, what /are/ these captures ? From re-reading David's original post : ... Here are some snippets of packet captures (tcpdump) to show what I mean. ... Tomcat to web server through JK connector, same for Sun One and Apache ... It is not really clear where this data was captured. Between Tomcat and the jk connector (emebedded in the webserver) ? In that case, we are looking at binary data in AJP protocol format, not at HTTP data per se. Not so ? And if so, what's to tell what this 1f f8 might really be there for ? Apologies if I'm the confused one. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JK connector and extra characters showing up
On 01.07.2010 03:00, Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
David,
On 6/30/2010 3:32 PM, David Brown wrote:
Problem:
Extra characters showing up in some content delivered from tomcat. I believe
they are from the JK connector when it breaks up the content into 8k
packets.
Setup:
Tomcat 5.5 - JK 1.2.30 - SunOne 6.1sp11
So you're using mod_jk 1.2.30 to connect Tomcat 5.5 and SunOne?
I tested using Apache2 and the problem does not show up there. Using apache
is not an option here.
Okay.
Tomcat to web server through JK connector, same for Sun One and Apache
Is this data /from/ Tomcat /to/ Sun One, or from Sun One /to/ Tomcat?
That is, are we looking at a request or a response? It kind of looks
like a response, but I just want to be sure.
0090 20 47 4d 54 00 00 0c 43 6f 6e 74 65 6e 74 2d 54 GMT...Content-T
00a0 79 70 65 00 00 08 74 65 78 74 2f 63 73 73 00 00 ype...text/css..
00b0 0e 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 00 .Content-Length.
00c0 00 05 32 32 33 37 33 00 41 42 1f fc 03 1f f8 40 ..22373.AB.@
00d0 43 48 41 52 53 45 54 20 22 55 54 46 2d 38 22 3b CHARSET UTF-8;
00e0 23 74 70 63 72 7b 62 61 63 6b 67 72 6f 75 6e 64 #tpcr{background
00f0 2d 63 6f 6c 6f 72 3a 57 68 69 74 65 3b 6d 61 72 -color:White;mar
0100 67 69 6e 3a 31 30 70 78 20 30 20 32 30 70 78 20 gin:10px 0 20px
Can you dump the whole response?
Browser from Apache
0120 76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 ve..Content-Type
0130 3a 20 74 65 78 74 2f 63 73 73 0d 0a 0d 0a 40 43 : text/css@c
0140 48 41 52 53 45 54 20 22 55 54 46 2d 38 22 3b 23 HARSET UTF-8;#
0150 74 70 63 72 7b 62 61 63 6b 67 72 6f 75 6e 64 2d tpcr{background-
0160 63 6f 6c 6f 72 3a 57 68 69 74 65 3b 6d 61 72 67 color:White;marg
0170 69 6e 3a 31 30 70 78 20 30 20 32 30 70 78 20 30 in:10px 0 20px 0
Why are the hex offsets different? Differing standard headers? Again,
can you post the whole response?
Browser from SunOne
00e0 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 GMT..Content-Typ
00f0 65 3a 20 74 65 78 74 2f 63 73 73 0d 0a 43 6f 6e e: text/css..Con
0100 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 32 33 tent-Length: 223
0110 37 33 0d 0a 54 72 61 6e 73 66 65 72 2d 65 6e 63 73..Transfer-enc
0120 6f 64 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a oding: chunked..
0130 0d 0a 31 66 66 38 0d 0a 40 43 48 41 52 53 45 54 ..1ff...@charset
0140 20 22 55 54 46 2d 38 22 3b 23 74 70 63 72 7b 62 UTF-8;#tpcr{b
0150 61 63 6b 67 72 6f 75 6e 64 2d 63 6f 6c 6f 72 3a ackground-color:
0160 57 68 69 74 65 3b 6d 61 72 67 69 6e 3a 31 30 70 White;margin:10p
0170 78 20 30 20 32 30 70 78 20 30 3b 7d 0a 23 74 70 x 0 20px 0;}.#tp
Are all of these dumps from the same response, but at different points
in the process?
I can see that there is a 1ff8 (in text) in that last dump. What is that?
It appears that some component is switching the Transfer-encoding to
chunked. Do you know if that's intentional?
The first snippet is from between the web server and tomcat through the JK
connector. This looks the same for either Apache or SunOne.
The thing to note is line 00c0 where the hex is 1f f8.
Is that a Greek Omicron? Or something else?
The second snippet is when a browser hits Apache. The thing to note is line
0130 where the hex is 0d 0a 0d 0a. (carriage return, line feed, carriage
return, line feed)
The CR LF CR LF seems to be more likely to be correct.
The third snippet is when a browser hits SunOne for the same file. Here on
line 0130 there is 0d 0a 31 66 66 38 0d 0a, notice the extra 4 characters
between the carriage return/line feeds.
Those 4 extra characters are likely to be the chunk size. 31 66 66 38
is, well, 1ff8, which is 792 in decimal. So, the chunk size is 792
bytes. Did you get 792 bytes after the next CR LF? Again, a complete
response would be helpful in determining what's happening.
And that is where my problem lies. These characters 1ff8 are showing up in
the body of the content and is causing errors.
Technically speaking, this is not content: it's header. Your client is
misinterpreting the data it's receiving from the server.
Take a look at http://www.httpwatch.com/httpgallery/chunked/ - the page
is chunked with each line of text in a separate chunk. I think it will
demonstrate what I'm talking about. If you can't view it any other way,
you can do this:
$ telnet www.httpwatch.com 80 temp.out
GET /httpgallery/chunked/
Connection closed by foreign host.
$ less temp.out
You should see content like this:
[snip]
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Pragma: no-cache
Expires: -1
Content-Type: text/html
7b
!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;
2d
html xmlns=http://www.w3.org/1999/xhtml;
[and so on]
9
/body
9
/html
2
0
[the 0 indicates the last chunk, which contains no data].
Is this what you're observing, here?
Re: JK connector and extra characters showing up
First let me thank everyone for looking at this. Now I'll try to answer some of the questions and clear up the confusion (if I can). All these dumps are from responses and not request. I'll post more complete dumps at he end of this message. The first one is the communications between tomcat and the web server, AJP protocol. Since it was the same for both Apache and SunOne I only posted one of them. The second and third are from between a browser and the web server, Apache and SunOne. The only difference is the web server and the JK connector (mod_jk vs jk_nsapi). Same tomcat, application, file (style sheet), browser, servers, and network. Now here's what I'm seeing. In dump A (tomcat jk) in packet 2 at line 00c0 look at the end of the line's hex. It's 03 1f f8 40. Pay attention to the 1f f8, it shows up latter. In dump B (Apache) in packet 2 at line 0130 towards the end of the line of hex is 0d 0a 0d 0a (CR LF CR LF). Normal Now in dump C (SunOne) in packet 2 at line 0130 towards the beginning is 0d 0a 31 66 66 38 0d 0a or CR LF 1f f8 CR LF. It seems to me that the hex 1f f8 seen the first dump is making its way into the output in the third dump. I'm thinking there's a difference in the behavior of the JK connector between Apache and SunOne. Now for some background. We've been running this setup for 6 or 7 years now without a problem. Browsers, wget, curl, Squid are not affected by this, maybe they see the break between header and body as the second CR LF. Recently we've tried using Varnish as our cache and it seems to see the break as the first CR LF and included the 1f f8 in the body of the response. This is where we are seeing errors. Yes, i am posting to Varnish's mailing list to to see if they can help. So I ether need consistent output from the JK connector or for Varnish to break the header/body at the second CR LF. Here's more dump for your reading pleasure A) Tomcat to web server (response) AJP Packet #1 0e 91 b2 32 3b 90 00 03 ba ec ea 76 08 00 45 00 ...2;..v..E. 0010 01 eb 4e 1a 40 00 40 06 00 00 c0 a8 b6 20 c0 a8 @.@.. .. 0020 b6 1e 80 7c 1f 49 ff 04 18 db e5 67 e9 83 50 18 ...|.I.g..P. 0030 c1 e8 00 00 00 00 12 34 01 bf 02 02 00 08 48 54 ...4..HT 0040 54 50 2f 31 2e 31 00 00 2b 2f 63 6f 6d 70 6f 6e TP/1.1..+/compon 0050 65 6e 74 73 2f 72 65 73 6f 75 72 63 65 73 2f 63 ents/resources/c 0060 73 73 2f 74 70 63 2d 61 67 67 72 65 67 61 74 65 ss/tpc-aggregate 0070 2e 63 73 73 00 00 0e 31 39 32 2e 31 36 38 2e 32 .css...192.168.2 0080 31 30 2e 36 35 00 ff ff 00 08 77 65 62 61 70 70 10.65.webapp 0090 2d 66 00 00 50 00 00 09 a0 0b 00 08 77 65 62 61 -f..P...weba 00a0 70 70 2d 66 00 a0 0e 00 61 4d 6f 7a 69 6c 6c 61 pp-faMozilla 00b0 2f 35 2e 30 20 28 4d 61 63 69 6e 74 6f 73 68 3b /5.0 (Macintosh; 00c0 20 55 3b 20 49 6e 74 65 6c 20 4d 61 63 20 4f 53 U; Intel Mac OS 00d0 20 58 20 31 30 2e 35 3b 20 65 6e 2d 55 53 3b 20 X 10.5; en-US; 00e0 72 76 3a 31 2e 39 2e 31 2e 31 30 29 20 47 65 63 rv:1.9.1.10) Gec 00f0 6b 6f 2f 32 30 31 30 30 35 30 34 20 46 69 72 65 ko/20100504 Fire 0100 66 6f 78 2f 33 2e 35 2e 31 30 00 a0 01 00 3f 74 fox/3.5.10?t 0110 65 78 74 2f 68 74 6d 6c 2c 61 70 70 6c 69 63 61 ext/html,applica 0120 74 69 6f 6e 2f 78 68 74 6d 6c 2b 78 6d 6c 2c 61 tion/xhtml+xml,a 0130 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c 3b 71 pplication/xml;q 0140 3d 30 2e 39 2c 2a 2f 2a 3b 71 3d 30 2e 38 00 00 =0.9,*/*;q=0.8.. 0150 0f 41 63 63 65 70 74 2d 4c 61 6e 67 75 61 67 65 .Accept-Language 0160 00 00 0e 65 6e 2d 75 73 2c 65 6e 3b 71 3d 30 2e ...en-us,en;q=0. 0170 35 00 00 0f 41 63 63 65 70 74 2d 45 6e 63 6f 64 5...Accept-Encod 0180 69 6e 67 00 00 0c 67 7a 69 70 2c 64 65 66 6c 61 ing...gzip,defla 0190 74 65 00 00 0e 41 63 63 65 70 74 2d 43 68 61 72 te...Accept-Char 01a0 73 65 74 00 00 1e 49 53 4f 2d 38 38 35 39 2d 31 set...ISO-8859-1 01b0 2c 75 74 66 2d 38 3b 71 3d 30 2e 37 2c 2a 3b 71 ,utf-8;q=0.7,*;q 01c0 3d 30 2e 37 00 00 0a 4b 65 65 70 2d 41 6c 69 76 =0.7...Keep-Aliv 01d0 65 00 00 03 33 30 30 00 a0 06 00 0a 6b 65 65 70 e...300.keep 01e0 2d 61 6c 69 76 65 00 a0 08 00 01 30 00 06 00 07 -alive.0 01f0 77 6f 72 6b 65 72 36 00 ff worker6.. Packet #2 00 03 ba ec ea 76 0e 91 b2 32 3b 90 08 00 45 00 .v...2;...E. 0010 05 dc 5b f5 40 00 3c 06 ef 96 c0 a8 b6 1e c0 a8 @.. 0020 b6 20 1f 49 80 7c e5 67 e9 83 ff 04 1a 9e 50 10 . .I.|.g..P. 0030 c1 e8 1b f3 00 00 41 42 00 8e 04 00 c8 00 02 4f ..AB...O 0040 4b 00 00 04 00 04 45 54 61 67 00 00 17 57 2f 22 K.ETag...W/ 0050 32 32 33 37 33 2d 31 32 37 37 34 39 39 37 33 39 22373-1277499739 0060 30 30 30 22 00 00 0d 4c 61 73 74 2d 4d 6f 64 69 000...Last-Modi 0070 66 69 65 64 00 00 1d 46 72 69 2c 20 32 35 20 4a fied...Fri, 25 J 0080 75 6e 20 32 30 31 30 20 32 31 3a 30 32 3a 31 39 un
RE: JK connector and extra characters showing up
From: David Brown [mailto:captki...@gmail.com] Subject: Re: JK connector and extra characters showing up Now here's what I'm seeing. In dump A (tomcat jk) in packet 2 at line 00c0 look at the end of the line's hex. It's 03 1f f8 40. Pay attention to the 1f f8, it shows up latter. Rainer already told you what the problem is; the webapp is violating the HTTP spec: It *seems* your application sends a Content-Length header and does chunked encoding at the same time. That's an invalid response. Your Apache snippet shows that it clears that up by dropping the Content-Length header. The SunONE snippet shows that combination send both variants and confuses the client. The root cause though would sit in your webapp, which needs to decide to send Content-Length only if it is not doing Transfer-Encoding chunked. httpd cleans up your error, but SunONE isn't that smart. Fix your webapp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JK connector and extra characters showing up
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
David,
On 6/30/2010 3:32 PM, David Brown wrote:
Problem:
Extra characters showing up in some content delivered from tomcat. I believe
they are from the JK connector when it breaks up the content into 8k
packets.
Setup:
Tomcat 5.5 - JK 1.2.30 - SunOne 6.1sp11
So you're using mod_jk 1.2.30 to connect Tomcat 5.5 and SunOne?
I tested using Apache2 and the problem does not show up there. Using apache
is not an option here.
Okay.
Tomcat to web server through JK connector, same for Sun One and Apache
Is this data /from/ Tomcat /to/ Sun One, or from Sun One /to/ Tomcat?
That is, are we looking at a request or a response? It kind of looks
like a response, but I just want to be sure.
0090 20 47 4d 54 00 00 0c 43 6f 6e 74 65 6e 74 2d 54 GMT...Content-T
00a0 79 70 65 00 00 08 74 65 78 74 2f 63 73 73 00 00 ype...text/css..
00b0 0e 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 00 .Content-Length.
00c0 00 05 32 32 33 37 33 00 41 42 1f fc 03 1f f8 40 ..22373.AB.@
00d0 43 48 41 52 53 45 54 20 22 55 54 46 2d 38 22 3b CHARSET UTF-8;
00e0 23 74 70 63 72 7b 62 61 63 6b 67 72 6f 75 6e 64 #tpcr{background
00f0 2d 63 6f 6c 6f 72 3a 57 68 69 74 65 3b 6d 61 72 -color:White;mar
0100 67 69 6e 3a 31 30 70 78 20 30 20 32 30 70 78 20 gin:10px 0 20px
Can you dump the whole response?
Browser from Apache
0120 76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 ve..Content-Type
0130 3a 20 74 65 78 74 2f 63 73 73 0d 0a 0d 0a 40 43 : text/css@c
0140 48 41 52 53 45 54 20 22 55 54 46 2d 38 22 3b 23 HARSET UTF-8;#
0150 74 70 63 72 7b 62 61 63 6b 67 72 6f 75 6e 64 2d tpcr{background-
0160 63 6f 6c 6f 72 3a 57 68 69 74 65 3b 6d 61 72 67 color:White;marg
0170 69 6e 3a 31 30 70 78 20 30 20 32 30 70 78 20 30 in:10px 0 20px 0
Why are the hex offsets different? Differing standard headers? Again,
can you post the whole response?
Browser from SunOne
00e0 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 GMT..Content-Typ
00f0 65 3a 20 74 65 78 74 2f 63 73 73 0d 0a 43 6f 6e e: text/css..Con
0100 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 32 33 tent-Length: 223
0110 37 33 0d 0a 54 72 61 6e 73 66 65 72 2d 65 6e 63 73..Transfer-enc
0120 6f 64 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a oding: chunked..
0130 0d 0a 31 66 66 38 0d 0a 40 43 48 41 52 53 45 54 ..1ff...@charset
0140 20 22 55 54 46 2d 38 22 3b 23 74 70 63 72 7b 62 UTF-8;#tpcr{b
0150 61 63 6b 67 72 6f 75 6e 64 2d 63 6f 6c 6f 72 3a ackground-color:
0160 57 68 69 74 65 3b 6d 61 72 67 69 6e 3a 31 30 70 White;margin:10p
0170 78 20 30 20 32 30 70 78 20 30 3b 7d 0a 23 74 70 x 0 20px 0;}.#tp
Are all of these dumps from the same response, but at different points
in the process?
I can see that there is a 1ff8 (in text) in that last dump. What is that?
It appears that some component is switching the Transfer-encoding to
chunked. Do you know if that's intentional?
The first snippet is from between the web server and tomcat through the JK
connector. This looks the same for either Apache or SunOne.
The thing to note is line 00c0 where the hex is 1f f8.
Is that a Greek Omicron? Or something else?
The second snippet is when a browser hits Apache. The thing to note is line
0130 where the hex is 0d 0a 0d 0a. (carriage return, line feed, carriage
return, line feed)
The CR LF CR LF seems to be more likely to be correct.
The third snippet is when a browser hits SunOne for the same file. Here on
line 0130 there is 0d 0a 31 66 66 38 0d 0a, notice the extra 4 characters
between the carriage return/line feeds.
Those 4 extra characters are likely to be the chunk size. 31 66 66 38
is, well, 1ff8, which is 792 in decimal. So, the chunk size is 792
bytes. Did you get 792 bytes after the next CR LF? Again, a complete
response would be helpful in determining what's happening.
And that is where my problem lies. These characters 1ff8 are showing up in
the body of the content and is causing errors.
Technically speaking, this is not content: it's header. Your client is
misinterpreting the data it's receiving from the server.
Take a look at http://www.httpwatch.com/httpgallery/chunked/ - the page
is chunked with each line of text in a separate chunk. I think it will
demonstrate what I'm talking about. If you can't view it any other way,
you can do this:
$ telnet www.httpwatch.com 80 temp.out
GET /httpgallery/chunked/
Connection closed by foreign host.
$ less temp.out
You should see content like this:
[snip]
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Pragma: no-cache
Expires: -1
Content-Type: text/html
7b
!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;
2d
html xmlns=http://www.w3.org/1999/xhtml;
[and so on]
9
/body
9
/html
2
0
[the 0 indicates the last chunk, which contains no data].
Is this what you're observing, here? If so, I think it's
RE: JK connector and extra characters showing up
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: JK connector and extra characters showing up Those 4 extra characters are likely to be the chunk size. 31 66 66 38 is, well, 1ff8, which is 792 in decimal. Not on my calculator; would you believe 8184 in decimal? There's an extremely low probability of having a decimal value containing fewer digits than its hex equivalent... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
