RE: Tomcat sucks at receiving large messages
Since I will be occasionally receiving messages in the 10Mbyte range, I
can't read in a line at a time - it takes too long.
The bug in the code below is because BufferedReader.read() will not
necessarily return the whole buffer. So I replace the line
reader.read(charArr);
With this:
int length = req.getContentLength();
char [] charArr = new char[length];
int readResult = 0;
int sum = 0;
do {
sum += readResult;
length -= readResult;
readResult = reader.read(charArr, sum, length);
} while (readResult length);
Thanks for your help. Any other critiques on the use of the standard
library are welcome.
Dan
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 30, 2003 11:30 AM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); } }
What happens if you ditch the req.getContentLength() approach (there are
times when it will be -1 anyways), and do something like: BufferedReader
reader = req.getReader(); StringBuffer contents = new StringBuffer();
String line = null; while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential,
proprietary and/or privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your
computer system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
This code will be running in a controlled environment, with known
clients, where the largest message size is known (~10M). This code
takes the entire body and forwards it on to another messaging system, so
I have no choice but to deal with the entire message. And I can't read
it a byte or line at a time, because it would take too long.
Take a look at my other response to this subject to see the code that
fixed my problem. I am open to any other suggestions
Dan
-Original Message-
From: Walker Chris [mailto:[EMAIL PROTECTED]
Sent: Wednesday, October 01, 2003 1:21 AM
To: 'Tomcat Users List'
Subject: RE: Tomcat sucks at receiving large messages
Hi,
I should have thought that as a general principle it's not a good idea
to try to store the response in a byte array. I recently worked on a
piece of code that did just that (worse, actually, it then copied the
array into a String). Sooner or later a really big upload will blow up
the application.
Reading and writing a byte at a time (with appropriate buffering)
requires a bit more ingenuity, especially when you're searching for
things like boundary strings in the response, but it's the only way to
remove any constraint on upload size.
Chris Walker
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: 30 September 2003 19:30
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); } }
What happens if you ditch the req.getContentLength() approach (there are
times when it will be -1 anyways), and do something like: BufferedReader
reader = req.getReader(); StringBuffer contents = new StringBuffer();
String line = null; while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential,
proprietary and/or privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your
computer system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Howdy,
Seems like a very decent fix. Thanks for posting it so others can have
a future reference solution ;)
I wonder if there's a java.nio solution that will perform better...
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: Stewart, Daniel J [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 02, 2003 12:34 PM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Since I will be occasionally receiving messages in the 10Mbyte range, I
can't read in a line at a time - it takes too long.
The bug in the code below is because BufferedReader.read() will not
necessarily return the whole buffer. So I replace the line
reader.read(charArr);
With this:
int length = req.getContentLength();
char [] charArr = new char[length];
int readResult = 0;
int sum = 0;
do {
sum += readResult;
length -= readResult;
readResult = reader.read(charArr, sum, length);
} while (readResult length);
Thanks for your help. Any other critiques on the use of the standard
library are welcome.
Dan
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 30, 2003 11:30 AM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); } }
What happens if you ditch the req.getContentLength() approach (there
are
times when it will be -1 anyways), and do something like:
BufferedReader
reader = req.getReader(); StringBuffer contents = new StringBuffer();
String line = null; while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential,
proprietary and/or privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your
computer system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
This e-mail, including any attachments, is a confidential business communication, and
may contain information that is confidential, proprietary and/or privileged. This
e-mail is intended only for the individual(s) to whom it is addressed, and may not be
saved, copied, printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your computer system
and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Unless your client is very conforming to the rules (ie. Content-Length is
is correct wrt to available bytes) you could be waiting for a while for the
stream of data to come across or until your socket read statement timeout
int length = req.getContentLength();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int count = 0
int total = 0;
byte[] buf = new byte[8192]; // not sure which OS u have but if u are on Windows,
// use 8192 for the default OS block size
InputStream is = req.getInputStream();
while ( (count = is.read(buf) ) != -1)
{
total += count;
baos.write(buf,0,count);
}
if (total != length)
{
// handle this case as u see fit.
}
last note: bytearray is prob. better than reader/char array unless
you don't intend to handle non-character data.
int length = req.getContentLength();
char [] charArr = new char[length];
int readResult = 0;
int sum = 0;
do {
sum += readResult;
length -= readResult;
readResult = reader.read(charArr, sum, length);
} while (readResult length);
Howdy,
Seems like a very decent fix. Thanks for posting it so
others can have
a future reference solution ;)
I wonder if there's a java.nio solution that will perform better...
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: Stewart, Daniel J [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 02, 2003 12:34 PM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Since I will be occasionally receiving messages in the
10Mbyte range, I
can't read in a line at a time - it takes too long.
The bug in the code below is because BufferedReader.read() will not
necessarily return the whole buffer. So I replace the line
reader.read(charArr);
With this:
int length = req.getContentLength();
char [] charArr = new char[length];
int readResult = 0;
int sum = 0;
do {
sum += readResult;
length -= readResult;
readResult = reader.read(charArr, sum, length);
} while (readResult length);
Thanks for your help. Any other critiques on the use of the standard
library are welcome.
Dan
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 30, 2003 11:30 AM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req,
HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); } }
What happens if you ditch the req.getContentLength() approach (there
are
times when it will be -1 anyways), and do something like:
BufferedReader
reader = req.getReader(); StringBuffer contents = new StringBuffer();
String line = null; while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure
you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential,
proprietary and/or privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your
computer system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
This e-mail, including any attachments, is a confidential
business communication, and may contain information that is
confidential, proprietary and/or privileged. This e-mail is
intended only for the individual(s) to whom it is addressed,
and may not be saved, copied, printed, disclosed or used by
anyone else. If you are not the(an) intended recipient,
please immediately delete this e-mail from your computer
system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL
RE: Tomcat sucks at receiving large messages
Hi,
I should have thought that as a general principle it's not a good idea to
try to store the response in a byte array. I recently worked on a piece of
code that did just that (worse, actually, it then copied the array into a
String). Sooner or later a really big upload will blow up the application.
Reading and writing a byte at a time (with appropriate buffering) requires a
bit more ingenuity, especially when you're searching for things like
boundary strings in the response, but it's the only way to remove any
constraint on upload size.
Chris Walker
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: 30 September 2003 19:30
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); }
}
What happens if you ditch the req.getContentLength() approach (there are
times when it will be -1 anyways), and do something like:
BufferedReader reader = req.getReader();
StringBuffer contents = new StringBuffer();
String line = null;
while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential, proprietary
and/or privileged. This e-mail is intended only for the individual(s) to
whom it is addressed, and may not be saved, copied, printed, disclosed or
used by anyone else. If you are not the(an) intended recipient, please
immediately delete this e-mail from your computer system and notify the
sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat sucks at receiving large messages
When receiving a HTTP 1.0 POST with a 10kbyte payload, my doPost() method writes the message body to a file. The file is the right size, but my data is nulled out (set to 0) after correctly receiving about 2kbytes. In frustration, I set the Connector bufferSize parameter to 100, to discover that I could now receive about 4kbytes of my 10kbyte message. I then proceeded to write my own Java server to receive the message and write it to a file, and it works just fine. I am at a loss - can anyone suggest what could be causing this problem? Vitals: Tomcat version: 4.1.27 Tomcat configuration: Out-of-the-box (except for my app's WEB-INF) OS: solaris 2.9 My servlet skill level: medium-low - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Howdy, Perhaps if you share your servlet which writes the message body to a file, we could help you write a better servlet ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Stewart, Daniel J [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2003 11:31 AM To: [EMAIL PROTECTED] Subject: Tomcat sucks at receiving large messages When receiving a HTTP 1.0 POST with a 10kbyte payload, my doPost() method writes the message body to a file. The file is the right size, but my data is nulled out (set to 0) after correctly receiving about 2kbytes. In frustration, I set the Connector bufferSize parameter to 100, to discover that I could now receive about 4kbytes of my 10kbyte message. I then proceeded to write my own Java server to receive the message and write it to a file, and it works just fine. I am at a loss - can anyone suggest what could be causing this problem? Vitals: Tomcat version: 4.1.27 Tomcat configuration: Out-of-the-box (except for my app's WEB-INF) OS: solaris 2.9 My servlet skill level: medium-low - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Here it is:
public class AdapterServlet extends HttpServlet {
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); }
}
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 30, 2003 8:34 AM
To: Tomcat Users List
Subject: RE: Tomcat sucks at receiving large messages
Howdy,
Perhaps if you share your servlet which writes the message body to a
file, we could help you write a better servlet ;)
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: Stewart, Daniel J [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 30, 2003 11:31 AM
To: [EMAIL PROTECTED]
Subject: Tomcat sucks at receiving large messages
When receiving a HTTP 1.0 POST with a 10kbyte payload, my doPost()
method writes the message body to a file. The file is the right size,
but my data is nulled out (set to 0) after correctly receiving about
2kbytes.
In frustration, I set the Connector bufferSize parameter to 100,
to discover that I could now receive about 4kbytes of my 10kbyte
message. I then proceeded to write my own Java server to receive the
message and write it to a file, and it works just fine.
I am at a loss - can anyone suggest what could be causing this problem?
Vitals:
Tomcat version: 4.1.27
Tomcat configuration: Out-of-the-box (except for my app's WEB-INF)
OS: solaris 2.9
My servlet skill level: medium-low
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
This e-mail, including any attachments, is a confidential business
communication, and may contain information that is confidential,
proprietary and/or privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your
computer system and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Howdy,
public void service(HttpServletRequest req, HttpServletResponse res) {
BufferedReader reader = req.getReader();
try {
char [] charArr = new char[req.getContentLength()];
reader.read(charArr);
String str = new String(charArr);
try {
File f = new File(servlet.out);
PrintWriter out = new PrintWriter(new FileWriter(f));
out.print(str);
out.flush();
out.close();
} catch(IOException err { System.err.println(err.toString()); }
} catch(IOException err) { System.err.println(err.toString()); }
}
What happens if you ditch the req.getContentLength() approach (there are
times when it will be -1 anyways), and do something like:
BufferedReader reader = req.getReader();
StringBuffer contents = new StringBuffer();
String line = null;
while((line = reader.readLine()) != null) {
contents.append(line);
}
System.out.println(contents);
(Later we'll worry about the writing -- first make sure you're reading
the entire contents).
Yoav Shapira
This e-mail, including any attachments, is a confidential business communication, and
may contain information that is confidential, proprietary and/or privileged. This
e-mail is intended only for the individual(s) to whom it is addressed, and may not be
saved, copied, printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your computer system
and notify the sender. Thank you.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
