RE: Tomcat sucks at receiving large messages
Unless your client is very conforming to the rules (ie. Content-Length is is correct wrt to available bytes) you could be waiting for a while for the stream of data to come across or until your socket read statement timeout int length = req.getContentLength(); ByteArrayOutputStream baos = new ByteArrayOutputStream(); int count = 0 int total = 0; byte[] buf = new byte[8192]; // not sure which OS u have but if u are on Windows, // use 8192 for the default OS block size InputStream is = req.getInputStream(); while ( (count = is.read(buf) ) != -1) { total += count; baos.write(buf,0,count); } if (total != length) { // handle this case as u see fit. } last note: bytearray is prob. better than reader/char array unless you don't intend to handle non-character data. > > int length = req.getContentLength(); > > char [] charArr = new char[length]; > > int readResult = 0; > > int sum = 0; > > do { > >sum += readResult; > >length -= readResult; > >readResult = reader.read(charArr, sum, length); > > } while (readResult < length); > > > Howdy, > Seems like a very decent fix. Thanks for posting it so > others can have > a future reference solution ;) > > I wonder if there's a java.nio solution that will perform better... > > Yoav Shapira > Millennium ChemInformatics > > > >-Original Message- > >From: Stewart, Daniel J [mailto:[EMAIL PROTECTED] > >Sent: Thursday, October 02, 2003 12:34 PM > >To: Tomcat Users List > >Subject: RE: Tomcat sucks at receiving large messages > > > >Since I will be occasionally receiving messages in the > 10Mbyte range, I > >can't read in a line at a time - it takes too long. > > > >The bug in the code below is because BufferedReader.read() will not > >necessarily return the whole buffer. So I replace the line > > reader.read(charArr); > >With this: > > int length = req.getContentLength(); > > char [] charArr = new char[length]; > > int readResult = 0; > > int sum = 0; > > do { > >sum += readResult; > >length -= readResult; > >readResult = reader.read(charArr, sum, length); > > } while (readResult < length); > > > > > >Thanks for your help. Any other critiques on the use of the standard > >library are welcome. > > > >Dan > >-Original Message- > >From: Shapira, Yoav [mailto:[EMAIL PROTECTED] > >Sent: Tuesday, September 30, 2003 11:30 AM > >To: Tomcat Users List > >Subject: RE: Tomcat sucks at receiving large messages > > > > > > > >Howdy, > > > >>public void service(HttpServletRequest req, > HttpServletResponse res) { > >> BufferedReader reader = req.getReader(); > >> try { > >>char [] charArr = new char[req.getContentLength()]; > >>reader.read(charArr); > >>String str = new String(charArr); > >> > >>try { > >> File f = new File("servlet.out"); > >> PrintWriter out = new PrintWriter(new FileWriter(f)); > >> out.print(str); > >> out.flush(); > >> out.close(); > >>} catch(IOException err { System.err.println(err.toString()); } > >> > >> } catch(IOException err) { System.err.println(err.toString()); } } > > > >What happens if you ditch the req.getContentLength() approach (there > are > >times when it will be -1 anyways), and do something like: > BufferedReader > >reader = req.getReader(); StringBuffer contents = new StringBuffer(); > >String line = null; while((line = reader.readLine()) != null) { > > contents.append(line); > >} > > > >System.out.println(contents); > > > >(Later we'll worry about the writing -- first make sure > you're reading > >the entire contents). > > > >Yoav Shapira > > > > > > > >This e-mail, including any attachments, is a confidential business > >communication, and may contain information that is confidential, > >proprietary and/or privileged. This e-mail is intended only for the > >individual(s) to whom it is addressed, and may not be saved, copied, > >printed, disclosed or used by anyone else. If you are not the(an) > >intended recipient, please immediately delete this e-mail from your > >computer system and notify the sender. Thank you. > > > > > >- > >To unsubscribe, e-mail: [EMAIL PROTECTED] > >For additional commands,
RE: Tomcat sucks at receiving large messages
Howdy, Seems like a very decent fix. Thanks for posting it so others can have a future reference solution ;) I wonder if there's a java.nio solution that will perform better... Yoav Shapira Millennium ChemInformatics >-Original Message- >From: Stewart, Daniel J [mailto:[EMAIL PROTECTED] >Sent: Thursday, October 02, 2003 12:34 PM >To: Tomcat Users List >Subject: RE: Tomcat sucks at receiving large messages > >Since I will be occasionally receiving messages in the 10Mbyte range, I >can't read in a line at a time - it takes too long. > >The bug in the code below is because BufferedReader.read() will not >necessarily return the whole buffer. So I replace the line > reader.read(charArr); >With this: > int length = req.getContentLength(); > char [] charArr = new char[length]; > int readResult = 0; > int sum = 0; > do { >sum += readResult; >length -= readResult; >readResult = reader.read(charArr, sum, length); > } while (readResult < length); > > >Thanks for your help. Any other critiques on the use of the standard >library are welcome. > >Dan >-Original Message- >From: Shapira, Yoav [mailto:[EMAIL PROTECTED] >Sent: Tuesday, September 30, 2003 11:30 AM >To: Tomcat Users List >Subject: RE: Tomcat sucks at receiving large messages > > > >Howdy, > >>public void service(HttpServletRequest req, HttpServletResponse res) { >> BufferedReader reader = req.getReader(); >> try { >>char [] charArr = new char[req.getContentLength()]; >>reader.read(charArr); >>String str = new String(charArr); >> >>try { >> File f = new File("servlet.out"); >> PrintWriter out = new PrintWriter(new FileWriter(f)); >> out.print(str); >> out.flush(); >> out.close(); >>} catch(IOException err { System.err.println(err.toString()); } >> >> } catch(IOException err) { System.err.println(err.toString()); } } > >What happens if you ditch the req.getContentLength() approach (there are >times when it will be -1 anyways), and do something like: BufferedReader >reader = req.getReader(); StringBuffer contents = new StringBuffer(); >String line = null; while((line = reader.readLine()) != null) { > contents.append(line); >} > >System.out.println(contents); > >(Later we'll worry about the writing -- first make sure you're reading >the entire contents). > >Yoav Shapira > > > >This e-mail, including any attachments, is a confidential business >communication, and may contain information that is confidential, >proprietary and/or privileged. This e-mail is intended only for the >individual(s) to whom it is addressed, and may not be saved, copied, >printed, disclosed or used by anyone else. If you are not the(an) >intended recipient, please immediately delete this e-mail from your >computer system and notify the sender. Thank you. > > >- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] > > >- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
This code will be running in a controlled environment, with known clients, where the largest message size is known (~10M). This code takes the entire body and forwards it on to another messaging system, so I have no choice but to deal with the entire message. And I can't read it a byte or line at a time, because it would take too long. Take a look at my other response to this subject to see the code that fixed my problem. I am open to any other suggestions Dan -Original Message- From: Walker Chris [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 01, 2003 1:21 AM To: 'Tomcat Users List' Subject: RE: Tomcat sucks at receiving large messages Hi, I should have thought that as a general principle it's not a good idea to try to store the response in a byte array. I recently worked on a piece of code that did just that (worse, actually, it then copied the array into a String). Sooner or later a really big upload will blow up the application. Reading and writing a byte at a time (with appropriate buffering) requires a bit more ingenuity, especially when you're searching for things like boundary strings in the response, but it's the only way to remove any constraint on upload size. Chris Walker -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 30 September 2003 19:30 To: Tomcat Users List Subject: RE: Tomcat sucks at receiving large messages Howdy, >public void service(HttpServletRequest req, HttpServletResponse res) { > BufferedReader reader = req.getReader(); > try { >char [] charArr = new char[req.getContentLength()]; >reader.read(charArr); >String str = new String(charArr); > >try { > File f = new File("servlet.out"); > PrintWriter out = new PrintWriter(new FileWriter(f)); > out.print(str); > out.flush(); > out.close(); >} catch(IOException err { System.err.println(err.toString()); } > > } catch(IOException err) { System.err.println(err.toString()); } } What happens if you ditch the req.getContentLength() approach (there are times when it will be -1 anyways), and do something like: BufferedReader reader = req.getReader(); StringBuffer contents = new StringBuffer(); String line = null; while((line = reader.readLine()) != null) { contents.append(line); } System.out.println(contents); (Later we'll worry about the writing -- first make sure you're reading the entire contents). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Since I will be occasionally receiving messages in the 10Mbyte range, I can't read in a line at a time - it takes too long. The bug in the code below is because BufferedReader.read() will not necessarily return the whole buffer. So I replace the line reader.read(charArr); With this: int length = req.getContentLength(); char [] charArr = new char[length]; int readResult = 0; int sum = 0; do { sum += readResult; length -= readResult; readResult = reader.read(charArr, sum, length); } while (readResult < length); Thanks for your help. Any other critiques on the use of the standard library are welcome. Dan -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2003 11:30 AM To: Tomcat Users List Subject: RE: Tomcat sucks at receiving large messages Howdy, >public void service(HttpServletRequest req, HttpServletResponse res) { > BufferedReader reader = req.getReader(); > try { >char [] charArr = new char[req.getContentLength()]; >reader.read(charArr); >String str = new String(charArr); > >try { > File f = new File("servlet.out"); > PrintWriter out = new PrintWriter(new FileWriter(f)); > out.print(str); > out.flush(); > out.close(); >} catch(IOException err { System.err.println(err.toString()); } > > } catch(IOException err) { System.err.println(err.toString()); } } What happens if you ditch the req.getContentLength() approach (there are times when it will be -1 anyways), and do something like: BufferedReader reader = req.getReader(); StringBuffer contents = new StringBuffer(); String line = null; while((line = reader.readLine()) != null) { contents.append(line); } System.out.println(contents); (Later we'll worry about the writing -- first make sure you're reading the entire contents). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Hi, I should have thought that as a general principle it's not a good idea to try to store the response in a byte array. I recently worked on a piece of code that did just that (worse, actually, it then copied the array into a String). Sooner or later a really big upload will blow up the application. Reading and writing a byte at a time (with appropriate buffering) requires a bit more ingenuity, especially when you're searching for things like boundary strings in the response, but it's the only way to remove any constraint on upload size. Chris Walker -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 30 September 2003 19:30 To: Tomcat Users List Subject: RE: Tomcat sucks at receiving large messages Howdy, >public void service(HttpServletRequest req, HttpServletResponse res) { > BufferedReader reader = req.getReader(); > try { >char [] charArr = new char[req.getContentLength()]; >reader.read(charArr); >String str = new String(charArr); > >try { > File f = new File("servlet.out"); > PrintWriter out = new PrintWriter(new FileWriter(f)); > out.print(str); > out.flush(); > out.close(); >} catch(IOException err { System.err.println(err.toString()); } > > } catch(IOException err) { System.err.println(err.toString()); } >} What happens if you ditch the req.getContentLength() approach (there are times when it will be -1 anyways), and do something like: BufferedReader reader = req.getReader(); StringBuffer contents = new StringBuffer(); String line = null; while((line = reader.readLine()) != null) { contents.append(line); } System.out.println(contents); (Later we'll worry about the writing -- first make sure you're reading the entire contents). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Howdy, >public void service(HttpServletRequest req, HttpServletResponse res) { > BufferedReader reader = req.getReader(); > try { >char [] charArr = new char[req.getContentLength()]; >reader.read(charArr); >String str = new String(charArr); > >try { > File f = new File("servlet.out"); > PrintWriter out = new PrintWriter(new FileWriter(f)); > out.print(str); > out.flush(); > out.close(); >} catch(IOException err { System.err.println(err.toString()); } > > } catch(IOException err) { System.err.println(err.toString()); } >} What happens if you ditch the req.getContentLength() approach (there are times when it will be -1 anyways), and do something like: BufferedReader reader = req.getReader(); StringBuffer contents = new StringBuffer(); String line = null; while((line = reader.readLine()) != null) { contents.append(line); } System.out.println(contents); (Later we'll worry about the writing -- first make sure you're reading the entire contents). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Here it is: public class AdapterServlet extends HttpServlet { public void service(HttpServletRequest req, HttpServletResponse res) { BufferedReader reader = req.getReader(); try { char [] charArr = new char[req.getContentLength()]; reader.read(charArr); String str = new String(charArr); try { File f = new File("servlet.out"); PrintWriter out = new PrintWriter(new FileWriter(f)); out.print(str); out.flush(); out.close(); } catch(IOException err { System.err.println(err.toString()); } } catch(IOException err) { System.err.println(err.toString()); } } -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 30, 2003 8:34 AM To: Tomcat Users List Subject: RE: Tomcat sucks at receiving large messages Howdy, Perhaps if you share your servlet which writes the message body to a file, we could help you write a better servlet ;) Yoav Shapira Millennium ChemInformatics >-Original Message- >From: Stewart, Daniel J [mailto:[EMAIL PROTECTED] >Sent: Tuesday, September 30, 2003 11:31 AM >To: [EMAIL PROTECTED] >Subject: Tomcat sucks at receiving large messages > > >When receiving a HTTP 1.0 POST with a 10kbyte payload, my doPost() >method writes the message body to a file. The file is the right size, >but my data is nulled out (set to 0) after correctly receiving about >2kbytes. > >In frustration, I set the Connector bufferSize parameter to "100", >to discover that I could now receive about 4kbytes of my 10kbyte >message. I then proceeded to write my own Java server to receive the >message and write it to a file, and it works just fine. > >I am at a loss - can anyone suggest what could be causing this problem? > >Vitals: >Tomcat version: 4.1.27 >Tomcat configuration: Out-of-the-box (except for my app's WEB-INF) >OS: solaris 2.9 >My servlet skill level: medium-low > > > >- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat sucks at receiving large messages
Howdy, Perhaps if you share your servlet which writes the message body to a file, we could help you write a better servlet ;) Yoav Shapira Millennium ChemInformatics >-Original Message- >From: Stewart, Daniel J [mailto:[EMAIL PROTECTED] >Sent: Tuesday, September 30, 2003 11:31 AM >To: [EMAIL PROTECTED] >Subject: Tomcat sucks at receiving large messages > > >When receiving a HTTP 1.0 POST with a 10kbyte payload, my doPost() >method writes the message body to a file. The file is the right size, >but my data is nulled out (set to 0) after correctly receiving about >2kbytes. > >In frustration, I set the Connector bufferSize parameter to "100", >to discover that I could now receive about 4kbytes of my 10kbyte >message. I then proceeded to write my own Java server to receive the >message and write it to a file, and it works just fine. > >I am at a loss - can anyone suggest what could be causing this problem? > >Vitals: >Tomcat version: 4.1.27 >Tomcat configuration: Out-of-the-box (except for my app's WEB-INF) >OS: solaris 2.9 >My servlet skill level: medium-low > > > >- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]