Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Hello, thanks for the information received thus far and trying to assist
me. With regards to the Integer problem:
public class test
{
public static void main (String []args)
{
int j = 10; int y = 2147483646;
int result = j + y;
System.out.println( result );
}
}
C:\Documents and Settings\David\Desktopjava test
-2147483640
It overflows when they are added together. I haven't checked where the
content length is pulled from a String but if it does cause a crash,
then it is handled internally, as I don't receive errors/log messages
due to this issue. I may have to check out the source for Tomcat or
Catalina (whichever does this handling) and see what goes on.
Now, if you were using an old version of mod_jk (which you didn't
mention), it's possible that the Content-Length header is being
corrupted. Since you have a filter chain, can you print the (String)
value of the Content-Length header before processing begins? That would
be helpful.
What is this mod_jk you speak of? Does it come with Tomcat or do I need
to set it up? I'll begin googling in a second for research, but a
straight forward answer would be wonderful. Also, with regards to
printing out the content length, the following happens whenever a large
(2.XGB+) file is uploaded:
I have a filter that displays all headers of the request, */ALL/ *of
this was output when I submitted that file _/*ONE */_time (it's like
Tomcat leaves the request hanging, never sets up the content-length/
etc, and loops until someone/something decides to handle this request).
I have trimmed the output substantially, because the same output is
issued approx every second or two for ONE request indefinitely (I left
my desk for about 15 minutes, and this same thing was still going).
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: host: localhost:8084
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: user-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept: */*
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-language: en-us,en;q=0.5
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-encoding: gzip,deflate
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: keep-alive: 300
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: connection: keep-alive
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: referer: http://localhost:8084/RosenthalFiles/
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: cookie: JSESSIONID=B9F5C30C710AAAFA048E85FE1B559668;
GUEST_LANGUAGE_ID=en_US; COOKIE_SUPPORT=true
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: if-modified-since: Mon, 13 Aug 2007 21:42:37 GMT
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: if-none-match: 1187041357000
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: host: localhost:8084
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: user-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept: */*
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-language: en-us,en;q=0.5
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-encoding: gzip,deflate
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: keep-alive: 300
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: connection: keep-alive
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: referer: http://localhost:8084/RosenthalFiles/
Aug 14, 2007 10:54:00 AM
com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing
INFO: cookie: JSESSIONID=B9F5C30C710AAAFA048E85FE1B559668;
GUEST_LANGUAGE_ID=en_US;
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
I have installed 6.0.14 now and the same problem persists. I am
starting to worry about our choice to use Java for this web application
project now... the client insisted that we used .NET framework or
'Microsoft' products if you will, but limitations arise. I just won't
be able to stand the 'we told you so' that is sure to happen if we can't
find a workaround with existing deployment options that allow huge
uploads. He never mentioned he wanted 2+ GB uploads anyways, but again
that's probably our fault for not asking, and then testing limitations
before designing/selecting frameworks, language, etc.
Caldarale, Charles R wrote:
From: David Hesson [mailto:[EMAIL PROTECTED]
Subject: Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
I haven't checked where the content length is pulled from
a String but if it does cause a crash, then it is handled
internally
Here's the code of interest:
public int getContentLength() {
long length = getContentLengthLong();
if (length Integer.MAX_VALUE) {
return (int) length;
}
return -1;
}
(from the 6.0.14 version of org/apache/coyote/Request.java). The real
problem is with the Servlet API spec, which defines the return type of
getContentLength() as an int. Until that problem is resolved, you'll
have to avoid using that API. Note that internally, Tomcat tracks the
size with a long, not an int, although as Rainer has pointed out, some
problems existed in older levels that have been addressed in 6.0.14.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
David Hesson
Software Engineer
NuRelm, Inc.
http://www.nurelm.com
Toll Free: 1-877-268-7356 ext. 207
Local: 1-724-430-0490 ext. 207
RE: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads I have installed 6.0.14 now and the same problem persists. Can you post your servlet/JSP code (if it's not excessively large)? IIRC, you're using Commons FileUpload 1.2; is that correct? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Currently, I am posting to another jsp page that just prints out the method used (should say post if all goes well). I have a Servlet that I use to write files that works on everything under 2GB, but on large uploads, the thread is never hit, thus I began posting to a tmp.jsp page until I figure out why tmp.jsp is never reached. Yes, lib snapshot of directory: /lib commons-fileupload-1.2.jar * ---**index.jsp---* [EMAIL PROTECTED] contentType=text/html% [EMAIL PROTECTED] pageEncoding=UTF-8% !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01 Transitional//EN http://www.w3.org/TR/html4/loose.dtd; html head meta http-equiv=Content-Type content=text/html; charset=UTF-8 titleJSP Page/title /head body form action=%=request.getContextPath ()%/tmp.jsp method=post enctype=multipart/form-data onsubmit=javascript: displayProgress(); input type=text name=fileTitle id=fileTitle value=File Title onclick=if (this.value == this.defaultValue) this.value='';/ input type=file name=file id=file/ input type=submit id=uploadButton value=Upload/ /form /body /html *---**tmp.jsp**---* [EMAIL PROTECTED] contentType=text/html% [EMAIL PROTECTED] pageEncoding=UTF-8% !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01 Transitional//EN http://www.w3.org/TR/html4/loose.dtd; html head meta http-equiv=Content-Type content=text/html; charset=UTF-8 titleJSP Page/title /head body %=request.getMethod ()% /body /html Caldarale, Charles R wrote: From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads I have installed 6.0.14 now and the same problem persists. Can you post your servlet/JSP code (if it's not excessively large)? IIRC, you're using Commons FileUpload 1.2; is that correct? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
David Hesson wrote: I have installed 6.0.14 now and the same problem persists. I am starting to worry about our choice to use Java for this web application project now... the client insisted that we used .NET framework or 'Microsoft' products if you will, but limitations arise. I just won't be able to stand the 'we told you so' that is sure to Do you know if .NOT will let upload these giant files? D - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
No clue, I guess that is an assumption on my behalf. If it doesn't, I'd be delighted to know that. Will do some research shortly. David kerber wrote: David Hesson wrote: I have installed 6.0.14 now and the same problem persists. I am starting to worry about our choice to use Java for this web application project now... the client insisted that we used .NET framework or 'Microsoft' products if you will, but limitations arise. I just won't be able to stand the 'we told you so' that is sure to Do you know if .NOT will let upload these giant files? D - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
From: David kerber [mailto:[EMAIL PROTECTED] Do you know if .NOT will let upload these giant files? Definitely not on 32-bit (see http://support.microsoft.com/kb/295626). The address space maxes out at 1 Gbyte, and IIS has to buffer the bytes in RAM before ASP.NET can process them. http://aspnetresources.com/articles/dark_side_of_file_uploads.aspx is an interesting read for all flavours - notably, don't expect to be able to do this sensibly any earlier than IIS7. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Wow, I wonder if I'm going to have to do some kind of Applet to get this to work properly? Another solution is forwarding requests to some FTP app, but they want progress bars. If I were to compile and run all this on a 64 bit, do integers use 32 bit still or are they knocked up to 64 in Java? Glad to see .NET isn't much better off. Peter Crowther wrote: From: David kerber [mailto:[EMAIL PROTECTED] Do you know if .NOT will let upload these giant files? Definitely not on 32-bit (see http://support.microsoft.com/kb/295626). The address space maxes out at 1 Gbyte, and IIS has to buffer the bytes in RAM before ASP.NET can process them. http://aspnetresources.com/articles/dark_side_of_file_uploads.aspx is an interesting read for all flavours - notably, don't expect to be able to do this sensibly any earlier than IIS7. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207
RE: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads If I were to compile and run all this on a 64 bit, do integers use 32 bit still Recompilation for different platforms is never needed for Java code - that's one of its advantages. However, a Java int is always 32 bits, regardless of platform. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David, David Hesson wrote: Hello, thanks for the information received thus far and trying to assist me. With regards to the Integer problem: [snip] C:\Documents and Settings\David\Desktopjava test -2147483640 It overflows when they are added together. True. This is integer addition overflow, not a negative number resulting from reading a too-large string of digits into a too-small int value. I haven't checked where the content length is pulled from a String but if it does cause a crash, then it is handled internally, as I don't receive errors/log messages due to this issue. That is true. One problem with Tomcat (really the servlet spec) is that the ServetRequest.getContentLength method returns a 32-bit int instead of a 64-bit long, which means that you need to read the Content-Length as a string and convert it to a long if you want to be able to correctly read Content-Length headers with numbers bigger than 2^32 - 1. What is this mod_jk you speak of? Does it come with Tomcat or do I need to set it up? I'll begin googling in a second for research, but a straight forward answer would be wonderful. mod_jk is an Apache httpd module that can be used to connect Apache httpd to Tomcat. IIRC, older versions of mod_jk read the Content-Length header as a 32-bit signed int and therefore broke uploads bigger than 2GB. Also, with regards to printing out the content length, the following happens whenever a large (2.XGB+) file is uploaded: I have a filter that displays all headers of the request, */ALL/ *of this was output when I submitted that file _/*ONE */_time (it's like Tomcat leaves the request hanging, never sets up the content-length/ etc, and loops until someone/something decides to handle this request). I have trimmed the output substantially, because the same output is issued approx every second or two for ONE request indefinitely (I left my desk for about 15 minutes, and this same thing was still going). [snip] INFO: if-modified-since: Mon, 13 Aug 2007 21:42:37 GMT if-modified-since on a file upload? That's odd... Aug 14, 2007 10:54:00 AM com.nurelm.rosenthalftp.filter.ContentLengthFilter doBeforeProcessing INFO: host: localhost:8084 This looks like a second request. Are you sure this was a single request? It might help to print out the requested URI before you print the headers. It might help you sort-out the header output you're seeing. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGybgD9CaO5/Lv0PARAiwsAKCMrDk2Jtq7wGbg6+mnLD7CU1apyQCgliB0 2fGMClmmWVAGWPCousnfRlM= =6dIQ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David, David Hesson wrote: I have installed 6.0.14 now and the same problem persists. I am starting to worry about our choice to use Java for this web application project now... I know for a fact that Tomcat 5.5 can accept bigger-than-2GB uploads. There must be something else going on, as evidenced by the fact that your code isn't being called. Can you toss-out your JSPs and try a vanilla servlet? Just something that can dump out the headers and maybe spit out progress to a log file for how many bytes it's receiving? Just print something like got another 10MB everytime you hit 10MB. the client insisted that we used .NET framework or 'Microsoft' products if you will, but limitations [arose]. Just 'cause I'm curious: what were those limitations? It's pretty ballsy to ignore a semi-major customer requirement like that. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGybwf9CaO5/Lv0PARApssAKCIdQ41r/1fwy7+UHjAx/BA6xi00gCbBK6E hWDIvpvlq88xQceW98nPOGY= =pGFC -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Hi,
a few days ago i had the same question/problem. i found:
http://www.motobit.com/help/scptutl/pa98.htm
If this is correct (my own limited tests confirmed it) you're effectivly
limited to 2GB uploads using HTTP and it's not tomcat's problem alone -
if at all.
kind regards,
Markus
David Hesson wrote:
I have installed 6.0.14 now and the same problem persists. I am
starting to worry about our choice to use Java for this web application
project now... the client insisted that we used .NET framework or
'Microsoft' products if you will, but limitations arise. I just won't
be able to stand the 'we told you so' that is sure to happen if we can't
find a workaround with existing deployment options that allow huge
uploads. He never mentioned he wanted 2+ GB uploads anyways, but again
that's probably our fault for not asking, and then testing limitations
before designing/selecting frameworks, language, etc.
Caldarale, Charles R wrote:
From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Re:
Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
I haven't checked where the content length is pulled from
a String but if it does cause a crash, then it is handled
internally
Here's the code of interest:
public int getContentLength() {
long length = getContentLengthLong();
if (length Integer.MAX_VALUE) {
return (int) length;
}
return -1;
}
(from the 6.0.14 version of org/apache/coyote/Request.java). The real
problem is with the Servlet API spec, which defines the return type of
getContentLength() as an int. Until that problem is resolved, you'll
have to avoid using that API. Note that internally, Tomcat tracks the
size with a long, not an int, although as Rainer has pointed out, some
problems existed in older levels that have been addressed in 6.0.14.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Markus, Markus Schiegl wrote: a few days ago i had the same question/problem. i found: http://www.motobit.com/help/scptutl/pa98.htm If this is correct (my own limited tests confirmed it) you're effectivly limited to 2GB uploads using HTTP and it's not tomcat's problem alone - if at all. Now that's /very/ interesting. If your browser won't upload your big 'old file, then maybe that's why the servlet (or JSP) never gets invoked. You might want to try using wget, which should be able to cram a ton of data down your servlet's throat. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGyb379CaO5/Lv0PARAm4uAJ4p8+poqcoBF0QSiirJ6gyyH8+jkwCeI+LU BOXPefXMjIrz7okqiFDW1R0= =9Omq -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Well I am a new developer (I was still in college when I began helping with this project) My boss told me the client would like to use .NET, and I got kind of excited because I haven't worked with .NET/ASP/C# for quite a bit, and I love the compiler, but he talked the client out of it (the client is personally known [can't really say friend as they don't get along too well] by my boss). When I say limitations arise, I meant that quite possibly even .NET could even have some types of issues with huge uploads, so I guess what I was saying is that 'limitations sometimes come with software programming, and thus just because we chose Java instead of .NET didn't mean .NET couldn't have had its own issues'. Servlets are never reached in the web application, only the filters are hit. Servlet calls seem to be getting skipped. Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David, David Hesson wrote: I have installed 6.0.14 now and the same problem persists. I am starting to worry about our choice to use Java for this web application project now... I know for a fact that Tomcat 5.5 can accept bigger-than-2GB uploads. There must be something else going on, as evidenced by the fact that your code isn't being called. Can you toss-out your JSPs and try a vanilla servlet? Just something that can dump out the headers and maybe spit out progress to a log file for how many bytes it's receiving? Just print something like got another 10MB everytime you hit 10MB. the client insisted that we used .NET framework or 'Microsoft' products if you will, but limitations [arose]. Just 'cause I'm curious: what were those limitations? It's pretty ballsy to ignore a semi-major customer requirement like that. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGybwf9CaO5/Lv0PARApssAKCIdQ41r/1fwy7+UHjAx/BA6xi00gCbBK6E hWDIvpvlq88xQceW98nPOGY= =pGFC -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David, David Hesson wrote: Servlets are never reached in the web application, only the filters are hit. Servlet calls seem to be getting skipped. Your filters are called, but not the servlet? That's odd. Can you post the code to your filters? Or disable as much as you can? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGycKq9CaO5/Lv0PARAuNCAKC95oyxjLA8yhZbNUzDClovVcfLmgCeMK5j t6OTwvSXY3pP59HkFxVP8tc= =qxAS -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Yes, I've seen problems with IE and Firefox uploading files 2GB (but
I haven't tested the latest versions). The browser either sends a
bogus Content-Length, or it doesn't send a request at all!
David, try your test JSP with the Opera browser. It seems to be able
to send large files.
--
Len
On 8/20/07, Markus Schiegl [EMAIL PROTECTED] wrote:
Hi,
a few days ago i had the same question/problem. i found:
http://www.motobit.com/help/scptutl/pa98.htm
If this is correct (my own limited tests confirmed it) you're effectivly
limited to 2GB uploads using HTTP and it's not tomcat's problem alone -
if at all.
kind regards,
Markus
David Hesson wrote:
I have installed 6.0.14 now and the same problem persists. I am
starting to worry about our choice to use Java for this web application
project now... the client insisted that we used .NET framework or
'Microsoft' products if you will, but limitations arise. I just won't
be able to stand the 'we told you so' that is sure to happen if we can't
find a workaround with existing deployment options that allow huge
uploads. He never mentioned he wanted 2+ GB uploads anyways, but again
that's probably our fault for not asking, and then testing limitations
before designing/selecting frameworks, language, etc.
Caldarale, Charles R wrote:
From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Re:
Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
I haven't checked where the content length is pulled from
a String but if it does cause a crash, then it is handled
internally
Here's the code of interest:
public int getContentLength() {
long length = getContentLengthLong();
if (length Integer.MAX_VALUE) {
return (int) length;
}
return -1;
}
(from the 6.0.14 version of org/apache/coyote/Request.java). The real
problem is with the Servlet API spec, which defines the return type of
getContentLength() as an int. Until that problem is resolved, you'll
have to avoid using that API. Note that internally, Tomcat tracks the
size with a long, not an int, although as Rainer has pointed out, some
problems existed in older levels that have been addressed in 6.0.14.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Thanks, will do. Len Popp wrote: Yes, I've seen problems with IE and Firefox uploading files 2GB (but I haven't tested the latest versions). The browser either sends a bogus Content-Length, or it doesn't send a request at all! David, try your test JSP with the Opera browser. It seems to be able to send large files. -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Opera totally works. I just uploaded a 4.2GB file with it :) Thank you guys so much. Solution to uploading 2GB files was indeed not a Tomcat issue. The login page will now contain the following text: To upload files 2GB, here are a list of browsers... 1) Opera 2) ? :) Cheers and many thanks, David Len Popp wrote: Yes, I've seen problems with IE and Firefox uploading files 2GB (but I haven't tested the latest versions). The browser either sends a bogus Content-Length, or it doesn't send a request at all! David, try your test JSP with the Opera browser. It seems to be able to send large files. -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
You guys have no idea how happy I am. David Hesson wrote: Opera totally works. I just uploaded a 4.2GB file with it :) Thank you guys so much. Solution to uploading 2GB files was indeed not a Tomcat issue. The login page will now contain the following text: To upload files 2GB, here are a list of browsers... 1) Opera 2) ? :) Cheers and many thanks, David Len Popp wrote: Yes, I've seen problems with IE and Firefox uploading files 2GB (but I haven't tested the latest versions). The browser either sends a bogus Content-Length, or it doesn't send a request at all! David, try your test JSP with the Opera browser. It seems to be able to send large files. -- David Hesson Software Engineer NuRelm, Inc. http://www.nurelm.com Toll Free: 1-877-268-7356 ext. 207 Local: 1-724-430-0490 ext. 207 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
DAvid,
David Hesson wrote:
The
content-length has a maximum value of 2.x billion, which is right under
two gigabytes.
Is this a limit on commons-upload? The header itself can contain an
arbitrarily high number, so there's no inherent file-size limit on
uploads over HTTP.
A 2.xGB file will result in a negative content length
from integer overflow into the final, negative bit position.
I don't think so. Java doesn't overflow like C does. Attempting to read
an int that's too big results in an exception, not a negative result:
$ cat IntReadTest.java
public class IntReadTest
{
public static void main(String[] args)
{
System.out.println(Integer.parseInt(args[0]));
}
}
^D
$ javac IntReadTest
$ java IntReadTest 2147483647
2147483647
$ java IntReadTest 2147483648
Exception in thread main java.lang.NumberFormatException: For input
string: 2147483648
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:463)
at java.lang.Integer.parseInt(Integer.java:497)
at IntReadTest.main(IntReadTest.java:5)
$
Now, if you were using an old version of mod_jk (which you didn't
mention), it's possible that the Content-Length header is being
corrupted. Since you have a filter chain, can you print the (String)
value of the Content-Length header before processing begins? That would
be helpful.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGxFDy9CaO5/Lv0PARAtz1AJ9G5qHlz2n6nY2km68upW80z5OfOwCfTlo4
rDUomEV/r/L3L7DcjruMysc=
=tjSt
-END PGP SIGNATURE-
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
There were in fact bugs with requests 2GB in both Tomcat and mod_jk
which were fixed recently, as Rainer pointed out. (And some of the
code was using -1 if the content-length was too large.) Upgrading to
6.0.14 or the next 5.5 release should fix the problem.
There's no size limit on commons-fileupload as far as I can tell, but
I've never tried it with a huge file.
--
Len
On 8/16/07, Christopher Schultz [EMAIL PROTECTED] wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
DAvid,
David Hesson wrote:
The
content-length has a maximum value of 2.x billion, which is right under
two gigabytes.
Is this a limit on commons-upload? The header itself can contain an
arbitrarily high number, so there's no inherent file-size limit on
uploads over HTTP.
A 2.xGB file will result in a negative content length
from integer overflow into the final, negative bit position.
I don't think so. Java doesn't overflow like C does. Attempting to read
an int that's too big results in an exception, not a negative result:
$ cat IntReadTest.java
public class IntReadTest
{
public static void main(String[] args)
{
System.out.println(Integer.parseInt(args[0]));
}
}
^D
$ javac IntReadTest
$ java IntReadTest 2147483647
2147483647
$ java IntReadTest 2147483648
Exception in thread main java.lang.NumberFormatException: For input
string: 2147483648
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:463)
at java.lang.Integer.parseInt(Integer.java:497)
at IntReadTest.main(IntReadTest.java:5)
$
Now, if you were using an old version of mod_jk (which you didn't
mention), it's possible that the Content-Length header is being
corrupted. Since you have a filter chain, can you print the (String)
value of the Content-Length header before processing begins? That would
be helpful.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGxFDy9CaO5/Lv0PARAtz1AJ9G5qHlz2n6nY2km68upW80z5OfOwCfTlo4
rDUomEV/r/L3L7DcjruMysc=
=tjSt
-END PGP SIGNATURE-
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
I am dealing with a client who needs multi-gigabyte uploads (4GB+, whatever he wants really, and he says it is needed/required by the system.) Anyways, I currently upload a 4GB file to my Tomcat server, and it is set to simply post to a JSP with the file (please note that files 2GB work fine, forms are set up fine), and that JSP page is never reached. It simply looks like Tomcat is invoking the servlets/target of the request or this is because something hasn't been dealt with yet in the request to cause this action. My logs are all clean of errors, however, and the filter chain works fine (to some extent, it loops forever with this request because it is waiting for it to be dealt with). I receive log messages in beforeProcessing, process, and afterProcessing. However, the JSP is never hit, or if I post the file to a servlet, the servlet is never invoked. I have already tried configuring the connector to use maxPostSize=0, as well as setting the maxHttpHeaderSize variable. I am clearly at a loss of what to do. I also tried creating a Request Listener whenever requests are created but the request never gets a contentLength or contentType. The server simply begins looping the filter chain over and over and over and never hits my servlet/upload.jsp page. Any help would be greatly appreciated. David - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Hi, Have a look at the commons-fileupload [1], it should help you out. Ben [1] http://commons.apache.org/fileupload/ On 8/15/07, David Hesson [EMAIL PROTECTED] wrote: I am dealing with a client who needs multi-gigabyte uploads (4GB+, whatever he wants really, and he says it is needed/required by the system.) Anyways, I currently upload a 4GB file to my Tomcat server, and it is set to simply post to a JSP with the file (please note that files 2GB work fine, forms are set up fine), and that JSP page is never reached. It simply looks like Tomcat is invoking the servlets/target of the request or this is because something hasn't been dealt with yet in the request to cause this action. My logs are all clean of errors, however, and the filter chain works fine (to some extent, it loops forever with this request because it is waiting for it to be dealt with). I receive log messages in beforeProcessing, process, and afterProcessing. However, the JSP is never hit, or if I post the file to a servlet, the servlet is never invoked. I have already tried configuring the connector to use maxPostSize=0, as well as setting the maxHttpHeaderSize variable. I am clearly at a loss of what to do. I also tried creating a Request Listener whenever requests are created but the request never gets a contentLength or contentType. The server simply begins looping the filter chain over and over and over and never hits my servlet/upload.jsp page. Any help would be greatly appreciated. David - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
Completely sorry, details follows: System: Windows XP (Home I believe) 32 bit 2GB Memory on my system Web Application Details/ Other Details: JSF Framework (1.1?) Commons File Uploads 1.2 attempted to be used Tomcat 5.5.17 I'm going to say that Sun is my JVM vendor?? JVM is version 1.6 Tomcat is using these settings: -Xms512m -Xmx512m (I am not receiving PermGens/OutOfMemoryExceptions by any means) I'm compiling at a source level of 1.6 for the web application, the JVM vendor is SUN I do have somewhat of a clue as to why the upload itself is not functional when just trying to use the Commons File Upload to stream the file to a temporary location when large uploads are detected. The content-length has a maximum value of 2.x billion, which is right under two gigabytes. A 2.xGB file will result in a negative content length from integer overflow into the final, negative bit position. Other than that, I cannot explain why the servlet/.jsp target of the form post is not being hit and the filter chain calls filters over and over but won't go any further. After pressing submit, the page acts like nothing happened, and no error messages are generated. I'm fresh out of college so my lack of experience/knowing what you meant by JVM level kind of threw me off, I have JRE1.6 and JDK1.6 :( Here is what came out in Command Prompt issuing this command: java -version java version 1.6.0 Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode) Thanks once again - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
From: David Hesson [mailto:[EMAIL PROTECTED] Subject: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads Any help would be greatly appreciated. Should we guess the Tomcat version you're using, or would you like to tell us? It would also be helpful to know the platform you're running on, the OS, and the JVM vendor and level. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi-Gigabyte Uploads, Tomcat 2GB and higher uploads
There have been fixes for 2GB size uploads and downloads between June 10 and June 5. If we assume, that those fixes will help, you've got a coupe of options: - try with Tomcat 6.0.14, which already contains the fixes. This is a major update, but since you are already using Java 5+, you shoujld be able to do it in much less than a day. - we plan to tag a new 5.5 release very soon (end of the week or shortly after). If the tag looks good, you can expect a new 5.5 release in around 2-4 weeks. A 5.5.25 tarball will be available earlier (likely first half of next week), so you can already test it, although it will not be officially released at that time. - You can build your own 5.5 from the sources in the publicly available source code management system subversion. Of course we don't know, if there will be more problems related to libs and the platform with the big uploads and downloads, but at least we checked, that Tomcat itself handles the Content-Length headers correctly with the above mentioned fixes. In case you additionally combine Tomcat with Apache HTTPD or IIS via mod_jk/isapi redirector: the same problem (handling large file contents) has been fixed there recently. This is included in release 1.2.24 of this web server plugin. Regards, Rainer David Hesson wrote: Completely sorry, details follows: System: Windows XP (Home I believe) 32 bit 2GB Memory on my system Web Application Details/ Other Details: JSF Framework (1.1?) Commons File Uploads 1.2 attempted to be used Tomcat 5.5.17 I'm going to say that Sun is my JVM vendor?? JVM is version 1.6 Tomcat is using these settings: -Xms512m -Xmx512m (I am not receiving PermGens/OutOfMemoryExceptions by any means) I'm compiling at a source level of 1.6 for the web application, the JVM vendor is SUN I do have somewhat of a clue as to why the upload itself is not functional when just trying to use the Commons File Upload to stream the file to a temporary location when large uploads are detected. The content-length has a maximum value of 2.x billion, which is right under two gigabytes. A 2.xGB file will result in a negative content length from integer overflow into the final, negative bit position. Other than that, I cannot explain why the servlet/.jsp target of the form post is not being hit and the filter chain calls filters over and over but won't go any further. After pressing submit, the page acts like nothing happened, and no error messages are generated. I'm fresh out of college so my lack of experience/knowing what you meant by JVM level kind of threw me off, I have JRE1.6 and JDK1.6 :( Here is what came out in Command Prompt issuing this command: java -version java version 1.6.0 Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode) Thanks once again - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
