Re: Weird problem: slow upload via Manager
On 06.07.2018 19:49, Chris Cheshire wrote: On Jul 6, 2018, at 12:56 PM, James H. H. Lampert wrote: Forgive the top-post. The reason why this particular case of an extremely slow Manager upload sticks out is because we've done, by now, hundreds of uploads of this same WAR file (or earlier versions of it), via Manager, on over a dozen different installations, most of them AS/400s, and none of the others have taken even half this long. The problem might be deeper than tomcat then. +1 Have you ruled out IO problems writing the WAR? Is the HD/SSD failing and trying to resolve bad blocks? Is the network device failing causing dropped packets? Has another process on the machine run amok and causing IO contention, or paging issues? With the information available so far, it is a bit hard to make a qualified guess. If you can do this, and you have a choice of WARs to upload to the same system, and as a first simple way of getting more information, I would try uploading e.g. a WAR that is 50% or 25% or 10% the size of that original one, and see if the times are consistent with your 50 MB WAR. And do this several times, to see if the times are consistent for the same size, or they vary widely over time. Uploading another WAR will use exactly the same steps and elements as your first try, varying only one item at a time : the size of the upload. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weird problem: slow upload via Manager
> On Jul 6, 2018, at 12:56 PM, James H. H. Lampert > wrote: > > Forgive the top-post. > > The reason why this particular case of an extremely slow Manager upload > sticks out is because we've done, by now, hundreds of uploads of this same > WAR file (or earlier versions of it), via Manager, on over a dozen different > installations, most of them AS/400s, and none of the others have taken even > half this long. > The problem might be deeper than tomcat then. Have you ruled out IO problems writing the WAR? Is the HD/SSD failing and trying to resolve bad blocks? Is the network device failing causing dropped packets? Has another process on the machine run amok and causing IO contention, or paging issues? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weird problem: slow upload via Manager
Forgive the top-post. The reason why this particular case of an extremely slow Manager upload sticks out is because we've done, by now, hundreds of uploads of this same WAR file (or earlier versions of it), via Manager, on over a dozen different installations, most of them AS/400s, and none of the others have taken even half this long. On 7/6/18, 12:54 AM, André Warnier (tomcat) wrote: - when a browser POSTs a file, the content is first Base64-encoded, which typically increases the size in bytes by some 30-50%. Of course the receiving server has to Base64-decode the received data, to restore the original file. - it is not unusual for encoding/decoding code to process all the data in memory, as a block. This is usually the fastest way, until some treshold is reached where the data does not fit anymore in memory all at once. And then there can be a sudden dramatic slowdown, as either there is some frantic allocation of additional memory, or swapping taking place. You might want to have a look at tomcat's JVM garbage collection statistics while such an upload is going on. - also, whatever is POSTed to a webserver, is usually first accumulated somewhere, until the POST is complete. And then the whole POST is processed, parsed into parts, decoded if need be, and made available to the web application. - if this all happens via HTTP, it is also quite possible that there is some firewall/virus scanning going on, which would add another round of decoding and examining the POSTed data, which may also be quite sensitive to the size of things. What I mean is that there is a lot more going on behind the scenes, as compared to a simple file transfer via FTP or so. And if each step introduce some additional overhead depending on the original POST size, you may have something that looks exponentially (or asymptotically) worse in the end. Like 10 MB = 10s., 40 MB = 40s., 50 MB = 50s., 51 MB = 10 minutes. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weird problem: slow upload via Manager
On 06.07.2018 01:23, James H. H. Lampert wrote: Earlier this week, on a customer AS/400 installation (Tomcat 7.0.67), we experienced the slowest WAR file upload we've ever encountered: several HOURS to install a roughly 100M WAR file (we customarily increase the max-file-size and max-request-size in manager/WEB-INF/web.xml from 50M to 500M). This on a system in which a 10M FTP download typically takes anywhere from 15 seconds to a minute or so. Has anybody ever heard of anything like this happening? Could there be something in the customer's firewall that's slowing down what Manager uses to transfer WAR files (I'm guessing either HTTP POST or PUT)? Any suggestions on alternate ways of getting WAR files onto the box? Not about that, but maybe some basic info : - a browser does not natively do PUTs, so I guess POST it is - when a browser POSTs a file, the content is first Base64-encoded, which typically increases the size in bytes by some 30-50%. Of course the receiving server has to Base64-decode the received data, to restore the original file. - it is not unusual for encoding/decoding code to process all the data in memory, as a block. This is usually the fastest way, until some treshold is reached where the data does not fit anymore in memory all at once. And then there can be a sudden dramatic slowdown, as either there is some frantic allocation of additional memory, or swapping taking place. You might want to have a look at tomcat's JVM garbage collection statistics while such an upload is going on. - also, whatever is POSTed to a webserver, is usually first accumulated somewhere, until the POST is complete. And then the whole POST is processed, parsed into parts, decoded if need be, and made available to the web application. - if this all happens via HTTP, it is also quite possible that there is some firewall/virus scanning going on, which would add another round of decoding and examining the POSTed data, which may also be quite sensitive to the size of things. What I mean is that there is a lot more going on behind the scenes, as compared to a simple file transfer via FTP or so. And if each step introduce some additional overhead depending on the original POST size, you may have something that looks exponentially (or asymptotically) worse in the end. Like 10 MB = 10s., 40 MB = 40s., 50 MB = 50s., 51 MB = 10 minutes. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weird problem: slow upload via Manager
If you are not remote, can you just copy them to the IFS folder? On 6 July 2018 at 00:23, James H. H. Lampert wrote: > Earlier this week, on a customer AS/400 installation (Tomcat 7.0.67), we > experienced the slowest WAR file upload we've ever encountered: several > HOURS to install a roughly 100M WAR file (we customarily increase the > max-file-size and max-request-size in manager/WEB-INF/web.xml from 50M to > 500M). This on a system in which a 10M FTP download typically takes > anywhere from 15 seconds to a minute or so. > > Has anybody ever heard of anything like this happening? Could there be > something in the customer's firewall that's slowing down what Manager uses > to transfer WAR files (I'm guessing either HTTP POST or PUT)? > > Any suggestions on alternate ways of getting WAR files onto the box? > > -- > James H. H. Lampert > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Weird problem: slow upload via Manager
Earlier this week, on a customer AS/400 installation (Tomcat 7.0.67), we experienced the slowest WAR file upload we've ever encountered: several HOURS to install a roughly 100M WAR file (we customarily increase the max-file-size and max-request-size in manager/WEB-INF/web.xml from 50M to 500M). This on a system in which a 10M FTP download typically takes anywhere from 15 seconds to a minute or so. Has anybody ever heard of anything like this happening? Could there be something in the customer's firewall that's slowing down what Manager uses to transfer WAR files (I'm guessing either HTTP POST or PUT)? Any suggestions on alternate ways of getting WAR files onto the box? -- James H. H. Lampert - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
