Re: File size truncated at 1.4GB during download from Tomcat WebApp
On Sun, Oct 4, 2020, 16:12 Mauro Tridici wrote: > > > > On 3 Oct 2020, at 19:32, calder wrote: > > > > On Sat, Oct 3, 2020, 11:43 Mauro Tridici wrote: > > > >> > >>> On 3 Oct 2020, at 17:03, calder wrote: > >>> > >>> On Sat, Oct 3, 2020, 09:58 calder wrote: > >>> > On Sat, Oct 3, 2020, 09:01 Mauro Tridici > wrote: > > > > > > > On 3 Oct 2020, at 15:14, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Mauro, > > > > On 10/3/20 08:47, Mauro Tridici wrote > > > > I’m struggling with the problem mentioned in this mail subject. > > When I try to download a 5GB sized file using two different > > > > > < snip > > > In your opinion, if the problem is related to the 32-bit overflow, is > > there something that I can do in order to solve this issue. > > Since I can’t modify any change to these two applications, I would > like > > to know if I can do something on the other sides. > > > > > This issue was reported to the IRODs team, they confirmed it, and > >> pushed a > fix. You should check your version. > > The issue, which is now closed. > > https://github.com/irods-contrib/irods-rest/issues/66 > > >>> > >>> I also see your involvement in this issue, which was marked as a Dup. > >>> > >>> https://github.com/irods-contrib/metalnx-web/issues/130 > >>> > >>> ... and updated here > >>> https://github.com/irods-contrib/metalnx-web/issues/143 > >> > >> > >> Hi Calder, > >> > >> thank you for your help and suggestions. > >> Unfortunately, I’m still experiencing the same problem despite I’m using > >> the last available version. > >> > >> Since I used also a different web application (not MetalNX) and I > noticed > >> the same problem, I was thinking that there is something to be changed > in > >> tomcat configuration. > >> I will try to start again from scentsratch. > >> > > > > Just to be sure ... did you check again to be sure you have the patch > > installed? I ask because two people who had the problem reported back > > that the patch fixed the issue for them. > > > > Reading the comments, they did find where an "int" was being used for > > "contentLength" in the code. > > > >> > > > > Yes, Calder, I checked it. > I also changed installation mode: I deployed MetalNX using docker-compose > (as indicate in the updated official page) > So, I think that also during installation latest version of MetalNX is > used. > > Now, my question is: why using two different web applications I obtained > the same behaviour? > Is there something to be changed in tomcat configuration? Another user, on > this forum, mentioned a “32bit problem”. > My virtual machine has a 64bit OS. I need to change it? Sorry for this > stupid question... > The "bit-ness" of the operating system doesn't influence the size (range) of Java integers ... also, an "int" is the same size in a 32bit JVM and 64bit JVM. You mention you have two different Java applications running on Tomcat ... do they both use IRODS? Or does one application use IRODS and the other application use some other library?
Re: File size truncated at 1.4GB during download from Tomcat WebApp
> On 3 Oct 2020, at 19:32, calder wrote: > > On Sat, Oct 3, 2020, 11:43 Mauro Tridici wrote: > >> >>> On 3 Oct 2020, at 17:03, calder wrote: >>> >>> On Sat, Oct 3, 2020, 09:58 calder wrote: >>> On Sat, Oct 3, 2020, 09:01 Mauro Tridici wrote: > > > On 3 Oct 2020, at 15:14, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Mauro, > > On 10/3/20 08:47, Mauro Tridici wrote > > I’m struggling with the problem mentioned in this mail subject. > When I try to download a 5GB sized file using two different > > < snip > In your opinion, if the problem is related to the 32-bit overflow, is > there something that I can do in order to solve this issue. > Since I can’t modify any change to these two applications, I would like > to know if I can do something on the other sides. > This issue was reported to the IRODs team, they confirmed it, and >> pushed a fix. You should check your version. The issue, which is now closed. https://github.com/irods-contrib/irods-rest/issues/66 >>> >>> I also see your involvement in this issue, which was marked as a Dup. >>> >>> https://github.com/irods-contrib/metalnx-web/issues/130 >>> >>> ... and updated here >>> https://github.com/irods-contrib/metalnx-web/issues/143 >> >> >> Hi Calder, >> >> thank you for your help and suggestions. >> Unfortunately, I’m still experiencing the same problem despite I’m using >> the last available version. >> >> Since I used also a different web application (not MetalNX) and I noticed >> the same problem, I was thinking that there is something to be changed in >> tomcat configuration. >> I will try to start again from scentsratch. >> > > Just to be sure ... did you check again to be sure you have the patch > installed? I ask because two people who had the problem reported back > that the patch fixed the issue for them. > > Reading the comments, they did find where an "int" was being used for > "contentLength" in the code. > >> Yes, Calder, I checked it. I also changed installation mode: I deployed MetalNX using docker-compose (as indicate in the updated official page) So, I think that also during installation latest version of MetalNX is used. Now, my question is: why using two different web applications I obtained the same behaviour? Is there something to be changed in tomcat configuration? Another user, on this forum, mentioned a “32bit problem”. My virtual machine has a 64bit OS. I need to change it? Sorry for this stupid question...
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
https://github.com/AtomGraph/letsencrypt-tomcat On Sun, Oct 4, 2020 at 8:04 PM Garret Wilson wrote: > > Hi, everyone. I'm back already. (I had intended to leave the list to > focus my efforts elsewhere, but … here I am again.) > > I just realized there is a big SSL problem for small applications, and I > want to fix it. First a little review of where we are. > > Servlet containers are becoming less important and less desirable in > today's world, because we don't want to deploy and maintain some sort of > high-level container infrastructure (in the Java EE container sense, not > the Docker sense) just to deploy an application in it. Modern > distributed micrososervice applications have a bunch of > service/worker/agent application that are identical and redundant. You > spin up as many as you need; if some go down, you (or an orchestrator) > spins up others. > > For this reason libraries like Spring Boot allow you to deploy your Java > application as a standalone JAR with embedded Tomcat. The JAR represents > the completely independent application. You just throw it on a node and > it runs and provides a web server or whatever. So we we should be able > to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and > it just runs. I found out it is far from that simple, and SSL is one of > the major problems. > > There seem to be two ways to get SSL support. On something like AWS > Elastic Beanstalk, you deploy a load balancer in front of your EC > instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure > SSL to the load balancer, spin up EC instances as needed (each running > your standalone JAR), and connect the load balancer to the EC instances, > all in a (sort of) automated fashion. But note that the SSL endpoint is > the load balancer, and the load balancer costs money! Even if you're > just running just a single standalone JAR instance requiring a single EC > instance, that load balancer sits there and drains cash. Significant > cash if you just want to run a little program with SSL support. > > What's the other option to deploy a standalone JAR? Configure an AWS EC > instance (or a VM with another provider), configure certbot, configure > Tomcat, save some files locally on the machine, etc. All this manual > work. I just want to run the standalone JAR! In short, if I have a > standalone program I want to run, I either have to configure and > maintain a VM like I did in the year 2000, or get into the nightmare of > Kubernetes-like orchestration with the endless configurations and/or the > high costs. > > I propose to create a module that integrates with embedded Tomcat that: > > 1. You indicate what domain you're hosting for (as part of the > application configuration or as an environment variable when > deployed, for example). > 2. When your application starts running, it automatically connects to > Let's Encrypt using RFC 8555 (or whatever is needed) and requests a > certificate, based upon the IP address it's running on. > 3. The module exposes the correct HTTP paths and/or connects to a > configured DNS as needed for validation. > 4. The module receives the certificates and caches them in memory or in > a temporary file as needed and provides them to Tomcat; Tomcat now > is serving using SSL/TLS. > 5. If the application dies, who cares? You start up another one. It > automatically does the same thing (on another machine or wherever it > is running) and the application is running SSL/TLS. It's that > simple. You don't need to run certbot. You don't need to manually > copy files on the system. You don't even need to know where the > application is going to run. You just need an executable JAR with > this new module, and you run it. Done. > 6. (Many variations exists where multiple JARs are running but one is > the "leader" for Let's Encrypt, and they communicate and share the > cashed certificate until the node dies. Or there are variations > using Docker. The first step is the radical one, and then all sorts > of possibilities open up.) > > From glancing over the Let's Encrypt docs and having had hands-on > experience embedding Tomcat, that seems completely doable to me. And I'm > ready to start. > > But first, what work has been done in this area already? I'm aware of > Chris' slides from 2018, but those techniques require some combination > of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually > file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned > some more work has been done on this, but I don't recall where it was. > > Please point me to the latest work and ideas for Tomcat+Let's Encrypt so > that I don't spend two months doing something that is already been done, > or before I find out it is impossible. > > As it stands I want fully automated SSL/TLS configuration just by > running a standalone JAR, and I don't see that existing anywhere. I'm > not
completely automated (for real) Let's Encrypt on embedded Tomcat
Hi, everyone. I'm back already. (I had intended to leave the list to focus my efforts elsewhere, but … here I am again.) I just realized there is a big SSL problem for small applications, and I want to fix it. First a little review of where we are. Servlet containers are becoming less important and less desirable in today's world, because we don't want to deploy and maintain some sort of high-level container infrastructure (in the Java EE container sense, not the Docker sense) just to deploy an application in it. Modern distributed micrososervice applications have a bunch of service/worker/agent application that are identical and redundant. You spin up as many as you need; if some go down, you (or an orchestrator) spins up others. For this reason libraries like Spring Boot allow you to deploy your Java application as a standalone JAR with embedded Tomcat. The JAR represents the completely independent application. You just throw it on a node and it runs and provides a web server or whatever. So we we should be able to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and it just runs. I found out it is far from that simple, and SSL is one of the major problems. There seem to be two ways to get SSL support. On something like AWS Elastic Beanstalk, you deploy a load balancer in front of your EC instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure SSL to the load balancer, spin up EC instances as needed (each running your standalone JAR), and connect the load balancer to the EC instances, all in a (sort of) automated fashion. But note that the SSL endpoint is the load balancer, and the load balancer costs money! Even if you're just running just a single standalone JAR instance requiring a single EC instance, that load balancer sits there and drains cash. Significant cash if you just want to run a little program with SSL support. What's the other option to deploy a standalone JAR? Configure an AWS EC instance (or a VM with another provider), configure certbot, configure Tomcat, save some files locally on the machine, etc. All this manual work. I just want to run the standalone JAR! In short, if I have a standalone program I want to run, I either have to configure and maintain a VM like I did in the year 2000, or get into the nightmare of Kubernetes-like orchestration with the endless configurations and/or the high costs. I propose to create a module that integrates with embedded Tomcat that: 1. You indicate what domain you're hosting for (as part of the application configuration or as an environment variable when deployed, for example). 2. When your application starts running, it automatically connects to Let's Encrypt using RFC 8555 (or whatever is needed) and requests a certificate, based upon the IP address it's running on. 3. The module exposes the correct HTTP paths and/or connects to a configured DNS as needed for validation. 4. The module receives the certificates and caches them in memory or in a temporary file as needed and provides them to Tomcat; Tomcat now is serving using SSL/TLS. 5. If the application dies, who cares? You start up another one. It automatically does the same thing (on another machine or wherever it is running) and the application is running SSL/TLS. It's that simple. You don't need to run certbot. You don't need to manually copy files on the system. You don't even need to know where the application is going to run. You just need an executable JAR with this new module, and you run it. Done. 6. (Many variations exists where multiple JARs are running but one is the "leader" for Let's Encrypt, and they communicate and share the cashed certificate until the node dies. Or there are variations using Docker. The first step is the radical one, and then all sorts of possibilities open up.) From glancing over the Let's Encrypt docs and having had hands-on experience embedding Tomcat, that seems completely doable to me. And I'm ready to start. But first, what work has been done in this area already? I'm aware of Chris' slides from 2018, but those techniques require some combination of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned some more work has been done on this, but I don't recall where it was. Please point me to the latest work and ideas for Tomcat+Let's Encrypt so that I don't spend two months doing something that is already been done, or before I find out it is impossible. As it stands I want fully automated SSL/TLS configuration just by running a standalone JAR, and I don't see that existing anywhere. I'm not prepared to pay AWS for a load balancer just to run a little app, and I got tired of manual Linux setup and scripts and general sysadmin work around 20 years ago. It's the cloud. It should act like the cloud. Garret
