Re: completely automated (for real) Let's Encrypt on embedded Tomcat
On 10/7/2020 10:12 AM, Garret Wilson wrote: … But anyway, let me tell you the idea I had this morning. In a way, you hinted at it in your reply. Why do I need to use S3 as a store if my application is running on AWS, and AWS already has the AWS Certificate Manager which already manages an SSL certificate with renewal! In essence the AWS Certificate Manager is the "data store/state" like S3, and I don't even need to call Let's Encrypt. Darn, it turns out AWS doesn't allow me to directly use AWS Certificate Manager certificates directly in my application. You cannot install your ACM certificate or your private ACM Private CA certificate directly on your AWS based website or application. (https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html) I guess it's back to Let's Encrypt using the original approach then. I've already purchased a book on the intricacies of SSL certificates and I have more on my shopping list. I'll start working on this and get back to you with questions. Cheers, Garret
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/7/20 13:12, Garret Wilson wrote: > As always thanks for the discussion, Chris. More replies and a new idea > below: > > On 10/6/2020 2:45 PM, Christopher Schultz wrote: >> … >> What if your Docker container would just run certbot on launch? > > But then I'm back to being a sysadmin, because the Docker container is > like a little OS and I have to set up the OS, update it, install certbot > if needed (based on the OS version!), ensure it's the certbot that has > the bugfixes I want and behaves as I expect, install Tomcat on the OS, > etc. I have to set up certbot with systemd or or whatever to run > periodically. All that stuff goes into my Dockerfile. So I'm really > doing the same thing as I would on a VM, except that Docker makes it > easier to reproduce. But it's conceptually not much different than being > sysadmin on a VM, then freezing the VM and duplicating that. Building a Dockerfile makes you a sysadmin. If you were deploying a WAR file into Elastic Beanstalk I might have more sympathy for your situation. >> If you build the keystore in-memory, I'm not exactly sure what you'd >> need to do in order to get Tomcat to bounce the SSLSocketFactory in that >> way. > > OK, I'll look into it. But it boggles my mind that an SSLSocketFactory > would have a harder time using actual bytes of a certificate than it > would loading it from disk. I didn't say that. I said that Tomcat's reload() method re-reads the configuration. If you use Tomcat embedded, then there is no configuration, and therefore reload() has nothing to reload. You will have to see what needs to be done, there. Perhaps you can re-set the SSLSocketFactory and then call reload() which will re-build the SSLContext, etc. and all will be well. I'm just not making any promises. > Because once it loads it from disk, it has > the bytes in memory, no? So the only problem would be the API—whether it > was built to unnecessarily require disk storage, or if it allows a > "hook" to provide the bytes at the point in the logic between > loading-from-disk and using-the-cert. Right. >>> I have no idea what a PEM-encoded DER file is, but I'll certainly learn. >> This: >> >> -BEGIN CERTIFICATE- >> stuff >> -END CERTIFICATE- >> > Ooh, that stuff. Yeah, I've definitely used that. I just never knew what > it was. I never really got into the deep murky depths of certificates, > but I guess now I'll have, seeing as that no one else in the last 20 > years has really made this easy. > >>> I still don't get why files have to be involved. I pulled a DER file >>> from S3. >> Sorry. I think of that as a "file". > > Ah, there's an important distinction to be made here. As developers we > often say "file" when we talk about a bunch of bytes that has some > coherent format — a JPEG file, an Ogg Vorbis file, a JSON file. But (and > the specs only really started making this clear around the time XML > became a spec, as least as far as I know) from another view it's only a > file if it's stored in a file system. (I'll explain why below.) So > really we can talk about an "XML document", for example, that may never > be stored in a file — we may construct an XML DOM tree completely in > memory and pass it to some other method. > >> >> So your web server is S3. > > Ah, no! > > S3 is a database. It could be PostgreSQL or Cassandra. It's a data > store. In particular it's a key-value store. It stores things. It has a > feature that allows you to set a configuration so that some web server > will automatically serve the BLOBs from the S3 data store, along with > some metadata. But is the web server "part of" S3? I'll bet not. S3 has a web interface, so it's definitely a web server. You can say it's a key-value store (which it is), but because you fetch objects from it using HTTP GET, it's also a web server. > Surely there is some pool of workers that pull data from the S3 data > store and serves the data. I could probably write my own web server > to do the same thing. But AWS makes it transparent; it's part of the > infrastucture. I just provide the data and declaratively tell it what > I want served. Then it is served. I don't have to configure a server. > There is some server in the cloud. > > But there's actually another step! Since I'm serving the site via > CloudFront, the CloudFront layer (servers around the world in different > countries!) actually connects to the web site serving the S3 data and > /copies it to CloudFront/! So my data is actually being served to the > end user by some CloudFront server, in some country closest to the user. > Is this Tomcat? Is it NGINX? Is it Apache? I don't know. Do I care what > it is? No. In fact, I'd rather not know, because I want to configure the > distribution and serving /independent from any server implementation/. CloudFront is a caching web server, just like if you were to use httpd + mod_cache. S3 is the origin server and CF is the caching reverse proxy. > So my web server is
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
As always thanks for the discussion, Chris. More replies and a new idea below: On 10/6/2020 2:45 PM, Christopher Schultz wrote: … What if your Docker container would just run certbot on launch? But then I'm back to being a sysadmin, because the Docker container is like a little OS and I have to set up the OS, update it, install certbot if needed (based on the OS version!), ensure it's the certbot that has the bugfixes I want and behaves as I expect, install Tomcat on the OS, etc. I have to set up certbot with systemd or or whatever to run periodically. All that stuff goes into my Dockerfile. So I'm really doing the same thing as I would on a VM, except that Docker makes it easier to reproduce. But it's conceptually not much different than being sysadmin on a VM, then freezing the VM and duplicating that. If you build the keystore in-memory, I'm not exactly sure what you'd need to do in order to get Tomcat to bounce the SSLSocketFactory in that way. OK, I'll look into it. But it boggles my mind that an SSLSocketFactory would have a harder time using actual bytes of a certificate than it would loading it from disk. Because once it loads it from disk, it has the bytes in memory, no? So the only problem would be the API—whether it was built to unnecessarily require disk storage, or if it allows a "hook" to provide the bytes at the point in the logic between loading-from-disk and using-the-cert. I have no idea what a PEM-encoded DER file is, but I'll certainly learn. This: -BEGIN CERTIFICATE- stuff -END CERTIFICATE- Ooh, that stuff. Yeah, I've definitely used that. I just never knew what it was. I never really got into the deep murky depths of certificates, but I guess now I'll have, seeing as that no one else in the last 20 years has really made this easy. I still don't get why files have to be involved. I pulled a DER file from S3. Sorry. I think of that as a "file". Ah, there's an important distinction to be made here. As developers we often say "file" when we talk about a bunch of bytes that has some coherent format — a JPEG file, an Ogg Vorbis file, a JSON file. But (and the specs only really started making this clear around the time XML became a spec, as least as far as I know) from another view it's only a file if it's stored in a file system. (I'll explain why below.) So really we can talk about an "XML document", for example, that may never be stored in a file — we may construct an XML DOM tree completely in memory and pass it to some other method. So your web server is S3. Ah, no! S3 is a database. It could be PostgreSQL or Cassandra. It's a data store. In particular it's a key-value store. It stores things. It has a feature that allows you to set a configuration so that some web server will automatically serve the BLOBs from the S3 data store, along with some metadata. But is the web server "part of" S3? I'll bet not. Surely there is some pool of workers that pull data from the S3 data store and serves the data. I could probably write my own web server to do the same thing. But AWS makes it transparent; it's part of the infrastucture. I just provide the data and declaratively tell it what I want served. Then it is served. I don't have to configure a server. There is some server in the cloud. But there's actually another step! Since I'm serving the site via CloudFront, the CloudFront layer (servers around the world in different countries!) actually connects to the web site serving the S3 data and /copies it to CloudFront/! So my data is actually being served to the end user by some CloudFront server, in some country closest to the user. Is this Tomcat? Is it NGINX? Is it Apache? I don't know. Do I care what it is? No. In fact, I'd rather not know, because I want to configure the distribution and serving /independent from any server implementation/. So my web server is something on some CloudFront deployment in some country. For all I know different CloudFront local deployments use different server types. I don't know and I don't care. ELB ~= CloudFront, right? Sort of. It is equivalent in that 1) it is the direct endpoint connection with the user, and 2) I can configure it to use the AWS managed certificates. If CloudFront handles your SSL for you, why not let ELB do it for you in this context? That's what I'm doing now! And that's what I would prefer doing. And that's what I would need to do for a large application! But the problem (going back to the motivation for all this) is that the ELB costs a lot of money. I don't absolutely need an ELB for a small app that I'm testing, or an app that will have 10 users, or an app that I don't care that crashes and immediately gets restarted. Why should I pay $16 or more per month for an ELB if I don't need it? When I have five little apps I want to deploy? Think about the thousands of little apps people want to deploy around the world. Is
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
On 10/6/20 2:48 PM, Christopher Schultz wrote: Thanks for mentioning LEGO. Any time I've been mentioning certbot, you can replace that with $your-favorite-acme-client. You're welcome. LEGO definitely cut my Gordian Knot on that particular project, wherein Certbot absolutely, positively, refused to work. The other AWS installations either have httpd handling the https front-ending, or they have a load-balancer handling it, with Amazon itself providing the certs. So I'm always likely to suggest it as a second choice (and one that appears to play much more nicely with Tomcat) if certbot won't do it in a given situation. -- JHHL - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
James, On 10/5/20 19:59, James H. H. Lampert wrote: > I'm coming into this conversation late, so what I say could be > completely irrelevant, but when I recently set up an independent (i.e., > not behind httpd) Tomcat server on one of our AWS EC2 instances, and > could not get certbot to function at all, to save my life, I ended up > using something called "LEGO." Thanks for mentioning LEGO. Any time I've been mentioning certbot, you can replace that with $your-favorite-acme-client. > It *does* require one to shut the Tomcat server down during the > renewal process (because it has to take over the port briefly), but > it also *does* play nicely with a Tomcat server that's doing its own > SSL. You *should* be able to do this without stopping Tomcat, but it might end up complicating other things. If you have a reverse proxy server, this is trivial to avoid. If you are binding Tomcat directly to port 80, this is not so easy. Another option is to use DNS-based authentication where your web server isn't involved. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/5/20 19:45, Garret Wilson wrote: > On 10/5/2020 2:42 PM, Christopher Schultz wrote: >> … >> Sure, it can contain S3 credentials and you can pick-up your key and >> certificate (or, better yet, the whole keystore) there, but at that >> point you have "moved" the problem outside of Tomcat, right? > > No, not at all. The major problems are: > > 1. Generating the certificate automatically. > 2. Feeding the certificate to Tomcat automatically. > > If the "extra" thing I have to do is specify an environment variable > with the name of the S3 bucket to use as a certificate state, then that > is a teeny, tiny problem. That is not really even a problem. > > Can you imagine if in my spring boot application I could run it using > "java -jar my-app.jar --cert-work-bucket my-bucket" and it would just go > get a certificate automatically? What if your Docker container would just run certbot on launch? >> You can have a "certificate renewal service" that writes to S3. > > I want it built into my application as a module. You just include the > module, specify the domain name (and maybe a couple of other details > needed by RFC 8555, such as a contact email), and boom, it all happens > automatically. > > Why does it have to be more difficult than that? It shouldn't be. Fair enough. >> I suppose you could put that directly into Tomcat, but Tomcat is not >> likely to ship with an Amazon-specific feature built-into it. > > > Read my original email. I never intended to put this directly into > standalone Tomcat (although if you want to put it into Tomcat you're > welcome to). I want to use this with an application running on embedded > Tomcat. Spring Boot is a prime example. I'll just extend the Spring Boot > embedded Tomcat module and extend/modify that as needed. If that > requires modifying Tomcat code, fine. > >> Another idea would be to use embedded Tomcat (or, at your suggestion, >> Spring Boot) and fetch the keystore from some "standard" location of >> your choosing. Again, that would be (appropriately, IMO) outside Tomcat >> code. > > That was always the intention. In my introduction to my original email I > explained that the modern approach is moving away from something like > standalone Tomcat to self-contained executables that run their own > servers, whether embedded Tomcat or whatever. (I was late to the party, > and even two years ago I wasn't getting it.) I'm not sure I'd say "modern approach". It's certainly popular these days, for sure. >> 1. Does acme4j allow me to verify my certificate behind another port? >> (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by >> default. I'm still reading RFC 8555 to find out if the ACME server >> has to connect back on a certain port for verification.) >> 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 >> for Tomcat completely in the application without shelling out to >> openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will >> allow me to do that. >> This can be done, but it's non-trivial. For example, Tomcat contains >> code to package PEM-encoded DER files (good old OpenSSL-style =BEGIN >> CERTIFICATE= things) into an in-memory keystore to configure JSSE. >> It seems like it would be straightforward, but it turns out not to be in >> all cases. YMMV. > > Non-trivial as it may be, it /only needs to be done once/. If I have a > converter, then I can use it a thousand times. A million times. And > suddenly the deployment becomes a piece of cake. > > In reality, today's style of handling SSL is what matches your > description of "It seems like it would be straightforward, but it turns > out not to be …". So why do we keep doing all this difficult, manual, > tedious, not-trivial stuff to deploy certificates the hard way, when we > could put our efforts into a single no-trivial task of making a > converter so that Tomcat can use the Let's Encrypt certificates > directly? We do it once. It's hard, but then everything else is easy. I > don't get why we want to spend another decade doing it the hard way when > we can spend one year on a different hard task and then do SSL the easy > way for the other nine. > > (The frustration isn't directed at you. It's just in general in software > development I see the industry—why does the most basic of things have to > be so difficult?) > >> 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? >> If it's a file on the disk, it's easy: just use the path. > > Can I pass it in memory? If not, why not? How is memory less accessible > than a file? Yes, you can pass it in-memory, but you have to arrange to load the key material from somewhere into memory. And sometimes the key material can be in a surprising series of formats. >>> Chris, where can I get more information on the latter questions about >>> getting this certificate to Tomcat once I have it? >> This mailing list is a good place to start (and likely
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
I'm coming into this conversation late, so what I say could be completely irrelevant, but when I recently set up an independent (i.e., not behind httpd) Tomcat server on one of our AWS EC2 instances, and could not get certbot to function at all, to save my life, I ended up using something called "LEGO." It *does* require one to shut the Tomcat server down during the renewal process (because it has to take over the port briefly), but it also *does* play nicely with a Tomcat server that's doing its own SSL. -- James H. H. Lampert - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
On 10/5/2020 2:42 PM, Christopher Schultz wrote: … Sure, it can contain S3 credentials and you can pick-up your key and certificate (or, better yet, the whole keystore) there, but at that point you have "moved" the problem outside of Tomcat, right? No, not at all. The major problems are: 1. Generating the certificate automatically. 2. Feeding the certificate to Tomcat automatically. If the "extra" thing I have to do is specify an environment variable with the name of the S3 bucket to use as a certificate state, then that is a teeny, tiny problem. That is not really even a problem. Can you imagine if in my spring boot application I could run it using "java -jar my-app.jar --cert-work-bucket my-bucket" and it would just go get a certificate automatically? You can have a "certificate renewal service" that writes to S3. I want it built into my application as a module. You just include the module, specify the domain name (and maybe a couple of other details needed by RFC 8555, such as a contact email), and boom, it all happens automatically. Why does it have to be more difficult than that? It shouldn't be. I suppose you could put that directly into Tomcat, but Tomcat is not likely to ship with an Amazon-specific feature built-into it. Read my original email. I never intended to put this directly into standalone Tomcat (although if you want to put it into Tomcat you're welcome to). I want to use this with an application running on embedded Tomcat. Spring Boot is a prime example. I'll just extend the Spring Boot embedded Tomcat module and extend/modify that as needed. If that requires modifying Tomcat code, fine. Another idea would be to use embedded Tomcat (or, at your suggestion, Spring Boot) and fetch the keystore from some "standard" location of your choosing. Again, that would be (appropriately, IMO) outside Tomcat code. That was always the intention. In my introduction to my original email I explained that the modern approach is moving away from something like standalone Tomcat to self-contained executables that run their own servers, whether embedded Tomcat or whatever. (I was late to the party, and even two years ago I wasn't getting it.) 1. Does acme4j allow me to verify my certificate behind another port? (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by default. I'm still reading RFC 8555 to find out if the ACME server has to connect back on a certain port for verification.) 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 for Tomcat completely in the application without shelling out to openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will allow me to do that. This can be done, but it's non-trivial. For example, Tomcat contains code to package PEM-encoded DER files (good old OpenSSL-style =BEGIN CERTIFICATE= things) into an in-memory keystore to configure JSSE. It seems like it would be straightforward, but it turns out not to be in all cases. YMMV. Non-trivial as it may be, it /only needs to be done once/. If I have a converter, then I can use it a thousand times. A million times. And suddenly the deployment becomes a piece of cake. In reality, today's style of handling SSL is what matches your description of "It seems like it would be straightforward, but it turns out not to be …". So why do we keep doing all this difficult, manual, tedious, not-trivial stuff to deploy certificates the hard way, when we could put our efforts into a single no-trivial task of making a converter so that Tomcat can use the Let's Encrypt certificates directly? We do it once. It's hard, but then everything else is easy. I don't get why we want to spend another decade doing it the hard way when we can spend one year on a different hard task and then do SSL the easy way for the other nine. (The frustration isn't directed at you. It's just in general in software development I see the industry—why does the most basic of things have to be so difficult?) 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? If it's a file on the disk, it's easy: just use the path. Can I pass it in memory? If not, why not? How is memory less accessible than a file? Chris, where can I get more information on the latter questions about getting this certificate to Tomcat once I have it? This mailing list is a good place to start (and likely finish). Of course I really super-appreciate the help on this mailing list, and I'll be asking lots more questions. But I also don't want to ask things you've already answered elsewhere. I thought sure in one of your ApacheCon 2019 presentations you mentioned you had made more progress than the ApacheCon 2018 slides, such as auto-reloading or something. If there is further documentation let me know. … Go back to my presentation on Let's Encrypt and you'll see how to use openssl to convert to a keystore if that's what you want. I want
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/5/20 12:21, Garret Wilson wrote: > Thank you so much for replying, Chris. Responses below. > > On 10/5/2020 8:53 AM, Christopher Schultz wrote: >> Microservices won't work the way you want with Let's Encrypt. You have >> two options: >> >> 1. Hit Let's Encrypt every time you launch a new instance of the >> microservice to deploy a new certificate >> >> 2. Handle the certificate provisioning elsewhere (e.g. ELB) >> >> #1 just won't work. LE won't re-issue a certificate more frequently than >> every 6 weeks or something like that. So that really leaves you with #2. > > It's good to know about the six-week limit, but you discarded #1 too > quickly. Can't the microservice simply store the credentials in S3 or > one of a hundred other data stores? (Note that I care less about > "microservices" as such at the moment. I just want a turnkey deployment > of a single application for now. But the idea is the same.) Sure, it can contain S3 credentials and you can pick-up your key and certificate (or, better yet, the whole keystore) there, but at that point you have "moved" the problem outside of Tomcat, right? You can have a "certificate renewal service" that writes to S3. I suppose you could put that directly into Tomcat, but Tomcat is not likely to ship with an Amazon-specific feature built-into it. I could imagine a LifecycleListener which has been written with this kind of thing in mind. (I'm not sure that a LifecycleListener would be able to intervene early enough in the process of a non-embedded Tomcat startup to do this, btw; just an idea.) Another idea would be to use embedded Tomcat (or, at your suggestion, Spring Boot) and fetch the keystore from some "standard" location of your choosing. Again, that would be (appropriately, IMO) outside Tomcat code. >> What you really want is for the orchestrator to provide the certificate >> and key to the nodes as they come on-line. > > Look these "orchestrators" are a configuration nightmare at the moment. > They end up being worse than my just configuring a CentOS machine from > scratch. Plus I have to pay for all those extra services. Read > https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and > try not to shudder. > >> So instead of trying to get LE to work with Tomcat (which does work, but >> requires some care and feeding), maybe we should try to get Tomcat to >> load its crypto material from other places. > > There is already a Java library (https://github.com/shred/acme4j) for > talking to Let's Encrypt. It sounds like it does everything I need. I'll > need to investigate more, but here are my initial doubts: > > 1. Does acme4j allow me to verify my certificate behind another port? > (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by > default. I'm still reading RFC 8555 to find out if the ACME server > has to connect back on a certain port for verification.) > 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 > for Tomcat completely in the application without shelling out to > openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will > allow me to do that. This can be done, but it's non-trivial. For example, Tomcat contains code to package PEM-encoded DER files (good old OpenSSL-style =BEGIN CERTIFICATE= things) into an in-memory keystore to configure JSSE. It seems like it would be straightforward, but it turns out not to be in all cases. YMMV. > 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? If it's a file on the disk, it's easy: just use the path. > Chris, where can I get more information on the latter questions about > getting this certificate to Tomcat once I have it? This mailing list is a good place to start (and likely finish). >> One way to do that would be with e.g. Amazon's key storage service. I'm >> not familiar with that. I know it can store various types of keys; not >> sure about certificates and if we can pull a private key out of it. > > I think your idea of storing this stuff elsewhere is a good first > stepping stone to get to where I want to go. > > Just to get the ball rolling, I could manually run some Let's Encrypt > client, and then store the certificate in S3. Then the first component I > would write would be steps #2 and #3 above. I am already familiar with > the AWS Java library, so I could quickly figure out how to pull the > certificates off S3. But I need your help in finding out how to convert > them to PKCS12 (or whatever; this is new territory for me) on the fly > and feed them to the embedded Tomcat. Go back to my presentation on Let's Encrypt and you'll see how to use openssl to convert to a keystore if that's what you want. Or, better yet, skip that step entirely and use the PEM-encoded DER files that Let's Encrypt already provides to you. You just have to work-out file-permissions issues. >> Honestly, your best bet would probably be to use ELB and just pay for >> it. You only pay for
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Thank you so much for replying, Chris. Responses below. On 10/5/2020 8:53 AM, Christopher Schultz wrote: Microservices won't work the way you want with Let's Encrypt. You have two options: 1. Hit Let's Encrypt every time you launch a new instance of the microservice to deploy a new certificate 2. Handle the certificate provisioning elsewhere (e.g. ELB) #1 just won't work. LE won't re-issue a certificate more frequently than every 6 weeks or something like that. So that really leaves you with #2. It's good to know about the six-week limit, but you discarded #1 too quickly. Can't the microservice simply store the credentials in S3 or one of a hundred other data stores? (Note that I care less about "microservices" as such at the moment. I just want a turnkey deployment of a single application for now. But the idea is the same.) What you really want is for the orchestrator to provide the certificate and key to the nodes as they come on-line. Look these "orchestrators" are a configuration nightmare at the moment. They end up being worse than my just configuring a CentOS machine from scratch. Plus I have to pay for all those extra services. Read https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and try not to shudder. So instead of trying to get LE to work with Tomcat (which does work, but requires some care and feeding), maybe we should try to get Tomcat to load its crypto material from other places. There is already a Java library (https://github.com/shred/acme4j) for talking to Let's Encrypt. It sounds like it does everything I need. I'll need to investigate more, but here are my initial doubts: 1. Does acme4j allow me to verify my certificate behind another port? (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by default. I'm still reading RFC 8555 to find out if the ACME server has to connect back on a certain port for verification.) 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 for Tomcat completely in the application without shelling out to openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will allow me to do that. 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? Chris, where can I get more information on the latter questions about getting this certificate to Tomcat once I have it? One way to do that would be with e.g. Amazon's key storage service. I'm not familiar with that. I know it can store various types of keys; not sure about certificates and if we can pull a private key out of it. I think your idea of storing this stuff elsewhere is a good first stepping stone to get to where I want to go. Just to get the ball rolling, I could manually run some Let's Encrypt client, and then store the certificate in S3. Then the first component I would write would be steps #2 and #3 above. I am already familiar with the AWS Java library, so I could quickly figure out how to pull the certificates off S3. But I need your help in finding out how to convert them to PKCS12 (or whatever; this is new territory for me) on the fly and feed them to the embedded Tomcat. Honestly, your best bet would probably be to use ELB and just pay for it. You only pay for data-transfer, so a dormant ELB costs you virtually nothing. Last month I deployed a test application on Elastic Beanstalk on a domain nobody knows about just to see how it worked. The ELB cost me $16 in a month with basically nobody using it! That adds up quickly. I have several little apps I want to toss up. See also my question https://serverfault.com/q/1036276 . Instead of spinning-up an EC2 instance for your service, maybe you should be looking at Lambda instead. You can probably get your costs down more that way than trying to eliminate the ELB. I want to drop either a self-contained JAR file or a Docker image somewhere, and have it immediately start running with SSL support, without my configuring a VM or running scripts. When I have a new version, I want to drop a new JAR or Docker image and have it automatically replace the other one. I don't want to maintain a VM. I have a target price of let's say $5/month for everything (although $3 would be better). Does AWS Lambda give me that? If so, please point me to the guides. Garret
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/4/20 14:04, Garret Wilson wrote: > Hi, everyone. I'm back already. (I had intended to leave the list to > focus my efforts elsewhere, but … here I am again.) > > I just realized there is a big SSL problem for small applications, and I > want to fix it. First a little review of where we are. > > Servlet containers are becoming less important and less desirable in > today's world, because we don't want to deploy and maintain some sort of > high-level container infrastructure (in the Java EE container sense, not > the Docker sense) just to deploy an application in it. Modern > distributed micrososervice applications have a bunch of > service/worker/agent application that are identical and redundant. You > spin up as many as you need; if some go down, you (or an orchestrator) > spins up others. Microservices won't work the way you want with Let's Encrypt. You have two options: 1. Hit Let's Encrypt every time you launch a new instance of the microservice to deploy a new certificate 2. Handle the certificate provisioning elsewhere (e.g. ELB) #1 just won't work. LE won't re-issue a certificate more frequently than every 6 weeks or something like that. So that really leaves you with #2. What you really want is for the orchestrator to provide the certificate and key to the nodes as they come on-line. This represents a bit of a security issue, but not any moreso than any node trying to connect to the orchestrator in the first place IMHO. So instead of trying to get LE to work with Tomcat (which does work, but requires some care and feeding), maybe we should try to get Tomcat to load its crypto material from other places. One way to do that would be with e.g. Amazon's key storage service. I'm not familiar with that. I know it can store various types of keys; not sure about certificates and if we can pull a private key out of it. Honestly, your best bet would probably be to use ELB and just pay for it. You only pay for data-transfer, so a dormant ELB costs you virtually nothing. Instead of spinning-up an EC2 instance for your service, maybe you should be looking at Lambda instead. You can probably get your costs down more that way than trying to eliminate the ELB. BTW, if your instance isn't running at all (e.g. the orchestrator has decided that zero nodes are required), what do clients contact to make that initial connection for the orchestrator to spin-up that first instance? > For this reason libraries like Spring Boot allow you to deploy your Java > application as a standalone JAR with embedded Tomcat. The JAR represents > the completely independent application. You just throw it on a node and > it runs and provides a web server or whatever. So we we should be able > to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and > it just runs. I found out it is far from that simple, and SSL is one of > the major problems. > > There seem to be two ways to get SSL support. On something like AWS > Elastic Beanstalk, you deploy a load balancer in front of your EC > instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure > SSL to the load balancer, spin up EC instances as needed (each running > your standalone JAR), and connect the load balancer to the EC instances, > all in a (sort of) automated fashion. But note that the SSL endpoint is > the load balancer, and the load balancer costs money! Even if you're > just running just a single standalone JAR instance requiring a single EC > instance, that load balancer sits there and drains cash. Significant > cash if you just want to run a little program with SSL support. > > What's the other option to deploy a standalone JAR? Configure an AWS EC > instance (or a VM with another provider), configure certbot, configure > Tomcat, save some files locally on the machine, etc. All this manual > work. I just want to run the standalone JAR! In short, if I have a > standalone program I want to run, I either have to configure and > maintain a VM like I did in the year 2000, or get into the nightmare of > Kubernetes-like orchestration with the endless configurations and/or the > high costs. > > I propose to create a module that integrates with embedded Tomcat that: > > 1. You indicate what domain you're hosting for (as part of the > application configuration or as an environment variable when > deployed, for example). > 2. When your application starts running, it automatically connects to > Let's Encrypt using RFC 8555 (or whatever is needed) and requests a > certificate, based upon the IP address it's running on. > 3. The module exposes the correct HTTP paths and/or connects to a > configured DNS as needed for validation. > 4. The module receives the certificates and caches them in memory or in > a temporary file as needed and provides them to Tomcat; Tomcat now > is serving using SSL/TLS. > 5. If the application dies, who cares? You start up another one. It > automatically does the same thing (on
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
https://github.com/AtomGraph/letsencrypt-tomcat On Sun, Oct 4, 2020 at 8:04 PM Garret Wilson wrote: > > Hi, everyone. I'm back already. (I had intended to leave the list to > focus my efforts elsewhere, but … here I am again.) > > I just realized there is a big SSL problem for small applications, and I > want to fix it. First a little review of where we are. > > Servlet containers are becoming less important and less desirable in > today's world, because we don't want to deploy and maintain some sort of > high-level container infrastructure (in the Java EE container sense, not > the Docker sense) just to deploy an application in it. Modern > distributed micrososervice applications have a bunch of > service/worker/agent application that are identical and redundant. You > spin up as many as you need; if some go down, you (or an orchestrator) > spins up others. > > For this reason libraries like Spring Boot allow you to deploy your Java > application as a standalone JAR with embedded Tomcat. The JAR represents > the completely independent application. You just throw it on a node and > it runs and provides a web server or whatever. So we we should be able > to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and > it just runs. I found out it is far from that simple, and SSL is one of > the major problems. > > There seem to be two ways to get SSL support. On something like AWS > Elastic Beanstalk, you deploy a load balancer in front of your EC > instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure > SSL to the load balancer, spin up EC instances as needed (each running > your standalone JAR), and connect the load balancer to the EC instances, > all in a (sort of) automated fashion. But note that the SSL endpoint is > the load balancer, and the load balancer costs money! Even if you're > just running just a single standalone JAR instance requiring a single EC > instance, that load balancer sits there and drains cash. Significant > cash if you just want to run a little program with SSL support. > > What's the other option to deploy a standalone JAR? Configure an AWS EC > instance (or a VM with another provider), configure certbot, configure > Tomcat, save some files locally on the machine, etc. All this manual > work. I just want to run the standalone JAR! In short, if I have a > standalone program I want to run, I either have to configure and > maintain a VM like I did in the year 2000, or get into the nightmare of > Kubernetes-like orchestration with the endless configurations and/or the > high costs. > > I propose to create a module that integrates with embedded Tomcat that: > > 1. You indicate what domain you're hosting for (as part of the > application configuration or as an environment variable when > deployed, for example). > 2. When your application starts running, it automatically connects to > Let's Encrypt using RFC 8555 (or whatever is needed) and requests a > certificate, based upon the IP address it's running on. > 3. The module exposes the correct HTTP paths and/or connects to a > configured DNS as needed for validation. > 4. The module receives the certificates and caches them in memory or in > a temporary file as needed and provides them to Tomcat; Tomcat now > is serving using SSL/TLS. > 5. If the application dies, who cares? You start up another one. It > automatically does the same thing (on another machine or wherever it > is running) and the application is running SSL/TLS. It's that > simple. You don't need to run certbot. You don't need to manually > copy files on the system. You don't even need to know where the > application is going to run. You just need an executable JAR with > this new module, and you run it. Done. > 6. (Many variations exists where multiple JARs are running but one is > the "leader" for Let's Encrypt, and they communicate and share the > cashed certificate until the node dies. Or there are variations > using Docker. The first step is the radical one, and then all sorts > of possibilities open up.) > > From glancing over the Let's Encrypt docs and having had hands-on > experience embedding Tomcat, that seems completely doable to me. And I'm > ready to start. > > But first, what work has been done in this area already? I'm aware of > Chris' slides from 2018, but those techniques require some combination > of certbot, keytool, non-embedded Tomcat, symlinks,OS scripts, manually > file system manipulation, etc. I think at ApacheCon 2019 Chris mentioned > some more work has been done on this, but I don't recall where it was. > > Please point me to the latest work and ideas for Tomcat+Let's Encrypt so > that I don't spend two months doing something that is already been done, > or before I find out it is impossible. > > As it stands I want fully automated SSL/TLS configuration just by > running a standalone JAR, and I don't see that existing anywhere. I'm > not
