Thank you for both of your answers. I'm using a custom Dockerfile as I install Puppet Server from our repositories. It's pretty similar though. I just add bits specific to our infrastracture.
Now, to be fair, saying 2 minutes is oversimplifying as that's a slightly high average. The fastest I've seen has been around 1 minutes and 10 seconds. That is with a free node (usually I fit two pods per node). In multiple occasions I can see that time going over the two minutes though. Going down to 30-45 seconds would be a bless to be honest. For sure I can give it a try. Regarding memory, that's not a problem as, like you said, there's a workaround. In any way, I'm aware of memory usage improvement in both, Puppet Server and JRuby. Right now I'm using memory oversized virtual machines to be able to run the server without crashes. The same machine that holds a Puppet Server with 4 instances in our production deployment, can't hold 2 pods with 2 instances even with this workaround. They share less data between threads so it's kinda expected, but it forces us to have slightly bulkier pods instead of smaller ones. Not a big deal, although annoying. Thank you both again for your answers. Very useful. On Monday, May 11, 2020 at 7:22:29 PM UTC+2, Justin Stoller wrote: > > The biggest single improvement you're going to get is moving from 5.x to > the latest 6.x. We started ahead-of-time (AOT) compiling the clojure code > to Java in the 6.x series as well as properly supporting Java 11. I think > the times Michael is quoting is for the 6.x release and I'd hope you'd see > start up times go from 1-2 minutes to 30-45 seconds. > > With Java 11 support also comes better support for running w/in containers > from Java itself. I don't know if that could be a factor, as there's work > around on Java 8[1]. Incorrect memory settings either on the container > itself or the other containers running on the VM could obviously affect > performance & start up. > <https://developers.redhat.com/blog/2017/03/14/java-inside-docker/> > > Ensuring enough entropy available is also probably the biggest cause of > slow start up time that isn't container specific. > > HTH, > Justin > > PS. One more thing in the more esoteric realm (but kinda like that > snapshotting you mentioned) that I've looked into is Java's AppCDS > loading[2]. We don't utilize it at the package level or internally that I > know of and last I looked (JDK 10, iirc) it had some bugs that prevented us > from using it. But might be usable in later JDKs with the latest Puppet > Server if you wanted to investigate. In a similar vein there's some JRuby > pre-compilation steps that I've tried to look into but I don't think > they're baked enough now. My dream is to see us AOT the core Puppet Ruby > code into JRuby bytecode and the JRuby and clojure code into JVM byte code > and then use AppCDS to load it all on start up, but I think we're a ways > off on that. > > 1. https://developers.redhat.com/blog/2017/03/14/java-inside-docker/ > 2. https://blog.codefx.org/java/application-class-data-sharing/ > > On Mon, May 11, 2020 at 8:57 AM Michael Smith <michae...@puppet.com > <javascript:>> wrote: > >> Are you using a Puppet Server container from >> https://hub.docker.com/r/puppet/puppetserver? If not, it'd be helpful to >> know what your Dockerfile is doing. >> >> I'm not sure there's been a lot of tuning done for the Puppet Server >> container we publish. The Puppet Server process itself usually starts in >> 15-30s (last time I looked at it), and that's mostly compiling and >> initializing Clojure library code. We periodically look at ways to improve >> this (GraalVM should be great once it's complete enough, but Puppet Server >> might need architectural changes to take advantage of it), but I'm not >> aware of any simple options. >> >> However 2 minutes sounds like a long time, so there are likely other >> things happening that could be sped up. If I start an instance locally, I >> see it taking about 40 seconds from invoking 'docker run' to being ready to >> handle requests: 10-15s in startup scripts, about 10s for the Clojure >> interpreter to start up, and 15-20s starting various services. There aren't >> simple toggles to improve any of those that I'm aware of. >> >> On Mon, May 11, 2020 at 7:10 AM David Moreno García <david...@gmail.com >> <javascript:>> wrote: >> >>> Hello, >>> >>> I'm using Puppet Server (Open Source 5.3.6) in a container environment >>> (Kubernetes 1.18). In this deployment, I'm scaling the pool of containers >>> taking into account the load (number of catalog compilations). I create new >>> containers when the load is high and I remove containers when they are not >>> needed. >>> >>> My problem with this approach is that the process is not as fast as I >>> would like. The Puppet Server, with 2 instances, takes around 2 minutes to >>> be ready. On top of that, the first compilation is slower. This means that >>> if I need a new container to process an incoming request, I will have to >>> wait a minimum of 2 minutes plus the extra time required to compile that >>> first catalog. >>> >>> Here I want to focus in the startup time for the Puppet Server. Would >>> there be any option to improve this time? >>> >>> I've been taking a look into CRIU <https://criu.org/Main_Page>, which >>> is not really an option. >>> >>> Any idea is welcome. Also, as this question is a bit abstract as right >>> now, feel free to ask for any specific detail about the deployment. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppe...@googlegroups.com <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-dev/b726a428-5b76-42f7-90be-9763348b6626%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/puppet-dev/b726a428-5b76-42f7-90be-9763348b6626%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppe...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-dev/CABy1mMKnzTeGmG-k_CXqdBQVQL27jG9AxZSMHDqfBOetY9nDiQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/puppet-dev/CABy1mMKnzTeGmG-k_CXqdBQVQL27jG9AxZSMHDqfBOetY9nDiQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/d001f7a2-a01e-40a7-abc2-ff269ebe2a05%40googlegroups.com.
