Glad to help. It's not surprising that this would cause trouble. It's certainly not something I would ever try in a production situation. Even the individual who commented on the Oracle forum [1] stated:
Short answer, restart app server after upgrading JDK to be safe. If you want to be on the edge and live dangerously, try it without restarting. Production is not really where you want to be "on the edge and live dangerously." Justin [1] https://community.oracle.com/tech/developers/discussion/4084875/upgrading-jvm-while-application-server-running On Thu, Jan 12, 2023 at 1:31 PM Vilius Šumskas <vilius.sums...@rivile.lt> wrote: > Just leaving this for others. > > My theory is confirmed, if you have upgraded OpenJDK under running live > Artemis broker and didn't restart it afterwards your > restartEmbeddedWebServer() command will fail. Hawtio console will be > stopped but won't automatically re-start. > > Reproducible steps under Rocky Linux 9: > 1. Login into running Hawtio console, click on Operations tab and Exexute > restartEmbeddedWebServer(). The user gets logged out but after few seconds > console is available again, which is correct behaviour. > 2. Now login into SSH on the server running Artemis broker and run "dnf > update java-17-openjdk-headless" or "dnf downgrade > java-17-openjdk-headless". > 3. Login back into running Hawtio console, click on Operations tab and > Exexute restartEmbeddedWebServer() again. The console is now down and never > comes up. > > The moral of the story - never upgrade JDK if you are unable to restart > Artemis broker at the same time. Else you will have problems later, for > example when reloading Hawtio console for TLS certificate change. > > Thank you for your help Justin! > > -- > Best Regards, > Vilius > > -----Original Message----- > From: Vilius Šumskas <vilius.sums...@rivile.lt> > Sent: Thursday, January 12, 2023 12:48 PM > To: users@activemq.apache.org > Subject: RE: reloading TLS certificate for Jolokia > > Yes, the upgrade is removing older Java version, however this is only for > minor versions. In theory internets say that it should depend on software > if that's safe or not > https://community.oracle.com/tech/developers/discussion/4084875/upgrading-jvm-while-application-server-running > For example in Jenkins case it's not safe > https://stackoverflow.com/questions/35758284/does-tomcat-need-to-be-restarted-after-a-jdk-is-updated > For Artemis I did this multiple time and it always seemed fine. > > I see that now it's more complicated to bring everything up, so I guess I > will make a maintenance window on production and restart the broker > completely. > > I will test my theory regarding JDK upgrade and console restart a little > bit later and report the findings. > > -- > Vilius > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: Wednesday, January 11, 2023 11:20 PM > To: users@activemq.apache.org > Subject: Re: reloading TLS certificate for Jolokia > > > Could it be that our Artemis instance is still using older JDK which > exist only in memory, and console "inherits" that by trying to start > itself, and then fails? > > I don't have any experience with such a use-case so it's hard to say. The > "console" isn't fundamentally different from any other part of the broker. > It's all just Java classes. Therefore, I wouldn't expect such a failure to > be isolated there. If the rest of the broker is working fine I would expect > the embedded web server to work fine as well. > > Are you _removing_ the version of Java that the broker is using during the > upgrade or are you just installing a new version alongside the existing > version? If the former, that seems dangerous in my opinion. > > > Is Artemis supposed to reload management.xml automatically... > > No. The contents of management.xml are not reloaded automatically. > > > ...do we have a chicken-egg problem now; cannot enable JMX because JMX > > is > down? > > You do have a chicken-egg problem with using JMX remotely. However, you > could still theoretically use it locally through a direct JVM connection. > > You also still have the option of using management messages [1]. You can > find an example of their use in the examples/features/standard/management > directory shipped with the broker. > > > Justin > > [1] > > https://activemq.apache.org/components/artemis/documentation/latest/management.html#using-management-message-api > > > > On Wed, Jan 11, 2023 at 2:49 PM Vilius Šumskas <vilius.sums...@rivile.lt> > wrote: > > > No, I don't see any of these messages in the log. I have posted full > > log what I see. > > > > This is production system so getting threadump is not so easy, but I > > have decided to test this on our testing environment which runs newest > 2.27.1. > > Running restartEmbeddedWebServer() via Hawtio console on that > > environment worked just fine. > > > > I have remembered one thing though. We have upgraded OpenJDK on our > > production environment some time ago, but never restarted Artemis. > > Usually this doesn't produce any issues, but if we decide to restart > > Artemis after JDK upgrade it usually produces some errors about java > > binaries cannot be found. As if Artemis was still loaded on older JDK > > version and stopping it unloads these non-existing older JDK binaries > from memory. > > Could it be that our Artemis instance is still using older JDK which > > exist only in memory, and console "inherits" that by trying to start > > itself, and then fails? > > > > Anyway, I see that JMX Remote is disabled by default. I have enabled > > JMX connector in management.xml, but I don't see port 1099 coming up. > > Is Artemis supposed to reload management.xml automatically, or do we > > have a chicken-egg problem now; cannot enable JMX because JMX is down? > > > > -- > > Vilius > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: Wednesday, January 11, 2023 6:46 PM > > To: users@activemq.apache.org > > Subject: Re: reloading TLS certificate for Jolokia > > > > Those exceptions look internal to Jetty. I wouldn't expect them to > > necessarily be a problem for the restart process. Perhaps it was > > handling a request at the moment you tried to restart it. > > > > Do you ever see any of these messages in the log: > > > > Stopped embedded web server > > > > or > > > > Embedded web server not restarted in x milliseconds > > > > or > > > > Embedded web server restart failed > > > > If not, could you perhaps get a thread dump? > > > > In any event, if you want to start the embedded web server you can do > > so using JMX directly (e.g. via JConsole). Just invoke the > > startEmbeddedWebServer operation on the ActiveMQServerControl MBean. > > > > > > Justin > > > > On Wed, Jan 11, 2023 at 6:41 AM Vilius Šumskas > > <vilius.sums...@rivile.lt> > > wrote: > > > > > Well, I have found " restartEmbeddedWebServer()" and tried to use > > > that from the console, however now the console went down and doesn't > come up. > > > The logs say: https://p.defau.lt/?C6ssFYW0JSJiZX1yszP0yQ > > > > > > Jolokia API is also down now. Is there a way to start embedded web > > > server without restarting the broker? > > > > > > -- > > > Vilius > > > > > > -----Original Message----- > > > From: Vilius Šumskas <vilius.sums...@rivile.lt> > > > Sent: Wednesday, January 11, 2023 1:55 PM > > > To: users@activemq.apache.org > > > Subject: reloading TLS certificate for Jolokia > > > > > > Hi, > > > > > > since because of https://issues.apache.org/jira/browse/ARTEMIS-3117 > > > ActiveMQ Artemis doesn‘t reload TLS certificates automatically > > > anymore I‘m wondering, what is the correct way to reload > > > certificates for embedded Jolokia Console? > > > > > > I have reloaded keystore for acceptors via console using reload() on > > > „artemis-ssl“ acceptor, but I just cannot figure out a way to reload > > > it for console itself. > > > > > > -- > > > Best Regards, > > > > > > Vilius Šumskas > > > Rivile > > > IT manager > > > > > > > > >
