Just leaving this for others.
My theory is confirmed, if you have upgraded OpenJDK under running live Artemis
broker and didn't restart it afterwards your restartEmbeddedWebServer() command
will fail. Hawtio console will be stopped but won't automatically re-start.
Reproducible steps under Rocky Linux 9:
1. Login into running Hawtio console, click on Operations tab and Exexute
restartEmbeddedWebServer(). The user gets logged out but after few seconds
console is available again, which is correct behaviour.
2. Now login into SSH on the server running Artemis broker and run "dnf update
java-17-openjdk-headless" or "dnf downgrade java-17-openjdk-headless".
3. Login back into running Hawtio console, click on Operations tab and Exexute
restartEmbeddedWebServer() again. The console is now down and never comes up.
The moral of the story - never upgrade JDK if you are unable to restart Artemis
broker at the same time. Else you will have problems later, for example when
reloading Hawtio console for TLS certificate change.
Thank you for your help Justin!
--
Best Regards,
Vilius
-----Original Message-----
From: Vilius Šumskas <vilius.sums...@rivile.lt>
Sent: Thursday, January 12, 2023 12:48 PM
To: users@activemq.apache.org
Subject: RE: reloading TLS certificate for Jolokia
Yes, the upgrade is removing older Java version, however this is only for minor
versions. In theory internets say that it should depend on software if that's
safe or not
https://community.oracle.com/tech/developers/discussion/4084875/upgrading-jvm-while-application-server-running
For example in Jenkins case it's not safe
https://stackoverflow.com/questions/35758284/does-tomcat-need-to-be-restarted-after-a-jdk-is-updated
For Artemis I did this multiple time and it always seemed fine.
I see that now it's more complicated to bring everything up, so I guess I will
make a maintenance window on production and restart the broker completely.
I will test my theory regarding JDK upgrade and console restart a little bit
later and report the findings.
--
Vilius
-----Original Message-----
From: Justin Bertram <jbert...@apache.org>
Sent: Wednesday, January 11, 2023 11:20 PM
To: users@activemq.apache.org
Subject: Re: reloading TLS certificate for Jolokia
> Could it be that our Artemis instance is still using older JDK which
exist only in memory, and console "inherits" that by trying to start itself,
and then fails?
I don't have any experience with such a use-case so it's hard to say. The
"console" isn't fundamentally different from any other part of the broker.
It's all just Java classes. Therefore, I wouldn't expect such a failure to be
isolated there. If the rest of the broker is working fine I would expect the
embedded web server to work fine as well.
Are you _removing_ the version of Java that the broker is using during the
upgrade or are you just installing a new version alongside the existing
version? If the former, that seems dangerous in my opinion.
> Is Artemis supposed to reload management.xml automatically...
No. The contents of management.xml are not reloaded automatically.
> ...do we have a chicken-egg problem now; cannot enable JMX because JMX
> is
down?
You do have a chicken-egg problem with using JMX remotely. However, you could
still theoretically use it locally through a direct JVM connection.
You also still have the option of using management messages [1]. You can find
an example of their use in the examples/features/standard/management
directory shipped with the broker.
Justin
[1]
https://activemq.apache.org/components/artemis/documentation/latest/management.html#using-management-message-api
On Wed, Jan 11, 2023 at 2:49 PM Vilius Šumskas <vilius.sums...@rivile.lt>
wrote:
> No, I don't see any of these messages in the log. I have posted full
> log what I see.
>
> This is production system so getting threadump is not so easy, but I
> have decided to test this on our testing environment which runs newest 2.27.1.
> Running restartEmbeddedWebServer() via Hawtio console on that
> environment worked just fine.
>
> I have remembered one thing though. We have upgraded OpenJDK on our
> production environment some time ago, but never restarted Artemis.
> Usually this doesn't produce any issues, but if we decide to restart
> Artemis after JDK upgrade it usually produces some errors about java
> binaries cannot be found. As if Artemis was still loaded on older JDK
> version and stopping it unloads these non-existing older JDK binaries from
> memory.
> Could it be that our Artemis instance is still using older JDK which
> exist only in memory, and console "inherits" that by trying to start
> itself, and then fails?
>
> Anyway, I see that JMX Remote is disabled by default. I have enabled
> JMX connector in management.xml, but I don't see port 1099 coming up.
> Is Artemis supposed to reload management.xml automatically, or do we
> have a chicken-egg problem now; cannot enable JMX because JMX is down?
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Wednesday, January 11, 2023 6:46 PM
> To: users@activemq.apache.org
> Subject: Re: reloading TLS certificate for Jolokia
>
> Those exceptions look internal to Jetty. I wouldn't expect them to
> necessarily be a problem for the restart process. Perhaps it was
> handling a request at the moment you tried to restart it.
>
> Do you ever see any of these messages in the log:
>
> Stopped embedded web server
>
> or
>
> Embedded web server not restarted in x milliseconds
>
> or
>
> Embedded web server restart failed
>
> If not, could you perhaps get a thread dump?
>
> In any event, if you want to start the embedded web server you can do
> so using JMX directly (e.g. via JConsole). Just invoke the
> startEmbeddedWebServer operation on the ActiveMQServerControl MBean.
>
>
> Justin
>
> On Wed, Jan 11, 2023 at 6:41 AM Vilius Šumskas
> <vilius.sums...@rivile.lt>
> wrote:
>
> > Well, I have found " restartEmbeddedWebServer()" and tried to use
> > that from the console, however now the console went down and doesn't come
> > up.
> > The logs say: https://p.defau.lt/?C6ssFYW0JSJiZX1yszP0yQ
> >
> > Jolokia API is also down now. Is there a way to start embedded web
> > server without restarting the broker?
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Vilius Šumskas <vilius.sums...@rivile.lt>
> > Sent: Wednesday, January 11, 2023 1:55 PM
> > To: users@activemq.apache.org
> > Subject: reloading TLS certificate for Jolokia
> >
> > Hi,
> >
> > since because of https://issues.apache.org/jira/browse/ARTEMIS-3117
> > ActiveMQ Artemis doesn‘t reload TLS certificates automatically
> > anymore I‘m wondering, what is the correct way to reload
> > certificates for embedded Jolokia Console?
> >
> > I have reloaded keystore for acceptors via console using reload() on
> > „artemis-ssl“ acceptor, but I just cannot figure out a way to reload
> > it for console itself.
> >
> > --
> > Best Regards,
> >
> > Vilius Šumskas
> > Rivile
> > IT manager
> >
> >
>
