Hello all
I first noticed the problem described below with Artemis 2.35.0, but version
2.37.0 shows the same behavior:
Calling the restartEmbeddedWebServer() operation via the web console does stop
jetty and tries to restart it but jetty then generates a lot of exceptions, and
the web console becomes disfunctional.
The browser (firefox) shows: The page isn't redirecting properly.
2024-08-29 22:15:48,490 INFO [org.apache.activemq.artemis] AMQ241005: Stopping
embedded web server
2024-08-29 22:15:48,504 INFO [io.hawt.HawtioContextListener] Destroying hawtio
services
2024-08-29 22:15:48,511 INFO
[org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
artemis-plugin plugin
2024-08-29 22:15:48,512 INFO
[org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed
activemq-branding plugin
2024-08-29 22:15:51,019 WARN [org.eclipse.jetty.server.session.SessionHandler]
Unable to release Session
Session@5b1320cb{id=node01ff9xvgbxku3o1u7s3geygd6fi0,x=node01ff9xvgbxku3o1u7s3geygd6fi0.node0,req=0,res=false}<mailto:Session@5b1320cb%7bid=node01ff9xvgbxku3o1u7s3geygd6fi0,x=node01ff9xvgbxku3o1u7s3geygd6fi0.node0,req=0,res=false%7d>
java.lang.IllegalStateException: Not started
at
org.eclipse.jetty.server.session.AbstractSessionDataStore.store(AbstractSessionDataStore.java:164)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.AbstractSessionCache.release(AbstractSessionCache.java:541)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.SessionHandler.complete(SessionHandler.java:358)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.Request.lambda$leaveSession$0(Request.java:427)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1469)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.Request.leaveSession(Request.java:427)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.Request.onCompleted(Request.java:1472)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:957)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:489)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
~[jetty-io-10.0.22.jar:10.0.22]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
~[jetty-io-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
~[jetty-io-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
~[jetty-util-10.0.22.jar:10.0.22]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-08-29 22:15:51,044 INFO [org.apache.activemq.artemis] AMQ241006: Stopped
embedded web server
2024-08-29 22:15:51,045 INFO [org.apache.activemq.artemis] AMQ241003: Starting
embedded web server
2024-08-29 22:15:51,110 INFO
[org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
activemq-branding plugin
2024-08-29 22:15:51,175 INFO
[org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
artemis-plugin plugin
2024-08-29 22:15:51,316 INFO [io.hawt.HawtioContextListener] Initialising
hawtio services
2024-08-29 22:15:51,318 INFO [io.hawt.system.ConfigManager] Configuration will
be discovered via system properties
2024-08-29 22:15:51,319 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio
2.17.7
2024-08-29 22:15:51,331 INFO [io.hawt.web.auth.AuthenticationConfiguration]
Starting hawtio authentication filter, JAAS realm: "activemq" authorized
role(s): "amq" role principal classes:
"org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2024-08-29 22:15:51,335 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio
loginRedirectFilter is using 1800 sec. HttpSession timeout
2024-08-29 22:15:51,349 INFO [org.apache.activemq.artemis] AMQ241001: HTTP
Server started at http://localhost:8161
2024-08-29 22:15:51,349 INFO [org.apache.activemq.artemis] AMQ241002: Artemis
Jolokia REST API available at http://localhost:8161/console/jolokia
2024-08-29 22:15:51,350 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
Console available at http://localhost:8161/console
2024-08-29 22:15:53,816 INFO
[io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
property: [key=policyLocation,
value=file:/opt/artemisbroker/etc/jolokia-access.xml]
2024-08-29 22:15:53,896 INFO [io.hawt.web.auth.LoginServlet] Hawtio login is
using 1800 sec. HttpSession timeout
2024-08-29 22:15:53,898 WARN [org.eclipse.jetty.server.HttpChannelState]
unhandled due to prior sendError
org.eclipse.jetty.io.EofException: Closed
at
org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:747)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:777)
~[jetty-server-10.0.22.jar:10.0.22]
at java.base/java.io.OutputStream.write(OutputStream.java:124) ~[?:?]
at
io.hawt.web.filters.BaseTagHrefFilter.doFilter(BaseTagHrefFilter.java:81) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:616)
~[jetty-security-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:215)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:135)
~[jetty-server-10.0.22.jar:10.0.22]
at io.hawt.web.auth.Redirector.doForward(Redirector.java:45) ~[?:?]
at io.hawt.web.auth.LoginServlet.doGet(LoginServlet.java:59) ~[?:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
~[jetty-servlet-api-4.0.6.jar:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
~[jetty-servlet-api-4.0.6.jar:?]2
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63)
~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:46) ~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:107)
~[?:?]
at
io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60)
~[?:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.apache.activemq.artemis.component.AuthenticationFilter.doFilter(AuthenticationFilter.java:43)
~[artemis-web-2.37.0.jar:2.37.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50)
~[artemis-web-2.37.0.jar:2.37.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
~[jetty-security-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
~[jetty-servlet-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.Server.handle(Server.java:563)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
~[jetty-server-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
~[jetty-io-10.0.22.jar:10.0.22]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
~[jetty-io-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
~[jetty-io-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
~[jetty-util-10.0.22.jar:10.0.22]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
~[jetty-util-10.0.22.jar:10.0.22]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-08-29 22:15:53,937 WARN [org.eclipse.jetty.server.HttpChannelState]
unhandled due to prior sendError
org.eclipse.jetty.io.EofException: Closed
at
org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:747)
~[jetty-server-10.0.22.jar:10.0.22]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:777)
~[jetty-server-10.0.22.jar:10.0.22]
at java.base/java.io.OutputStream.write(OutputStream.java:124) ~[?:?]
at
io.hawt.web.filters.BaseTagHrefFilter.doFilter(BaseTagHrefFilter.java:81)
~[hawtio-system-2.17.7.jar:2.17.7]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
~[jetty-servlet-10.0.22.jar:10.0.22]
...
...
This last exception is repeated multiple times and no management pages can be
shown in the browser.
The only way to repair that I found is to restart Artemis completely.
This is a problem because we want to add a Spring XML based Camel web app to
jetty. We foresee the need to restart the web app (via
restartEmbeddedWebserver()) now and then, e.g. when changing the XML Camel
routes).
The operation restartEmbeddedWebserver() was still functioning correctly in
Artemis 2.32.0.
Is this a known problem?
Erik Devriendt
