Well, Here is the report after I've traced the errors down.
I've noticed that when my custom java message adapter halts BlazeDS'
messaging still runs. Several diagnostic applications indicate the
durability of BlazeDS messaging.
I have however tracked down the source the halting of my java message
adapter thread to be the MessageBroker...
My custom message adapter is multithreaded in that it manages two
tasks: One thread listens for messages from a server and then starts a
"fire-and-forget" messaging thread that then packages and routes the
message into an AsyncMessage for broadcast to BlazeDS before dying.
Traces indicate that after about 6 days of reliable message routing
the messaging threads generated by my listener thread start halting at
the msgBroker.routeMessageToService(msg, null); method call. (Trace
statements to console halt after "Trying to routeMessageToService").
public void sendMessage()
{
System.out.println(name+" Connecting to MessageBroker");
msgBroker = MessageBroker.getMessageBroker(null);
System.out.println("MessageBroker.getId() = "+msgBroker.getId());
String clientID = UUIDUtils.createUUID(false);
msg.setClientId(clientID);
msg.setMessageId(UUIDUtils.createUUID(false));
msg.setTimestamp(System.currentTimeMillis());
System.out.println(name+" Trying to routeMessageToService");
msgBroker.routeMessageToService(msg, null);
System.out.println(name+" Successful message route");
}
That is to say that msgBroker's routeMessageToService() method call is
not returning and the messaging threads are all halting and piling up
in the JVM. After about 4000 of these halted messaging threads piling
up the JVM runs out of memory and the listener thread can't
instantiate any more messaging threads and the whole thing comes to a
halt.
You can say that my problem is thus: MessageBroker's
routeMessageToService() method stops returning control to my message
thread after about 6 days halting it until the unsent message threads
pile up leading up to an out of memory crash.
I'm working on a fix that will kill the halted message thread if it
runs longer than a few seconds. This should aleviate the thread
traffic jam but I don't know if the MessageBroker will start routing
messages again. I don't know why it stops either.
Hope you could shed some light on this.
Best,
Justin
--- In flexcoders@yahoogroups.com, "Justin Fujita" <[EMAIL PROTECTED]>
wrote:
>
> Thanks Mete,
>
> I really appreciate your help. I've changed up my java message
producer a
> bit to see if that helps as well. I guess I'll let you know how it
works
> out in about 5-6 days
>
>
> Thanks again,
> -Justin
>
> On Tue, Apr 22, 2008 at 11:15 AM, meteatamel <[EMAIL PROTECTED]> wrote:
>
> > I've added some info to the bug report. Please take a look.
> >
> > -Mete
> >
> >
> > --- In flexcoders@yahoogroups.com <flexcoders%40yahoogroups.com>,
> > "jfujita1" <justin.fujita@> wrote:
> > >
> > > Hi,
> > >
> > > I've built a GIS FLEX application that uses blazeDS' messaging
> > > capability to push location (latitude/longitude/geocode) info to
flex
> > > clients.
> > >
> > > The message producer is a custom Java bootstrapper and adapter that
> > > configures the messaging destination endpoints upon server startup.
> > >
> > > The destination runtime config java bootstrap method for our
> > > "fleet_demo" destination looks like this:
> > >
> > > public void initializeDestinations()
> > > {
> > > String destinationId = "fleet_demo";
> > > String serviceId = "message-service";
> > > MessageBroker broker = MessageBroker.getMessageBroker(null);
> > > MessageService service = (MessageService)
> > > broker.getService(serviceId);
> > > MessageDestination destination = (MessageDestination)
> > > service.createDestination(destinationId);
> > >
> > > destination.addChannel("my-streaming-amf");
> > > destination.addChannel("my-polling-amf");
> > >
> > > ServerSettings serverSettings = new ServerSettings();
> > > serverSettings.setMaxCacheSize(1000);
> > > serverSettings.setMessageTTL(0);
> > > serverSettings.setDurable(false);
> > >
> > > destination.setServerSettings(serverSettings);
> > >
> > > if (service.isStarted())
> > > {
> > > //initialize and run the destination
> > > destination.start();
> > > }
> > > }
> > >
> > >
> > > The bootstrapper also spawns a separate thread that continuously
> > > receives location data from our server and packages it into an
> > > AsyncMessage and routes it to the appropriate destination using a
> > > method similar to this:
> > >
> > > private void sendMessage(latitudeFromServer:double,
> > > longitudeFromServer:double)
> > > {
> > > MessageBroker msgBroker = MessageBroker.getMessageBroker(null);
> > > String clientID = UUIDUtils.createUUID(false);
> > > AsyncMessage msg = new AsyncMessage();
> > >
> > > msg.setDestination("fleet_demo");
> > > msg.setHeader("latitude", latitudeFromServer);
> > > msg.setHeader("longitude", longitudeFromServer);
> > >
> > > msg.setClientId(clientID);
> > > msg.setMessageId(UUIDUtils.createUUID(false));
> > > msg.setTimestamp(System.currentTimeMillis());
> > > msgBroker.routeMessageToService(msg, null);
> > > }
> > >
> > >
> > > Now I start up the server and the bootstrapper initializes the
> > > "fleet_demo" endpoint and the sendMessage thread routes location
data
> > > perfectly to my flex client(averaging about 500-1600 messages per
> > > hour) It works! This keeps up for about 5-6 days. After that I stop
> > > receiving any messages when I should be. Its as though the
> > > MessageBrokerServlet just goes silent. The only way to fix this
is to
> > > restart jboss and when I try to shutdown the server I get this NPE
> > > message from the MessageBrokerServlet:
> > >
> > > 11:22:36,295 INFO [Server] Runtime shutdown hook called,
forceHalt: true
> > > 11:22:36,295 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
> > > 11:22:36,296 INFO [TomcatDeployer] undeploy, ctxPath=/jmx-console,
> > > warUrl=.../deploy/jmx-console.war/
> > > 11:22:36,330 INFO [TomcatDeployer] undeploy, ctxPath=/enterprise,
> > > warUrl=.../deploy/enterprise.war/
> > > 11:22:36,331 INFO [StandardWrapper] Waiting for 5 instance(s) to be
> > > deallocated
> > > 11:22:37,350 INFO [StandardWrapper] Waiting for 5 instance(s) to be
> > > deallocated
> > > 11:22:38,373 INFO [StandardWrapper] Waiting for 3 instance(s) to be
> > > deallocated
> > > 11:22:38,512 ERROR [[MessageBrokerServlet]] Servlet.service() for
> > > servlet MessageBrokerServlet threw exception
> > > java.lang.NullPointerException
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.handleFlexClientStreamingOpenRequest(BaseStreamingHTTPEndpoint.java:959)
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.serviceStreamingRequest(BaseStreamingHTTPEndpoint.java:1131)
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.service(BaseStreamingHTTPEndpoint.java:468)
> > > at
> > >
> >
flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:377)
> > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > at
> > >
> >
> >
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > at
> > >
> >
> >
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
> > > at
> > >
> >
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> > > at
> > >
> >
> >
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> > > at
> > >
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
> > > at java.lang.Thread.run(Thread.java:595)
> > > 11:22:38,514 ERROR [[MessageBrokerServlet]] Servlet.service() for
> > > servlet MessageBrokerServlet threw exception
> > > java.lang.NullPointerException
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.handleFlexClientStreamingOpenRequest(BaseStreamingHTTPEndpoint.java:959)
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.serviceStreamingRequest(BaseStreamingHTTPEndpoint.java:1131)
> > > at
> > >
> >
> >
flex.messaging.endpoints.BaseStreamingHTTPEndpoint.service(BaseStreamingHTTPEndpoint.java:468)
> > > at
> > >
> >
flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:377)
> > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > at
> > >
> >
> >
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > at
> > >
> >
> >
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > at
> > >
> >
> >
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
> > > at
> > >
> >
> >
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > at
> > >
> >
> >
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
> > > at
> > >
> >
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
> > > at
> > >
> >
> >
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> > > at
> > >
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
> > > at java.lang.Thread.run(Thread.java:595)
> > >
> > >
> > >
> > > I've created a bug report at the Adobe BlazeDS bug and issue
> > > management page but I need a robust messaging solution for this
> > > project and I was wondering if anybody here had any experience
> > > regarding this problem.
> > >
> > >
> > > Thanks for reading,
> > >
> > > -Justin
> > >
> >
> >
> >
>
>
>
> --
> Thanks for reading. -Justin
> Flickr: http://www.flickr.com/photos/neopan/
>
