Depending on your deployment architecture, this exec and wait may no longer be the right strategy. Yes, by managing the executor pool you can bound how many non-http-serving threads (jobs) are executing concurrently on your web server's host, but it is all premised on using your own hardware. If running "in the cloud", beware that this doesn't lead you to overprovison (overspend on) always-on infrastructure...and thus limit your ability to scale dynamically for variable load requirements.
These days I'd capture the data describing the pending work items somewhere (db/s3/kinesis/lots of other possibilities) and ensure some resilient scheme translates that into a broadcast message for each. Probably subscribe a queue to that broadcast and use that to trigger the work in parallel. If queue listeners are executing on ec2/ecs/eks, perhaps scale as that queue size grows. Or most likely execute the steps with serverless lambdas, potentially with massive concurrency and low cost. Today there are lots of alternatives, depending on the workload: step functions, etc. Usually you can find a way to ensure at-least-once execution of each task, so this other side of the coin is to make the work idempotent so duplicative execution cannot introduce issues. Perhaps a DLQ for problematic jobs. Make sure something monitors that DLQ. When exec-and-wait was originally implemented we lived in an only intermittently connected world and owned all the hardware. I used it to sync up work done by a disconnected instance with "the real, system" upon reconnection. (For me this meant photographers at parties, building engagement after the party by virtualizing the real world connections. It displayed photos live during the event, and allowed assistants trailing behind the photographers to tag event-goers in those photos (by scanning barcodes on their wristbands/entering account ids/collecting email addresses on the spot to create accounts). This allowed us to email event photos to the subjects in them by the next morning, and gave participants an easy way to connect with each other afterwards just by having a photographer take their picture together. At the time, facial recognition was just becoming possible and was also leveraged. But back to the technical side this meant I ran either ran a full instance (db/struts web server/etc) at the front door managing barcode assignments to match prepaid ticket holders, and supporting the whole event via wifi (I'd plug in repeaters as necessary thru the venue and/or stick battery-powered ones in the back pockets of security guards) or that I ran a full instance on the device carried around by the photographer's assistant. This was all before tablets/smartphones/social networks/etc. Today we live in a very different world--always connected, cloud based, etc. Some tools/techniques still apply. Many now have much better solutions (more reliable/resilient/scalable/simpler; less expensive to build/launch/operate/etc.). YMMV. -Dale > On Sep 23, 2023, at 9:52 PM, Dale Newfield <d...@newfield.org> wrote: > > If it runs again then look at the logic where it's gotta decide whether this > is the initial call or if the job is already running and this p my try just > a status check. Is it looking at the request to figure that out? If so, and > the expected info isn't found, walk the data path to figure out where it is > getting lost (sounds like the session filter might be a good place to start). > Is it looking in a database/shared memory/file system? Perhaps these are > not in sync? Perhaps the long running job is doing everything in a single > transaction, not committed until the very end, and the "I'm running!" flag > set within the transaction is not visible unless the query is > READ_UNCOMMITTED? > > This run/not decision is critical, so it should be determined atomically. > Perhaps one transaction for the decision (which either concludes "report > status" or ensures its record is visible to any other query) and others for > each separable unit of work within the long running job? > > Oh, yeah, you said Redis. I've somehow managed to avoid that, so I don't > know what persistent data check/update can be done atomically, but that's > where you're hanging your hat. Maybe the response to your "is anyone else > already running this job?" query is being cached, and so the first "no" > response is being returned every time? > > So many places this decision could go wrong in a distributed platform. Every > possible race condition will eventually happen. Can you confidently say that > two concurrent requests on distinct hosts will always result in a single > "winner"? > > (Then later on there's a "liveness" detector you'll need to figure out in > case the winner fails. Perhaps it should periodically update that record to > show when it was last known to be running? (How far into the job it is right > then would also be great to snapshot.) if that time stamp is older than X, > is it safe to assume it failed and to elect another leader to take its place? > Make sure time zone differences on different hosts can't impact the timeout > checks. Is it safe, even in the face of a network disconnection, so > eventually the evicted leader can reappear? Would one of them detect the > issue and kill itself?) > > -Dale > >> On Sep 23, 2023, at 4:23 PM, Burton Rhodes <burtonrho...@gmail.com> wrote: >> >> I am attempting to implement a centralized session store to an existing >> application using Spring Session (Redis) and Struts. I have everything >> working on a basic level, but I am running into an issue when a Struts >> Action uses ExecuteAndWait. The refresh attempts don't seem to pick up the >> running action using the provided Struts Token (at least I think this is the >> case). So on each refresh, the action seems to run again as if it was the >> first time duplicating database inserts, etc. It becomes an infinite >> process, and Struts never returns the final "SUCCESS" result JSP page to the >> browser. It's worth noting that if I disable the Spring Session filter, >> everything works fine. Anyone have a clue as to what might be going on or >> how I might begin to troubleshoot? >> >> I've included a thread dump so you can see the filters involved. The >> "SessionRepositoryFilter" is where the HttpSession is wrapped by Spring >> (Redis) and is before any Struts filters. I've also included a basic action >> definition that is causing the issue (although this is happening on all >> actions that use the ExecuteAndWait logic). Any help is appreciated. >> >> ** Example Action Definition ** >> >> <action name="ContactBatchCategories_update" >> class="com.afs.web.struts.action.contact.ContactBatchCategoriesAction" >> method="update"> >> <interceptor-ref name="myDefaultStack"/> >> <interceptor-ref name="openSessionExecuteAndWaitInterceptor" > >> <param name="delay">0</param> >> </interceptor-ref> >> <result >> name="input">/struts/search/actions/contactBatchCategories_modal.jsp</result> >> <result name="wait">/struts/common/progressMonitorWait_modal.jsp</result> >> <result>/struts/common/progressMonitorSuccess_modal.jsp</result> >> </action> >> >> >> ** Thread Dump ** >> >> java.lang.Thread.State: RUNNABLE >> at >> com.afs.web.struts.action.contact.ContactBatchCategoriesAction.prepare(ContactBatchCategoriesAction.java:62) >> at >> com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171) >> at >> com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) >> at >> org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:154) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) >> at >> com.afs.web.config.struts.StrutsAccountInterceptor.intercept(StrutsAccountInterceptor.java:67) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) >> at >> com.afs.web.config.struts.StrutsSessionInterceptor.intercept(StrutsSessionInterceptor.java:44) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) >> at >> com.afs.web.common.struts.interceptor.ExceptionInterceptor.intercept(ExceptionInterceptor.java:33) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299) >> at >> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) >> at >> org.apache.struts2.factory.StrutsActionProxy.execute(StrutsActionProxy.java:48) >> at >> org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:651) >> at >> org.apache.struts2.dispatcher.ExecuteOperations.executeAction(ExecuteOperations.java:79) >> at >> org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.handleRequest(StrutsPrepareAndExecuteFilter.java:157) >> at >> org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.tryHandleRequest(StrutsPrepareAndExecuteFilter.java:140) >> at >> org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:128) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> com.afs.web.config.filter.MyAppSessionFilter.doFilter(MyAppSessionFilter.java:85) >> at >> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) >> at >> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352) >> at >> org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) >> at >> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131) >> at >> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) >> at >> org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:151) >> at >> org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:129) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) >> at >> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) >> at >> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) >> at >> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) >> at >> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> at >> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361) >> at >> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225) >> at >> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190) >> at >> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) >> at >> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186) >> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142) >> at >> org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) >> at >> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) >> at >> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) >> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) >> at >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) >> at >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) >> at >> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) >> at org.eclipse.jetty.server.Server.handle(Server.java:563) >> at >> org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) >> at >> org.eclipse.jetty.server.HttpChannel$$Lambda$1353.1731450897.dispatch(Unknown >> Source:-1) >> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) >> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) >> at >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) >> at >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) >> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) >> at >> org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) >> at >> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$1338.510268842.run(Unknown >> Source:-1) >> at >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) >> at java.lang.Thread.run(Thread.java:834) --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@struts.apache.org For additional commands, e-mail: user-h...@struts.apache.org
