> The second seems to the result of a cluster message received which seems odd on the machine where the session is being created
I think this was the issue. I've changed: StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor(); int clusterMemberCount = Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount")); { StaticMember localMember = new StaticMember(); localMember.setPort(port); localMember.setSecurePort(-1); localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress")); localMember.setDomain("publish-cluster"); localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId")); interceptor.setLocalMember(localMember); interceptor.addStaticMember(localMember);//Removed } ...to... StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor(); int clusterMemberCount = Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount")); { StaticMember localMember = new StaticMember(); localMember.setLocal(true);//Added localMember.setPort(port); localMember.setSecurePort(-1); localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress")); localMember.setDomain("publish-cluster"); localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId")); interceptor.setLocalMember(localMember); } ...and it seems to be fine now. On Fri, Oct 9, 2020 at 12:26 PM Tim N <tnti...@gmail.com> wrote: > Here's the two stack traces, in order of occurrence, for the DeltaSession > creation. The second seems to the result of a cluster message received > which seems odd on the machine where the session is being created: > > <init>:109, DeltaSession (org.apache.catalina.ha.session) > createEmptySession:471, DeltaManager (org.apache.catalina.ha.session) > createSession:708, ManagerBase (org.apache.catalina.session) > createSession:422, DeltaManager (org.apache.catalina.ha.session) > createSession:410, DeltaManager (org.apache.catalina.ha.session) > doGetSession:3043, Request (org.apache.catalina.connector) > getSession:2441, Request (org.apache.catalina.connector) > getSession:908, RequestFacade (org.apache.catalina.connector) > getSession:920, RequestFacade (org.apache.catalina.connector) > getSession:253, HttpServletRequestWrapper (javax.servlet.http) > getSession:253, HttpServletRequestWrapper (javax.servlet.http) > getSession:253, HttpServletRequestWrapper (javax.servlet.http) > onAuthentication:66, RegisterSessionAuthenticationStrategy > (org.springframework.security.web.authentication.session) > doFilter:218, AbstractAuthenticationProcessingFilter > (org.springframework.security.web.authentication) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:200, AbstractAuthenticationProcessingFilter > (org.springframework.security.web.authentication) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:116, LogoutFilter > (org.springframework.security.web.authentication.logout) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilterInternal:74, HeaderWriterFilter > (org.springframework.security.web.header) > doFilter:119, OncePerRequestFilter (org.springframework.web.filter) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:74, CSRFSameOriginHeaderCheckFilter (ourcompany.common.web) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:36, SessionTrackingFilter (ourcompany.common.web) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:105, SecurityContextPersistenceFilter > (org.springframework.security.web.context) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilterInternal:56, WebAsyncManagerIntegrationFilter > (org.springframework.security.web.context.request.async) > doFilter:119, OncePerRequestFilter (org.springframework.web.filter) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:157, ChannelProcessingFilter > (org.springframework.security.web.access.channel) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilter:87, MetadataGeneratorFilter > (org.springframework.security.saml.metadata) > doFilter:334, FilterChainProxy$VirtualFilterChain > (org.springframework.security.web) > doFilterInternal:215, FilterChainProxy (org.springframework.security.web) > doFilter:178, FilterChainProxy (org.springframework.security.web) > invokeDelegate:358, DelegatingFilterProxy (org.springframework.web.filter) > doFilter:271, DelegatingFilterProxy (org.springframework.web.filter) > internalDoFilter:193, ApplicationFilterChain (org.apache.catalina.core) > doFilter:166, ApplicationFilterChain (org.apache.catalina.core) > invoke:202, StandardWrapperValve (org.apache.catalina.core) > __invoke:96, StandardContextValve (org.apache.catalina.core) > invoke:41002, StandardContextValve (org.apache.catalina.core) > invoke:541, AuthenticatorBase (org.apache.catalina.authenticator) > invoke:139, StandardHostValve (org.apache.catalina.core) > invoke:182, JvmRouteBinderValve (org.apache.catalina.ha.session) > invoke:330, ReplicationValve (org.apache.catalina.ha.tcp) > invoke:92, ErrorReportValve (org.apache.catalina.valves) > invoke:74, StandardEngineValve (org.apache.catalina.core) > invoke:747, RemoteIpValve (org.apache.catalina.valves) > service:343, CoyoteAdapter (org.apache.catalina.connector) > service:373, Http11Processor (org.apache.coyote.http11) > process:65, AbstractProcessorLight (org.apache.coyote) > process:868, AbstractProtocol$ConnectionHandler (org.apache.coyote) > doRun:1589, NioEndpoint$SocketProcessor (org.apache.tomcat.util.net) > run:49, SocketProcessorBase (org.apache.tomcat.util.net) > runWorker:1149, ThreadPoolExecutor (java.util.concurrent) > run:624, ThreadPoolExecutor$Worker (java.util.concurrent) > run:61, TaskThread$WrappingRunnable (org.apache.tomcat.util.threads) > run:748, Thread (java.lang) > > ...and the second session creation: > > <init>:109, DeltaSession (org.apache.catalina.ha.session) > createEmptySession:471, DeltaManager (org.apache.catalina.ha.session) > handleSESSION_CREATED:1314, DeltaManager (org.apache.catalina.ha.session) > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session) > messageDataReceived:949, DeltaManager (org.apache.catalina.ha.session) > messageReceived:77, ClusterSessionListener (org.apache.catalina.ha.session) > messageReceived:788, SimpleTcpCluster (org.apache.catalina.ha.tcp) > messageReceived:771, SimpleTcpCluster (org.apache.catalina.ha.tcp) > messageReceived:335, GroupChannel (org.apache.catalina.tribes.group) > messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group) > messageReceived:97, StaticMembershipInterceptor > (org.apache.catalina.tribes.group.interceptors) > messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group) > messageReceived:175, TcpPingInterceptor > (org.apache.catalina.tribes.group.interceptors) > messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group) > messageReceived:117, TcpFailureDetector > (org.apache.catalina.tribes.group.interceptors) > messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group) > messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group) > messageReceived:274, ChannelCoordinator (org.apache.catalina.tribes.group) > messageDataReceived:261, ReceiverBase (org.apache.catalina.tribes.transport) > drainChannel:216, NioReplicationTask > (org.apache.catalina.tribes.transport.nio) > run:101, NioReplicationTask (org.apache.catalina.tribes.transport.nio) > runWorker:1149, ThreadPoolExecutor (java.util.concurrent) > run:624, ThreadPoolExecutor$Worker (java.util.concurrent) > run:748, Thread (java.lang) > > > > On Fri, Oct 9, 2020 at 11:40 AM Tim N <tnti...@gmail.com> wrote: > >> > Can you show us how you configured this cluster please? >> >> Sure. >> >> Tomcat tomcat = new Tomcat(); >> tomcat.setBaseDir(baseDir); >> tomcat.getServer().setAddress("127.0.0.1"); >> tomcat.getServer().setPort(shutDownPort); >> >> tomcat.getServer().setShutdown("SHUTDOWN"); >> tomcat.getServer().addLifecycleListener(new AprLifecycleListener()); >> tomcat.getServer().addLifecycleListener(new >> JreMemoryLeakPreventionListener()); >> tomcat.getServer().addLifecycleListener(new >> GlobalResourcesLifecycleListener()); >> tomcat.getServer().addLifecycleListener(new >> ThreadLocalLeakPreventionListener()); >> >> tomcat.setPort(httpPort); >> tomcat.getConnector().getPort();//The default connector will only be created >> if getConnector is called >> >> LaunchServerHelper.addRemoteIpValve(tomcat); >> >> //... >> >> StandardHost hostManage = getStandardHost(tomcat, "manage." + sgDomain, >> "sg-manage", serverProperties); >> tomcat.getEngine().addChild(hostManage); >> >> tomcat.start(); >> tomcat.getServer().await(); >> tomcat.stop(); >> >> >> private static StandardHost getStandardHost(Tomcat tomcat, String name, >> String module, Properties serverProperties) { >> StandardHost host = new StandardHost(); >> >> modifyHost(tomcat, name, module, host, serverProperties); >> return host; >> } >> >> >> private static void modifyHost(Tomcat tomcat, String name, String module, >> StandardHost host, Properties serverProperties) { >> String webappDirLocation = module + "/src/main/webapp/"; >> File webAppDir = new File(webappDirLocation); >> >> host.setName(name); >> String[] aliases = serverProperties.getProperty("tomcat-" + module + >> "-aliases", "").split(","); >> for (String alias : aliases) { >> if (alias.length() > 0) { >> logger.log(Level.INFO, "Adding alias {0} for {1}", new Object[] >> {alias, module}); >> host.addAlias(alias); >> } >> } >> host.setParent(tomcat.getEngine()); >> host.setAppBase(webAppDir.getAbsolutePath()); >> >> StandardContext ctx = (StandardContext) tomcat.addWebapp(host, "/", >> webAppDir.getAbsolutePath()); >> ((StandardJarScanner) ctx.getJarScanner()).setScanManifest(false); >> ctx.setCookieProcessor(new LegacyCookieProcessor()); >> >> // Declare an alternative location for your "WEB-INF/classes" dir >> // Servlet 3.0 annotation will work >> File additionWebInfClasses = new File(module + "/target/classes"); >> WebResourceRoot resources = new StandardRoot(ctx); >> resources.addPreResources(new DirResourceSet(resources, >> "/WEB-INF/classes", additionWebInfClasses.getAbsolutePath(), "/")); >> ctx.setResources(resources); >> >> LaunchServerHelper.addSimpleTcpCluster(host, 4000, serverProperties); >> } >> >> >> public static void addSimpleTcpCluster(StandardHost host, int port, >> Properties serverProperties) { >> if (!serverProperties.containsKey("tomcat-clusterMemberCount")) { >> return; >> } >> >> SimpleTcpCluster cluster = new SimpleTcpCluster(); >> cluster.setClusterName("SG-cluster"); >> cluster.setChannelStartOptions(3); >> cluster.setChannelSendOptions(8); >> >> { >> DeltaManager clusterManager = new DeltaManager(); >> clusterManager.setName("SG-cluster-manager"); >> clusterManager.setExpireSessionsOnShutdown(false); >> clusterManager.setNotifyListenersOnReplication(true); >> cluster.registerManager(clusterManager); >> } >> >> { >> GroupChannel channel = new GroupChannel(); >> channel.setName("SG-channel"); >> { >> NioReceiver channelReceiver = new NioReceiver(); >> >> channelReceiver.setAddress(serverProperties.getProperty("tomcat-clusterAddress")); >> channelReceiver.setPort(port); >> channelReceiver.setAutoBind(100); >> channelReceiver.setSelectorTimeout(5000); >> channelReceiver.setMaxThreads(6); >> channel.setChannelReceiver(channelReceiver); >> } >> cluster.setChannel(channel); >> >> { >> StaticMembershipInterceptor interceptor = new >> StaticMembershipInterceptor(); >> int clusterMemberCount = >> Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount")); >> { >> StaticMember localMember = new StaticMember(); >> localMember.setPort(port); >> localMember.setSecurePort(-1); >> >> localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress")); >> localMember.setDomain("publish-cluster"); >> >> localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId")); >> interceptor.setLocalMember(localMember); >> interceptor.addStaticMember(localMember); >> } >> for (int i = 1; i <= clusterMemberCount; i++) { >> StaticMember member = new StaticMember(); >> member.setLocal(false); >> member.setPort(port); >> member.setSecurePort(-1); >> >> member.setHost(serverProperties.getProperty("tomcat-clusterMemberAddress" + >> i)); >> member.setDomain("publish-cluster"); >> >> member.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId" >> + i)); >> interceptor.addStaticMember(member); >> } >> channel.addInterceptor(interceptor); >> } >> channel.addInterceptor(new TcpPingInterceptor()); >> channel.addInterceptor(new TcpFailureDetector()); >> channel.addInterceptor(new MessageDispatchInterceptor()); >> } >> >> >> ReplicationTransmitter replicationTransmitter = new >> ReplicationTransmitter(); >> PooledParallelSender sender = new PooledParallelSender(); >> sender.setTimeout(60000); >> sender.setMaxRetryAttempts(5); >> replicationTransmitter.setTransport(sender); >> >> ReplicationValve replicationValve = new ReplicationValve(); >> replicationValve.setFilter(""); >> cluster.addValve(replicationValve); >> cluster.addValve(new JvmRouteBinderValve()); >> >> FarmWarDeployer farmDeployer = new FarmWarDeployer(); >> farmDeployer.setTempDir("/tmp/war-temp/"); >> farmDeployer.setDeployDir("/tmp/war-deploy/"); >> farmDeployer.setWatchDir("/tmp/war-listen/"); >> farmDeployer.setWatchEnabled(false); >> cluster.setClusterDeployer(farmDeployer); >> >> //cluster.addClusterListener(new >> JvmRouteSessionIDBinderListener());//Removed in Tomcat 8. >> https://tomcat.apache.org/migration-8.html#Clustering >> cluster.addClusterListener(new ClusterSessionListener()); >> >> host.setCluster(cluster); >> } >> >> >> tomcat-clusterAddress=192.168.0.2 >> tomcat-clusterMemberUniqueId={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1} >> tomcat-clusterMemberCount=1 >> tomcat-clusterMemberAddress1=192.168.0.3 >> tomcat-clusterMemberUniqueId1={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2} >> >> >> This was initially converted from a server.xml configuration a year ago >> on Tomcat 7 with minor modifications as we've upgraded to Tomcat 8 then 9. >> >> Cheers, >> >> Tim >> >> >> On Thu, Oct 8, 2020 at 10:17 PM Mark Thomas <ma...@apache.org> wrote: >> >>> On 08/10/2020 10:04, Tim N wrote: >>> > Hi, >>> > >>> > I'm in the early stages of analysing this problem: >>> > >>> > - Tomcat Embedded 9.0.37 with clustering enabled >>> > - SpringBoot application, 2.1.16 >>> > - Login with no existing JSESSIONID fails >>> > >>> > I can see the following code is executed twice within one request, the >>> > login attempt: >>> > >>> > >>> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677* >>> > >>> > public void add(Session session) { >>> > sessions.put(session.getIdInternal(), session); >>> > >>> > The first time it's executed, the session with the Spring context is >>> added. >>> > The second time it's executed, a second session with the same ID, but >>> > without the Spring context, or any other session attribute I add for >>> that >>> > matter, overwrites the existing session, and login fails. If I debug >>> and >>> > prevent this by renaming the second session ID, login works because the >>> > original session is preserved. >>> > >>> > The stack-trace for the first call is shown below: >>> > >>> > add:678, ManagerBase (org.apache.catalina.session) >>> > setId:358, StandardSession (org.apache.catalina.session) >>> > setId:327, DeltaSession (org.apache.catalina.ha.session) >>> > setId:345, DeltaSession (org.apache.catalina.ha.session) >>> > createSession:719, ManagerBase (org.apache.catalina.session) >>> > createSession:422, DeltaManager (org.apache.catalina.ha.session) >>> > createSession:410, DeltaManager (org.apache.catalina.ha.session) >>> > doGetSession:3043, Request (org.apache.catalina.connector) >>> > getSession:2441, Request (org.apache.catalina.connector) >>> >>> <snip/> >>> >>> This is the app triggering the creation of the session. >>> >>> > The stack-trace for the second call is shown below: >>> > >>> > add:678, ManagerBase (org.apache.catalina.session) >>> > setId:358, StandardSession (org.apache.catalina.session) >>> > setId:327, DeltaSession (org.apache.catalina.ha.session) >>> > handleSESSION_CREATED:1322, DeltaManager >>> (org.apache.catalina.ha.session) >>> > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session) >>> >>> <snip/> >>> >>> This is the DeltaManager receiving notification that a new session has >>> been created. >>> >>> > Any help would be appreciated. I can replicate this every time and >>> spend >>> > some time investigating this. >>> >>> The new session created message should be send to (and then processed >>> on) every node *except* the node on which the session was originally >>> created. >>> >>> Can you show us how you configured this cluster please? >>> >>> Mark >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>>