> The second seems to the result of a cluster message received which seems
odd on the machine where the session is being created
I think this was the issue. I've changed:
StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
int clusterMemberCount =
Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
{
StaticMember localMember = new StaticMember();
localMember.setPort(port);
localMember.setSecurePort(-1);
localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
localMember.setDomain("publish-cluster");
localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
interceptor.setLocalMember(localMember);
interceptor.addStaticMember(localMember);//Removed
}
...to...
StaticMembershipInterceptor interceptor = new StaticMembershipInterceptor();
int clusterMemberCount =
Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
{
StaticMember localMember = new StaticMember();
localMember.setLocal(true);//Added
localMember.setPort(port);
localMember.setSecurePort(-1);
localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
localMember.setDomain("publish-cluster");
localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
interceptor.setLocalMember(localMember);
}
...and it seems to be fine now.
On Fri, Oct 9, 2020 at 12:26 PM Tim N <[email protected]> wrote:
> Here's the two stack traces, in order of occurrence, for the DeltaSession
> creation. The second seems to the result of a cluster message received
> which seems odd on the machine where the session is being created:
>
> <init>:109, DeltaSession (org.apache.catalina.ha.session)
> createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
> createSession:708, ManagerBase (org.apache.catalina.session)
> createSession:422, DeltaManager (org.apache.catalina.ha.session)
> createSession:410, DeltaManager (org.apache.catalina.ha.session)
> doGetSession:3043, Request (org.apache.catalina.connector)
> getSession:2441, Request (org.apache.catalina.connector)
> getSession:908, RequestFacade (org.apache.catalina.connector)
> getSession:920, RequestFacade (org.apache.catalina.connector)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> getSession:253, HttpServletRequestWrapper (javax.servlet.http)
> onAuthentication:66, RegisterSessionAuthenticationStrategy
> (org.springframework.security.web.authentication.session)
> doFilter:218, AbstractAuthenticationProcessingFilter
> (org.springframework.security.web.authentication)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:200, AbstractAuthenticationProcessingFilter
> (org.springframework.security.web.authentication)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:116, LogoutFilter
> (org.springframework.security.web.authentication.logout)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilterInternal:74, HeaderWriterFilter
> (org.springframework.security.web.header)
> doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:74, CSRFSameOriginHeaderCheckFilter (ourcompany.common.web)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:36, SessionTrackingFilter (ourcompany.common.web)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:105, SecurityContextPersistenceFilter
> (org.springframework.security.web.context)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilterInternal:56, WebAsyncManagerIntegrationFilter
> (org.springframework.security.web.context.request.async)
> doFilter:119, OncePerRequestFilter (org.springframework.web.filter)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:157, ChannelProcessingFilter
> (org.springframework.security.web.access.channel)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilter:87, MetadataGeneratorFilter
> (org.springframework.security.saml.metadata)
> doFilter:334, FilterChainProxy$VirtualFilterChain
> (org.springframework.security.web)
> doFilterInternal:215, FilterChainProxy (org.springframework.security.web)
> doFilter:178, FilterChainProxy (org.springframework.security.web)
> invokeDelegate:358, DelegatingFilterProxy (org.springframework.web.filter)
> doFilter:271, DelegatingFilterProxy (org.springframework.web.filter)
> internalDoFilter:193, ApplicationFilterChain (org.apache.catalina.core)
> doFilter:166, ApplicationFilterChain (org.apache.catalina.core)
> invoke:202, StandardWrapperValve (org.apache.catalina.core)
> __invoke:96, StandardContextValve (org.apache.catalina.core)
> invoke:41002, StandardContextValve (org.apache.catalina.core)
> invoke:541, AuthenticatorBase (org.apache.catalina.authenticator)
> invoke:139, StandardHostValve (org.apache.catalina.core)
> invoke:182, JvmRouteBinderValve (org.apache.catalina.ha.session)
> invoke:330, ReplicationValve (org.apache.catalina.ha.tcp)
> invoke:92, ErrorReportValve (org.apache.catalina.valves)
> invoke:74, StandardEngineValve (org.apache.catalina.core)
> invoke:747, RemoteIpValve (org.apache.catalina.valves)
> service:343, CoyoteAdapter (org.apache.catalina.connector)
> service:373, Http11Processor (org.apache.coyote.http11)
> process:65, AbstractProcessorLight (org.apache.coyote)
> process:868, AbstractProtocol$ConnectionHandler (org.apache.coyote)
> doRun:1589, NioEndpoint$SocketProcessor (org.apache.tomcat.util.net)
> run:49, SocketProcessorBase (org.apache.tomcat.util.net)
> runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
> run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
> run:61, TaskThread$WrappingRunnable (org.apache.tomcat.util.threads)
> run:748, Thread (java.lang)
>
> ...and the second session creation:
>
> <init>:109, DeltaSession (org.apache.catalina.ha.session)
> createEmptySession:471, DeltaManager (org.apache.catalina.ha.session)
> handleSESSION_CREATED:1314, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
> messageDataReceived:949, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:77, ClusterSessionListener (org.apache.catalina.ha.session)
> messageReceived:788, SimpleTcpCluster (org.apache.catalina.ha.tcp)
> messageReceived:771, SimpleTcpCluster (org.apache.catalina.ha.tcp)
> messageReceived:335, GroupChannel (org.apache.catalina.tribes.group)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:97, StaticMembershipInterceptor
> (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:175, TcpPingInterceptor
> (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:117, TcpFailureDetector
> (org.apache.catalina.tribes.group.interceptors)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:91, ChannelInterceptorBase (org.apache.catalina.tribes.group)
> messageReceived:274, ChannelCoordinator (org.apache.catalina.tribes.group)
> messageDataReceived:261, ReceiverBase (org.apache.catalina.tribes.transport)
> drainChannel:216, NioReplicationTask
> (org.apache.catalina.tribes.transport.nio)
> run:101, NioReplicationTask (org.apache.catalina.tribes.transport.nio)
> runWorker:1149, ThreadPoolExecutor (java.util.concurrent)
> run:624, ThreadPoolExecutor$Worker (java.util.concurrent)
> run:748, Thread (java.lang)
>
>
>
> On Fri, Oct 9, 2020 at 11:40 AM Tim N <[email protected]> wrote:
>
>> > Can you show us how you configured this cluster please?
>>
>> Sure.
>>
>> Tomcat tomcat = new Tomcat();
>> tomcat.setBaseDir(baseDir);
>> tomcat.getServer().setAddress("127.0.0.1");
>> tomcat.getServer().setPort(shutDownPort);
>>
>> tomcat.getServer().setShutdown("SHUTDOWN");
>> tomcat.getServer().addLifecycleListener(new AprLifecycleListener());
>> tomcat.getServer().addLifecycleListener(new
>> JreMemoryLeakPreventionListener());
>> tomcat.getServer().addLifecycleListener(new
>> GlobalResourcesLifecycleListener());
>> tomcat.getServer().addLifecycleListener(new
>> ThreadLocalLeakPreventionListener());
>>
>> tomcat.setPort(httpPort);
>> tomcat.getConnector().getPort();//The default connector will only be created
>> if getConnector is called
>>
>> LaunchServerHelper.addRemoteIpValve(tomcat);
>>
>> //...
>>
>> StandardHost hostManage = getStandardHost(tomcat, "manage." + sgDomain,
>> "sg-manage", serverProperties);
>> tomcat.getEngine().addChild(hostManage);
>>
>> tomcat.start();
>> tomcat.getServer().await();
>> tomcat.stop();
>>
>>
>> private static StandardHost getStandardHost(Tomcat tomcat, String name,
>> String module, Properties serverProperties) {
>> StandardHost host = new StandardHost();
>>
>> modifyHost(tomcat, name, module, host, serverProperties);
>> return host;
>> }
>>
>>
>> private static void modifyHost(Tomcat tomcat, String name, String module,
>> StandardHost host, Properties serverProperties) {
>> String webappDirLocation = module + "/src/main/webapp/";
>> File webAppDir = new File(webappDirLocation);
>>
>> host.setName(name);
>> String[] aliases = serverProperties.getProperty("tomcat-" + module +
>> "-aliases", "").split(",");
>> for (String alias : aliases) {
>> if (alias.length() > 0) {
>> logger.log(Level.INFO, "Adding alias {0} for {1}", new Object[]
>> {alias, module});
>> host.addAlias(alias);
>> }
>> }
>> host.setParent(tomcat.getEngine());
>> host.setAppBase(webAppDir.getAbsolutePath());
>>
>> StandardContext ctx = (StandardContext) tomcat.addWebapp(host, "/",
>> webAppDir.getAbsolutePath());
>> ((StandardJarScanner) ctx.getJarScanner()).setScanManifest(false);
>> ctx.setCookieProcessor(new LegacyCookieProcessor());
>>
>> // Declare an alternative location for your "WEB-INF/classes" dir
>> // Servlet 3.0 annotation will work
>> File additionWebInfClasses = new File(module + "/target/classes");
>> WebResourceRoot resources = new StandardRoot(ctx);
>> resources.addPreResources(new DirResourceSet(resources,
>> "/WEB-INF/classes", additionWebInfClasses.getAbsolutePath(), "/"));
>> ctx.setResources(resources);
>>
>> LaunchServerHelper.addSimpleTcpCluster(host, 4000, serverProperties);
>> }
>>
>>
>> public static void addSimpleTcpCluster(StandardHost host, int port,
>> Properties serverProperties) {
>> if (!serverProperties.containsKey("tomcat-clusterMemberCount")) {
>> return;
>> }
>>
>> SimpleTcpCluster cluster = new SimpleTcpCluster();
>> cluster.setClusterName("SG-cluster");
>> cluster.setChannelStartOptions(3);
>> cluster.setChannelSendOptions(8);
>>
>> {
>> DeltaManager clusterManager = new DeltaManager();
>> clusterManager.setName("SG-cluster-manager");
>> clusterManager.setExpireSessionsOnShutdown(false);
>> clusterManager.setNotifyListenersOnReplication(true);
>> cluster.registerManager(clusterManager);
>> }
>>
>> {
>> GroupChannel channel = new GroupChannel();
>> channel.setName("SG-channel");
>> {
>> NioReceiver channelReceiver = new NioReceiver();
>>
>> channelReceiver.setAddress(serverProperties.getProperty("tomcat-clusterAddress"));
>> channelReceiver.setPort(port);
>> channelReceiver.setAutoBind(100);
>> channelReceiver.setSelectorTimeout(5000);
>> channelReceiver.setMaxThreads(6);
>> channel.setChannelReceiver(channelReceiver);
>> }
>> cluster.setChannel(channel);
>>
>> {
>> StaticMembershipInterceptor interceptor = new
>> StaticMembershipInterceptor();
>> int clusterMemberCount =
>> Integer.parseInt(serverProperties.getProperty("tomcat-clusterMemberCount"));
>> {
>> StaticMember localMember = new StaticMember();
>> localMember.setPort(port);
>> localMember.setSecurePort(-1);
>>
>> localMember.setHost(serverProperties.getProperty("tomcat-clusterAddress"));
>> localMember.setDomain("publish-cluster");
>>
>> localMember.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"));
>> interceptor.setLocalMember(localMember);
>> interceptor.addStaticMember(localMember);
>> }
>> for (int i = 1; i <= clusterMemberCount; i++) {
>> StaticMember member = new StaticMember();
>> member.setLocal(false);
>> member.setPort(port);
>> member.setSecurePort(-1);
>>
>> member.setHost(serverProperties.getProperty("tomcat-clusterMemberAddress" +
>> i));
>> member.setDomain("publish-cluster");
>>
>> member.setUniqueId(serverProperties.getProperty("tomcat-clusterMemberUniqueId"
>> + i));
>> interceptor.addStaticMember(member);
>> }
>> channel.addInterceptor(interceptor);
>> }
>> channel.addInterceptor(new TcpPingInterceptor());
>> channel.addInterceptor(new TcpFailureDetector());
>> channel.addInterceptor(new MessageDispatchInterceptor());
>> }
>>
>>
>> ReplicationTransmitter replicationTransmitter = new
>> ReplicationTransmitter();
>> PooledParallelSender sender = new PooledParallelSender();
>> sender.setTimeout(60000);
>> sender.setMaxRetryAttempts(5);
>> replicationTransmitter.setTransport(sender);
>>
>> ReplicationValve replicationValve = new ReplicationValve();
>> replicationValve.setFilter("");
>> cluster.addValve(replicationValve);
>> cluster.addValve(new JvmRouteBinderValve());
>>
>> FarmWarDeployer farmDeployer = new FarmWarDeployer();
>> farmDeployer.setTempDir("/tmp/war-temp/");
>> farmDeployer.setDeployDir("/tmp/war-deploy/");
>> farmDeployer.setWatchDir("/tmp/war-listen/");
>> farmDeployer.setWatchEnabled(false);
>> cluster.setClusterDeployer(farmDeployer);
>>
>> //cluster.addClusterListener(new
>> JvmRouteSessionIDBinderListener());//Removed in Tomcat 8.
>> https://tomcat.apache.org/migration-8.html#Clustering
>> cluster.addClusterListener(new ClusterSessionListener());
>>
>> host.setCluster(cluster);
>> }
>>
>>
>> tomcat-clusterAddress=192.168.0.2
>> tomcat-clusterMemberUniqueId={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1}
>> tomcat-clusterMemberCount=1
>> tomcat-clusterMemberAddress1=192.168.0.3
>> tomcat-clusterMemberUniqueId1={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2}
>>
>>
>> This was initially converted from a server.xml configuration a year ago
>> on Tomcat 7 with minor modifications as we've upgraded to Tomcat 8 then 9.
>>
>> Cheers,
>>
>> Tim
>>
>>
>> On Thu, Oct 8, 2020 at 10:17 PM Mark Thomas <[email protected]> wrote:
>>
>>> On 08/10/2020 10:04, Tim N wrote:
>>> > Hi,
>>> >
>>> > I'm in the early stages of analysing this problem:
>>> >
>>> > - Tomcat Embedded 9.0.37 with clustering enabled
>>> > - SpringBoot application, 2.1.16
>>> > - Login with no existing JSESSIONID fails
>>> >
>>> > I can see the following code is executed twice within one request, the
>>> > login attempt:
>>> >
>>> >
>>> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
>>> >
>>> > public void add(Session session) {
>>> > sessions.put(session.getIdInternal(), session);
>>> >
>>> > The first time it's executed, the session with the Spring context is
>>> added.
>>> > The second time it's executed, a second session with the same ID, but
>>> > without the Spring context, or any other session attribute I add for
>>> that
>>> > matter, overwrites the existing session, and login fails. If I debug
>>> and
>>> > prevent this by renaming the second session ID, login works because the
>>> > original session is preserved.
>>> >
>>> > The stack-trace for the first call is shown below:
>>> >
>>> > add:678, ManagerBase (org.apache.catalina.session)
>>> > setId:358, StandardSession (org.apache.catalina.session)
>>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>>> > setId:345, DeltaSession (org.apache.catalina.ha.session)
>>> > createSession:719, ManagerBase (org.apache.catalina.session)
>>> > createSession:422, DeltaManager (org.apache.catalina.ha.session)
>>> > createSession:410, DeltaManager (org.apache.catalina.ha.session)
>>> > doGetSession:3043, Request (org.apache.catalina.connector)
>>> > getSession:2441, Request (org.apache.catalina.connector)
>>>
>>> <snip/>
>>>
>>> This is the app triggering the creation of the session.
>>>
>>> > The stack-trace for the second call is shown below:
>>> >
>>> > add:678, ManagerBase (org.apache.catalina.session)
>>> > setId:358, StandardSession (org.apache.catalina.session)
>>> > setId:327, DeltaSession (org.apache.catalina.ha.session)
>>> > handleSESSION_CREATED:1322, DeltaManager
>>> (org.apache.catalina.ha.session)
>>> > messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)
>>>
>>> <snip/>
>>>
>>> This is the DeltaManager receiving notification that a new session has
>>> been created.
>>>
>>> > Any help would be appreciated. I can replicate this every time and
>>> spend
>>> > some time investigating this.
>>>
>>> The new session created message should be send to (and then processed
>>> on) every node *except* the node on which the session was originally
>>> created.
>>>
>>> Can you show us how you configured this cluster please?
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>>
>>>
