Re: [EXT] Datadog _ JMX Integration facing connection issues.
Sai Vamsi, > Am 13.12.2023 um 19:59 schrieb Chuck Caldarale : > > >> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) >> wrote: >> >> as you just asked ., >> I do have a process with Catalina. >> >> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn >> Active Internet connections (only servers) >> Proto Recv-Q Send-Q Local Address Foreign Address State >>PID/Program name >> tcp6 0 0 :::34753:::*LISTEN >>1/java >> tcp6 0 0 :::9109 :::*LISTEN >>1/java >> tcp6 0 0 :::10109:::*LISTEN >>1/java >> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C >> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep >> catalina >> root 744 0.0 0.0 6460 680 pts/1S+ 11:47 0:00 grep >> --color=auto catalina >> root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# > > you have to figure out WHY tomcat is not starting! There should be log files or error messages on the console. It seems you have put an error somewhere in any of the configfiles. It's not at all a question of the ports not being allocated. Take a step back and make tomcat launch again. After that we figure out where you have to set the options... Please detail how you start tomcat and show the output of startup (the beginning and last lines should be enough). Again, don't put any java options for tomcat in any global environment options (JAVA_OPTS, CATALINA_OPTS) in your shell. Only in setenv.sh . Peter > That shows only the grep process looking for catalina, not anything using > catalina. If Tomcat were actually running, you’d see something like this > (slightly reformatted for clarity): > > chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina > chuck16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM > 0:00.00 grep catalina > chuck16874 0.0 0.9 415316912 153296 s000 S12:53PM > 0:02.66 > /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java > -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dtest_port=9090 > -Dignore.endorsed.dirs= > -classpath > /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar > -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83 > -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83 > -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp > org.apache.catalina.startup.Bootstrap > start > > > - Chuck > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXT] Datadog _ JMX Integration facing connection issues.
Hai Chuck., Thanks for jumping in., thats what my concern is. as previously added by Chris, I have added my annotations in Catalina_opts ., Which is preferred by Java_Process for adding the annotations., But Here by using catalina_opts., they are not reflecting. where as adding them in java-tool-options., Makes a process with Tomcat and a process that listens on the mentioned port., but throws port is already binded., as its an environment variable, its already looking for a port, where as its already created., throws binding error. Yesterday ., I thought of adding both the formats, adding java_tool_opts and Catalina opts., inorder to utilize the previous thigns., as below - name: JAVA_TOOL_OPTIONS value: {{ .Values.deployment.javaVMMemoryArgument }} - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CATALINA_OPTS value: >- -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49151 -Djava.rmi.server.hostname=$POD_IP -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Xms2048M -Xmx10240M Java_tool_opts are as : -XX:+UseStringDeduplication -XX:+UseContainerSupport -Dcom.sun.management.jmxremote just added to initiate a JMX process., but nothing is working out., I would like to request you for further inputs cc :chris Thanks & Regards, -- SAI VAMSI .B Senior DevOps Engineer From: Chuck Caldarale Sent: Thursday, December 14, 2023 00:28 To: Tomcat Users List Subject: Re: [EXT] Datadog _ JMX Integration facing connection issues. > On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) > wrote: > > as you just asked ., > I do have a process with Catalina. > > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State > PID/Program name > tcp6 0 0 :::34753:::*LISTEN > 1/java > tcp6 0 0 :::9109 :::*LISTEN > 1/java > tcp6 0 0 :::10109:::*LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep > catalina > root 744 0.0 0.0 6460 680 pts/1S+ 11:47 0:00 grep > --color=auto catalina > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity): chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina chuck16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina chuck16874 0.0 0.9 415316912 153296 s000 S12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dtest_port=9090 -Dignore.endorsed.dirs= -classpath /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83 -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83 -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap start - Chuck
Re: 9.0.83 addSslHostConfig JMX Operation Regression (Sample Code Attached)
The object and operation I'm trying to address is Catalina --> ProtocolHandler --> --> --> operations --> addSslHostConfig. The parameters are an SslHostConfig object and the boolean value "true". The operation is "addSslHostConfig". The code I sent in the previous message works 100% of the time in 9.0.82. In 9.0.83, it works about 50% of the time. I can always query that the operation exists, but roughly half the time it will fail with a JMX exception saying that the operation does not exist. I am not positive, but I believe the behavior in 9.0.83 might have to do with the fact that the catalina java code now has a one argument and two argument variant of the same method. On Wed, Dec 13, 2023 at 10:27 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > Daniel, > > On 12/12/23 19:45, Daniel Skiles wrote: > > I apologize for it being a bit rough - it's what I was using to > > troubleshoot locally. > > > > import static java.util.Objects.nonNull; > > > > import java.lang.management.ManagementFactory; > > import java.util.Arrays; > > import java.util.HashMap; > > import java.util.Map; > > import java.util.Optional; > > import java.util.function.Predicate; > > > > import javax.management.MBeanInfo; > > import javax.management.MBeanOperationInfo; > > import javax.management.MBeanServer; > > import javax.management.ObjectName; > > > > import org.apache.logging.log4j.LogManager; > > import org.apache.logging.log4j.Logger; > > import org.apache.tomcat.util.net.SSLHostConfig; > > import org.apache.tomcat.util.net.SSLHostConfigCertificate; > > import org.apache.tomcat.util.net.SSLHostConfigCertificate.Type; > > > > @javax.annotation.ManagedBean > > public class MbeanFailure { > > private static final Logger LOGGER = LogManager.getLogger(); > > > > private static final String LOCALHOST = "127.0.0.1"; > > private static final String SUBTYPE = "subType"; > > private static final String ADD_SSL_HOST_CONFIG_OP = "addSslHostConfig"; > > > > private static final Predicate NOT_LOCALHOST = > Predicate.not(on > > -> > > > Optional.ofNullable(on).map(ObjectName::getCanonicalName).orElse("").contains(LOCALHOST)); > > private static final Predicate NOT_SUBTYPE = > Predicate.not(on > > -> > > > Optional.ofNullable(on).map(ObjectName::getCanonicalName).orElse("").contains(SUBTYPE)); > > > > @javax.annotation.PostConstruct > > public void run() throws Exception { > > final MBeanServer server = ManagementFactory.getPlatformMBeanServer(); > > > > final SSLHostConfig config = new SSLHostConfig(); > > > > config.setProtocols("TLSv1.2"); > > config.setHostName("test.test.com"); > > config.setCiphers("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"); > > > > final SSLHostConfigCertificate cert = new > SSLHostConfigCertificate(config, > > Type.UNDEFINED); > > > > config.addCertificate(cert); > > final Map references = > getObjectReferences(server, > > "ProtocolHandler"); > > > > references.forEach((s, op) -> invoke(s, op, ADD_SSL_HOST_CONFIG_OP, new > > Object[] {config}, new String[] > {SSLHostConfig.class.getCanonicalName()})); > > } > > > > public Map getObjectReferences(final MBeanServer > > server, final String discriminator) { > > > > final Map results = new HashMap<>(); > > > > final Predicate extendedFilters = > > NOT_LOCALHOST.and(NOT_SUBTYPE); > > > > final Optional candidate = server.queryNames(null, > > null).stream() > > .filter(on -> nonNull(on.getCanonicalName())) > > .filter(on -> on.getCanonicalName().contains(discriminator)) > > .filter(extendedFilters) > > .findAny(); > > > > candidate.ifPresent(on -> results.put(server, on)); > > > > return Map.copyOf(results); > > } > > > > public Object invoke(final MBeanServer server, final ObjectName > objectName, > > final String method, final Object[] params, final String[] signature) { > > try { > > //This should return addSslHostConfig(SSLHostConfig, boolean) > > final MBeanInfo info = server.getMBeanInfo(objectName); > > > > final MBeanOperationInfo methodInfo = Arrays.stream(info.getOperations()) > > .filter(i -> i.getName().equals(method)) > > .findAny() > > .orElseThrow(() -> new RuntimeException("Could not find method named" + > > method)); > > > > LOGGER.error("Found available operation {}", methodInfo); > > > > final Object result = server.invoke(objectName, method, params, > signature); > > return result; > > } catch (final Exception e) { > > throw new RuntimeException("Error invoking " + method + " with params " + > > Arrays.toString(params) + " and signature " + Arrays.toString(signature), > > e); > > } > > } > > } > > What objctName do you think you are addressing, here? What parameters > are you passing it and what types? What parameters and types are > expected by the operation you are trying to invoke? > > -chris > > > On Fri, Dec 8, 2023 at 4:55 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> Daniel, > >> > >> On 12/7/23 13:25, Daniel Skiles wrote: > >>> All, > >>> I've been doing some testing, and I'm pretty sure the a
Re: [EXT] Datadog _ JMX Integration facing connection issues.
> On Dec 13, 2023, at 10:36, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) > wrote: > > as you just asked ., > I do have a process with Catalina. > > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State > PID/Program name > tcp6 0 0 :::34753:::*LISTEN > 1/java > tcp6 0 0 :::9109 :::*LISTEN > 1/java > tcp6 0 0 :::10109:::*LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep > catalina > root 744 0.0 0.0 6460 680 pts/1S+ 11:47 0:00 grep > --color=auto catalina > root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# That shows only the grep process looking for catalina, not anything using catalina. If Tomcat were actually running, you’d see something like this (slightly reformatted for clarity): chuck@Chuck-MacBookPro apache-tomcat-9.0.83 > ps aux | grep catalina chuck16879 0.0 0.0 408626896 1376 s000 S+ 12:53PM 0:00.00 grep catalina chuck16874 0.0 0.9 415316912 153296 s000 S12:53PM 0:02.66 /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/chuck/Downloads/apache-tomcat-9.0.83/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dtest_port=9090 -Dignore.endorsed.dirs= -classpath /Users/chuck/Downloads/apache-tomcat-9.0.83/bin/bootstrap.jar:/Users/chuck/Downloads/apache-tomcat-9.0.83/bin/tomcat-juli.jar -Dcatalina.base=/Users/chuck/Downloads/apache-tomcat-9.0.83 -Dcatalina.home=/Users/chuck/Downloads/apache-tomcat-9.0.83 -Djava.io.tmpdir=/Users/chuck/Downloads/apache-tomcat-9.0.83/temp org.apache.catalina.startup.Bootstrap start - Chuck
Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Hello Christopher. as you just asked ., I do have a process with Catalina. root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::34753:::*LISTEN 1/java tcp6 0 0 :::9109 :::*LISTEN 1/java tcp6 0 0 :::10109:::*LISTEN 1/java root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ^C root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# ps aux | grep catalina root 744 0.0 0.0 6460 680 pts/1S+ 11:47 0:00 grep --color=auto catalina root@lab1workflow4scalsvc2zus1-deployment-659dd79df7-wg59g:/# Thanks & Regards, -- SAI VAMSI .B Senior DevOps Engineer From: Christopher Schultz Sent: Saturday, December 9, 2023 03:27 To: users@tomcat.apache.org Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues. Sai Vamsi, On 12/8/23 00:43, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote: > Hey Christopher., > Greetings of the day. > > >1. > Might I have confused you with posting the arguments directly ., Yeah as i > just shared you the annotations with comments , to state you the stuff i am > using., But in my deployment ., I am using them in catalina opts., and trying > to call them from values.yaml., which looks like this : > > javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication > -XX:+UseContainerSupport -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.local.only=false > -Dcom.sun.management.jmxremote.port=49151 > -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local > " > > and I am referring the word "javaVMMemoryArgument" from values yaml and > calling it in Catalina_opts, so that it would fetch all these > annotations as mentioned above, during the deployment. This is my deployment > part., where I am referring to the above values from values.yaml > env: > - name: CATALINA_OPTS >value: {{ .Values.deployment.javaVMMemoryArgument }} > > > > > >1. > Coming to Process., I have searched for Java process that listens on my > mentioned port ie., 49151, but none of the process is listening to that > process. > > I even tried with > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep LISTEN > tcp6 0 0 :::10109:::*LISTEN > 1/java > tcp6 0 0 :::9109 :::*LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > more > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State > PID/Program name > tcp6 0 0 :::10109:::*LISTEN > 1/java > tcp6 0 0 :::9109 :::*LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep ':443'netstat -tulpn | grep ':443'^C > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep ':49151' > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# > > which confirms me that , any of the process is being listening on the port > 49151. > >2. > I would like to request you to suggest me with a better approach ., where i > am missing anything in this process! Good question. What is pid #1? Do those port numbers make any sense for your Tomcat-based service? Is Tomcat even running? Try 'ps aux | grep catalina' to see if there are any. Are you launching Tomcat using catalina.sh / startup.sh or similar? Or are you running Tomcat "embedded" within your own application? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 9.0.83 addSslHostConfig JMX Operation Regression (Sample Code Attached)
Daniel, On 12/12/23 19:45, Daniel Skiles wrote: I apologize for it being a bit rough - it's what I was using to troubleshoot locally. import static java.util.Objects.nonNull; import java.lang.management.ManagementFactory; import java.util.Arrays; import java.util.HashMap; import java.util.Map; import java.util.Optional; import java.util.function.Predicate; import javax.management.MBeanInfo; import javax.management.MBeanOperationInfo; import javax.management.MBeanServer; import javax.management.ObjectName; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.apache.tomcat.util.net.SSLHostConfig; import org.apache.tomcat.util.net.SSLHostConfigCertificate; import org.apache.tomcat.util.net.SSLHostConfigCertificate.Type; @javax.annotation.ManagedBean public class MbeanFailure { private static final Logger LOGGER = LogManager.getLogger(); private static final String LOCALHOST = "127.0.0.1"; private static final String SUBTYPE = "subType"; private static final String ADD_SSL_HOST_CONFIG_OP = "addSslHostConfig"; private static final Predicate NOT_LOCALHOST = Predicate.not(on -> Optional.ofNullable(on).map(ObjectName::getCanonicalName).orElse("").contains(LOCALHOST)); private static final Predicate NOT_SUBTYPE = Predicate.not(on -> Optional.ofNullable(on).map(ObjectName::getCanonicalName).orElse("").contains(SUBTYPE)); @javax.annotation.PostConstruct public void run() throws Exception { final MBeanServer server = ManagementFactory.getPlatformMBeanServer(); final SSLHostConfig config = new SSLHostConfig(); config.setProtocols("TLSv1.2"); config.setHostName("test.test.com"); config.setCiphers("TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"); final SSLHostConfigCertificate cert = new SSLHostConfigCertificate(config, Type.UNDEFINED); config.addCertificate(cert); final Map references = getObjectReferences(server, "ProtocolHandler"); references.forEach((s, op) -> invoke(s, op, ADD_SSL_HOST_CONFIG_OP, new Object[] {config}, new String[] {SSLHostConfig.class.getCanonicalName()})); } public Map getObjectReferences(final MBeanServer server, final String discriminator) { final Map results = new HashMap<>(); final Predicate extendedFilters = NOT_LOCALHOST.and(NOT_SUBTYPE); final Optional candidate = server.queryNames(null, null).stream() .filter(on -> nonNull(on.getCanonicalName())) .filter(on -> on.getCanonicalName().contains(discriminator)) .filter(extendedFilters) .findAny(); candidate.ifPresent(on -> results.put(server, on)); return Map.copyOf(results); } public Object invoke(final MBeanServer server, final ObjectName objectName, final String method, final Object[] params, final String[] signature) { try { //This should return addSslHostConfig(SSLHostConfig, boolean) final MBeanInfo info = server.getMBeanInfo(objectName); final MBeanOperationInfo methodInfo = Arrays.stream(info.getOperations()) .filter(i -> i.getName().equals(method)) .findAny() .orElseThrow(() -> new RuntimeException("Could not find method named" + method)); LOGGER.error("Found available operation {}", methodInfo); final Object result = server.invoke(objectName, method, params, signature); return result; } catch (final Exception e) { throw new RuntimeException("Error invoking " + method + " with params " + Arrays.toString(params) + " and signature " + Arrays.toString(signature), e); } } } What objctName do you think you are addressing, here? What parameters are you passing it and what types? What parameters and types are expected by the operation you are trying to invoke? -chris On Fri, Dec 8, 2023 at 4:55 PM Christopher Schultz < ch...@christopherschultz.net> wrote: Daniel, On 12/7/23 13:25, Daniel Skiles wrote: All, I've been doing some testing, and I'm pretty sure the addSslHostConfig operation on ProtocolHandler is busted in 9.0.83. In versions prior to 9.0.82, you can call the operation with a single argument of type SSLHostConfig. In 9.0.82, that contract seems to have been broken, and you had to call it with two arguments: an SSLHostConfig and a boolean. In 9.0.83, it seems as though both operations are present, but which one is actually accessible at runtime is non-deterministic. This behavior presents through a direct invoke(...) call and via a JMX Proxy object instantiated through JMX.newMBeanProxy. I have attached a sample file that reproduces the behavior (sometimes, as it is nondeterministic). Is this a bug, or am I simply using the available feature incorrectly? If it is the former, how do I formally report this? If it is the latter, what is the "correct" way to call this operation from JMX? I think your attachment was stripped. Can it be posted in-line? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: user