Hey Christopher.,
Greetings of the day.
1.
Might I have confused you with posting the arguments directly ., Yeah as i just
shared you the annotations with comments , to state you the stuff i am using.,
But in my deployment ., I am using them in catalina opts., and trying to call
them from values.yaml., which looks like this :
javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication
-XX:+UseContainerSupport -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=49151
-Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
"
and I am referring the word "javaVMMemoryArgument" from values yaml and
calling it in Catalina_opts, so that it would fetch all these
annotations as mentioned above, during the deployment. This is my deployment
part., where I am referring to the above values from values.yaml
env:
- name: CATALINA_OPTS
value: {{ .Values.deployment.javaVMMemoryArgument }}
1.
Coming to Process., I have searched for Java process that listens on my
mentioned port ie., 49151, but none of the process is listening to that process.
I even tried with
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn |
grep LISTEN
tcp6 0 0 :::10109 :::* LISTEN
1/java
tcp6 0 0 :::9109 :::* LISTEN
1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn |
more
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp6 0 0 :::10109 :::* LISTEN
1/java
tcp6 0 0 :::9109 :::* LISTEN
1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn |
grep ':443'netstat -tulpn | grep ':443'^C
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn |
grep ':49151'
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/#
which confirms me that , any of the process is being listening on the port
49151.
2.
I would like to request you to suggest me with a better approach ., where i am
missing anything in this process!
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <[email protected]>
Sent: Thursday, December 7, 2023 23:33
To: [email protected] <[email protected]>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Hello Christopher.
> Thanks for the response!
>
> Let me walk you through the Environment.,
> I have a cluster and I am installing multiple microservices in the cluster.
> And one of the microservices I am integrating the JMX on of the
> microservices.,
> and the mentioned IP of the pod is being changed dynamically after each
> deployment .
>
> Here are the annotations that I am using in my deployment:
> - name: JAVA_OPTS
> # value: >-
> # -Dcom.sun.management.jmxremote
> # -Dcom.sun.management.jmxremote.authenticate=false
> # -Dcom.sun.management.jmxremote.ssl=false
> # -Dcom.sun.management.jmxremote.local.only=false
> # -Dcom.sun.management.jmxremote.port=49151
> # -Dcom.sun.management.jmxremote.rmi.port=49151
> #
> -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
If that's YAML, then all of the above is comments. Those lines do nothing.
> as i said , I have implemented these in three scenarios., Catalina_opts /
> Java_opts / Java_Tool_opts
You should definitely use CATALINA_OPTS and none of the other things you
mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
Tomcat-related scripts, and you'll find that even "shutdown" causes an
error... because it's trying to initialize JMX when shutting-down, which
is again useless.
> I have tried deploying catalina_ots and Java_opts as in the case of above
> mentioned.
> I also tried other way, mentioning all these in the values.yaml file ( I am
> using Helm Deployment to upgrade my deployments) and referring these
> annotations as , and mentioning them in the Value
> env:
> - name: CATALINA_OPTS
> value: {{ .Values.deployment.javaVMMemoryArgument }}
>
>
> getting netstat output from my pod:
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp6 0 0 :::10109 :::* LISTEN
> 1/java
> tcp6 0 0 :::9109 :::* LISTEN
> 1/java
>
> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep
> tomcat
> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep
> --color=auto tomcat
Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
'tomcat' does not appear in a normal JVM launch command for Tomcat.
> right now , I am passing my args with Catalina_opts in my deployment
> and I am not able to get any tomcat process pid while i check with
> netstat -apn , I can see multiple Java Process but not single tomcat process
>
> My Jmx port is being configured as a service., so that it could be
> bounded with Tcp protocol externally to the container., and allow all
> tcp connections.
>
> Hope I am able to provide you some detailed information., Please let
> me know., if I can help you with anything in further!
I think maybe you are both configuring your YAML file incorrectly and
also looking for the process incorrectly.
-chris
> ________________________________
> From: Christopher Schultz <[email protected]>
> Sent: Wednesday, December 6, 2023 20:40
> To: [email protected] <[email protected]>
> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>
> Sai Vamsi,
>
> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> I have added my annotations in Catalina_opts ., and I am not able to see any
>> Java Process with the JMX, neither the port is being listened.
>> It came back to previous stage where Annotations are not on the JVM .,
>> Came back to previous stage, where connection refused error., as follows.
>>
>> Could not initialize instance: tomcat-10.75.12.136-49151:
>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to
>> retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root
>> exception is java.rmi.ConnectException: Connection refused to host:
>> 10.75.12.136; nested exception is:
>> java.net.ConnectException: Connection refused (Connection refused)]
>> at
>> java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at
>> java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
>> at org.datadog.jmxfetch.App$5.invoke(App.java:927)
>> at
>> org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
>> at org.datadog.jmxfetch.App.init(App.java:917)
>> at org.datadog.jmxfetch.App.start(App.java:413)
>> at org.datadog.jmxfetch.App.run(App.java:173)
>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
>> at org.datadog.jmxfetch.App.main(App.java:91)
>
> Right: the DataDog server is unable to connect to your appication via
> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
> and that DataDog isn't running on that server. It's on some other IP, right?
>
>> Here we were not able to find any process , to confirm that atleast
>> port is being listened to.
> What is your exact CATALINA_OPTS value (masking any secrets that may be
> in there).
>
> Is Tomcat definitely running? Can you prove that with a "ps" listing,
> for example? That will also confirm it's running with the proper
> CATALINA_OPTS settings.
>
> Can you run "netstat" and show all of the ports your Tomcat process is
> listening to?
>
> I'm assuming that your Tomcat process is being hosted in some kind of
> container. Is the IP address 10.75.12.136 the IP of the container, or
> the IP of the host which hosts the container? Is your JMX port
> configured to be visible external to the container itself, or only
> internally to the container?
>
> -chris
>
>
>
>> Thanks & Regards,
>> --
>>
>> SAI VAMSI .B
>> Senior DevOps Engineer
>>
>> ________________________________
>> From: Christopher Schultz <[email protected]>
>> Sent: Tuesday, December 5, 2023 19:19
>> To: [email protected] <[email protected]>
>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>
>> Sai Vamsa,
>>
>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>> Thanks for the response..
>>>
>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead
>>> of Java_Opts and Java_tool_options.,
>>>
>>> But its not even working out., Java process is not getting created., and
>>> port is not bound to any process.
>>> Let me know if I am missing anything.
>>
>> Okay. Can you please summarize the current situation? I think you've
>> made a few changes, etc. so it would help to know the current state.
>>
>> Stop Tomcat and any other related services, delete all appropriate log
>> files, etc. and start Tomcat (or attempt to). Then post any output that
>> is relevant, and if you get things like "address in use" then please
>> post the details about which process id owns that port and the full
>> command-line that launched that process.
>>
>> Thanks,
>> -chris
>>
>>> ________________________________
>>> From: Christopher Schultz <[email protected]>
>>> Sent: Monday, December 4, 2023 23:22
>>> To: [email protected] <[email protected]>
>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
>>>
>>> Sai Vamsi,
>>>
>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>>>> Firstly thanks for adding a point me in asking me to check, if the
>>>> annotations are reflecting in the Java process, which opened me a door to
>>>> add the concerned annotations in correct place., by adding in
>>>> java_tool_options in stead of Java_opts.
>>>
>>> You will probably want to use CATALINA_OPTS instead of any of the other
>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't
>>> need to have the JMX subsystem starting-up and trying to grab a port.
>>>
>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching
>>> processes, like jps for example...
>>>
>>>> yeah they are reflecting and creating a Java Process.
>>>> but I am facing a problem here., while i am checking JSP, Thats : the
>>>> port i am using here to enable JMX is been opening a process with the
>>>> mentioned port and at the same time shows port is being used.:
>>>>
>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M
>>>> -XX:+UseStringDeduplication -XX:+UseContainerSupport
>>>> -Dcom.sun.management.jmxremote
>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>> -Dcom.sun.management.jmxremote.local.only=false
>>>> -Dcom.sun.management.jmxremote.port=49151
>>>> -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local
>>>> -javaagent:/datadog-lib/dd-java-agent.jar
>>>> -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh
>>>> -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException:
>>>> Port already in use: 49151; nested exception is:
>>>> java.net.BindException: Address already in use (Bind failed)
>>>
>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
>>> to your port which is aready bound by your Tomcat process.
>>>
>>> The solution is to use only CATALINA_OPTS to set these options.
>>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
