I want to monitor my hadoop cluster services using check_jmx nagios plugin. I use following env. variables in the hadoop-env.sh file export HADOOP_OPTS=”-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false” # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS=”-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8004″ export HADOOP_SECONDARYNAMENODE_OPTS=”-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS -Dcom.sun.management.jmxremote.port=8005″ export HADOOP_DATANODE_OPTS=”-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS -Dcom.sun.management.jmxremote.port=8006″ export HADOOP_BALANCER_OPTS=”-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS -Dcom.sun.management.jmxremote.port=8007″ export HADOOP_JOBTRACKER_OPTS=”-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS -Dcom.sun.management.jmxremote.port=8008″ export HADOOP_TASKTRACKER_OPTS=”-Dcom.sun.management.jmxremote.port=8009″
but the problem I am facing is that my hadoop machine is behind firewall and I can't open multiple ports. The JMX RMI connector opens two ports: one is for the RMI registry, and it's the port that you usually supply with the -Dcom.sun.management.jmxremote.port=<port> property. The other port is used to export JMX RMI connection objects. This second port is usually dynamically allocated at random. So I am not able to connect using Jconsole or check_jmx plugin. I tried using example provided at * http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx*, by changing env. variable like this "export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Dexample.rmi.agent.port=3000 -javaagent:/root/install/asl-hadoop-0.20.1/lib/CustomAgent.jar $HADOOP_NAMENODE_OPTS"" The CustomAgent.jar file I created using above mentioned blog entry. Then when I start hadoop cluster using bin/start-all.sh I get following error. root/install/asl-hadoop-0.20.1/bin/hadoop-daemon.sh: line 96: 8983 Aborted nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null Create RMI registry on port 3000 Get the platform's MBean server Initialize the environment map Create an RMI connector server Start the RMI connector server on port 3000 service:jmx:rmi://domU-12-31-38-00-B4-F8:3000/jndi/rmi://domU-12-31-38-00-B4-F8:3000/jmxrmi Create RMI registry on port 3000 Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) localhost: starting datanode, logging to /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-datanode-domU-12-31-38-00-B4-F8.out localhost: starting secondarynamenode, logging to /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-secondarynamenode-domU-12-31-38-00-B4-F8.out starting jobtracker, logging to /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-jobtracker-domU-12-31-38-00-B4-F8.out localhost: starting tasktracker, logging to /root/install/asl-hadoop-0.20.1/bin/../logs/hadoop-root-tasktracker-domU-12-31-38-00-B4-F8.out can some help me, what I am doing wrong. Thanks, Viral.
