Java JMX

JMX

All Java applications can expose metrics via JMX interface, some of them do it by default and for others it is obligatory to be configured specially for that. While JMX is great way to get statistics from Java servers, it requires Java to read it. This means, that we run Java program for each attempt to get stats or keep small Java program up and running all the time to read JMX, to convert to more common api and allow non java collector to get these stats.

It is not effective and quite expensive to start and stop JVM each time, when we need to get statistics. Better way is to keep JVM running all the time, but in any case its another running JVM, which seems more expensive that we want to.

To solve this problem we have decided to use Jolokia JVM agent, which can be attached to the running JVM as an agent and expose metrics via Json. {AGENT_HOME}/lib contains of Jolokia JVM agent.jarversion, which works perfectly for us, but if you need fresher or just another version of it, feel free to download JVM agent jar from https://jolokia.org/download.html and replace with it the existing agent.jar file.
After downloading your prefered version of Jolokia JVM agent, make sure that you have renamed it to agent.jar

Jolokia agent uses Java Attach API to attach agent.jar to running JVM and get statistics via HTTP/Json. The trick is, that newest JVM's will not allow other users to attach agent to JVM. So if you are running for example Tomcat on behalf of system user tomcat (Hope you are not running Java as root !), agent.jar should be attached via tomcat user as well.
To make this happen Agent must be able to execute java as user tomcat without password. Code below is an example of how /etc/sudoers will be configured :

oddeye    ALL = (tomcat) NOPASSWD: /opt/java/bin/java

Change oddeye , tomcat and /opt/java/bin/java to actual values for your system.

Install
cd ${OE_AGENT_HOME}/checks_enabled
ln -s ../checks_available/check_jmx.py ./
Configure

To attach agent to running JVM, Agent needs to know the main class name, which is used to run your java application server. If you already do not know main class name, you can get it via command below.

cd {AGENT_HOME}
java -jar lib/agent.jar list

Output of it should be :

{PID} {MAIN_CLASS} {Something_else}

15425   org.apache.catalina.startup.Bootstrap start  
28240   org.apache.hadoop.hbase.rest.RESTServer start
14759   org.apache.hadoop.hbase.thrift.ThriftServer start

Above are examples of main classes for Tomcat, HBase Thrift and HBase Rest. Typically main class name should be in second column of output. Edit conf/java.ini and put correct fields in section [JMX]

[JMX]
jmx: http://127.0.0.1:7777/oddeye/read
user: tomcat
class : org.apache.catalina.startup.Bootstrap
java : /opt/java/bin/java
Restart
${OE_AGENT_HOME}/oddeye.sh restart

If configuration is correct, after restarting Agent daemon, you would see that your existing Java program magically started to listen TCP:7777 and exposes JMX via HTTP/Json. You can change port to whatever you prefer, but do not change context path ot IP address.

Provides
Name Description Type Unit
jmx_daemonthreadcount Amount of running java daemon threads gauge None
jmx_gc_collectioncount Java garbage collections count counter None
jmx_gc_collectiontime Time spend on GC during last iteration rate Milliseconds
jmx_heap_committed Committed Java Heap Memory gauge Bytes
jmx_heap_max Max Java Heap Memory gauge Bytes
jmx_heap_used Used Java Heap Memory gauge Bytes
jmx_lastgcinfo Last GC duration gauge Milliseconds
jmx_nonheap_committed Committed Java Non Heap Memory gauge Bytes
jmx_nonheap_max Max Java Non Heap Memory gauge Bytes
jmx_nonheap_used Used Java Non Heap Memory gauge Bytes
jmx_peakthreadcount Peak amount of running Java Threads gauge None
jmx_threadcount Total amount of started threads counter None