JMX Nagios

Last week I set off trying to understand how to tie JMX to Nagios. Almost all of the services we built at TomTom are based on Java and Spring, which makes exposing data through JMX a breeze. However, most of what we do to track the health of our systems is based on Nagios. So question is how to tie those two together.

While digging around, I stumbled across a couple of solutions:

1. Mogwy
2. Jmxquery
3. JBossNagiosPlugin
4. Check_jmx
5. Monju
   

So, I figured I'd give them a try.

Mogwy

Mogwy version 1.0.0 was released on 16 February, 2008. A minor upgrade was released the very same day, but fails to unzip for some reason.

Version 1.0.0 failed to connect to a local running JVM using the process id retrieved using jps. Exposing JMX remotely on the service to monitor got me a little further. However, when requesting the attributes of a specific MBean, it kept telling that the attribute was "Unknown".

It's not entirely clear how mogwy is supposed to be working. It does include a document explaining how to use it, but it could do with a little bit more detail. Take for instance the next fragment; it's not clear at all what is meant by "a specific logger".

"The messages are not actually sent by Mogwy, you have to pipe the Mogwy standard console output to a specific logger that will send the output to the MNode."

One of the things it promises is the ability to declaratively map MBean attributes to NCSA formatted data. Interesting stuff; it would be nice to see it working, but if it currently even fails to connect directly without using the declarative mapping, I'm not going to bother to give it a try.

The latest version of mogwy depends on Java 6. Looking at the sources, it turns out it does have some test coverage, but tags for released versions are missing.

All in all, it seems mogwy is an interesting tool, but it simply isn't ready yet.

Jmxquery

Jmxquery is based on a plugin on NagiosExchange. I don't consider it to be mature. This is what I got while trying to run it:

wilfred@proulx:jmxquery-read-only$ java -cp target/jmxquery-1.0-SNAPSHOT.jar jmxquery.JMXQuery
JMX UNKNOWN Required options not specified Usage: check_jmx -help 
wilfred@proulx:jmxquery-read-only$ java -cp target/jmxquery-1.0-SNAPSHOT.jar jmxquery.JMXQuery -help
JMX UNKNOWN null Usage: check_jmx -help 

It has never been released, there are no tests and the code seems to be seriously broken. The -help option is supposed to provide this:

Usage: check_jmx -U url -O object_name -A attribute [-K compound_key] [-I attribute_info] [-J attribute_info_key] -w warn_limit -c crit_limit [-v[vvv]] [-help]

, where options are:

-help 
   Prints this page

-U
   JMX URL, for example: "service:jmx:rmi:///jndi/rmi://localhost:1616/jmxrmi"

-O
   Object name to be checked, for example, "java.lang:type=Memory"

-A
   Attribute of the object to be checked, for example, "NonHeapMemoryUsage"

-K
   Attribute key for -A attribute compound data, for example, "used" (optional)

-I
   Attribute of the object containing information for text output (optional)

-J
   Attribute key for -I attribute compound data, for example, "used" (optional)

-v[vvv]
   verbatim level controlled as a number of v (optional)

-w
   warning integer value

-c
   critical integer value

-username
   monitorRole jmx username

-password
   jmx password

Note that if warning level > critical, system checks object attribute value to be LESS THAN OR EQUAL warning, critical
If warning level < critical, system checks object attribute value to be MORE THAN OR EQUAL warning, critical

But as long as it doesn't even provide this, I am not going to bother any further.

JBossNagiosPlugin

The JBoss Nagios Plugin is simply a tiny wrapper around twiddle, the commandline tool shipping with JBoss. Although it will be able to access all MBeans exposed by your service deployed in JBoss, it requires the JNDI service of JBoss to locate those MBeans. So it's not a general purpose remote JMX plugin. It's only capable of setting high watermarks. So you can make it generate critical alerts if a certain value exceeds a certain threshold. Not if it drops below a certain number.

Check_jmx

This is the plugin listed on NagiosExchange. It's clearly the foundation of the Jmxquery project hosted at code.google.com. The help is (almost) exactly the same. However, this time it actually seems to work:

./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:18050/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -vvvv -w 10000000 -c 20000000

gives

JMX CRITICAL HeapMemoryUsage.used=28391512

The only unfortunate thing is that it is impossible to find who created this package, and the plugin output is not totally in sync with the developer guidelines. Instead of the above result, it should have included a colon after the status:

JMX CRITICAL:HeapMemoryUsage.used=28391512

Monju

Monju is different from the previous options in a number of ways.

• First of all, it's only intended to be used with JBoss.
• It performs some adaption of raw JMX provided data serverside.
• So it requires modules to be plugged into your application server.
  

Monju supports different modes of operation. You can:

• Simply define a simple check based on a single MBean attribute. Monju will take care of turning this into Nagios acceptable output.
• You can write your own check and hotdeploy that to the server. Again, Monju will take care of taking the results and turning it into Nagios acceptable output.
• You can access JMX provided data easily using simple HTTP gets. Compiling this into Nagios acceptable output is left as an exercise.

I tested Monju, and it seems to work quite good. A couple of things I noted:
Again, the output doesn't seem to be formatted exactly according to the developer guidelines.

• There is no way to customize the message once it exceeds or drops below a certain threshold.
• The last version dates back from August 31, 2006.
• The project is currently inactive.
• It might not work for newer versions of JBoss.
  

Preliminary Conclusions

All of the projects listed above suffer from a lack of real traction. Many of the projects are dormant, and the ones that are not haven't released a real working version yet. If you are concerned about JBoss deployed components only, then their twiddle-based nagios plugin might be the thing you need. The disadvantage would be that you actually need a JBoss deployment on your Nagios monitoring node. Another disadvantage is that it will rely on RMI, which might not be the thing you really want in your server. In those cases, Monju might be an option. It seems to be working ok, and it will not require Java processes on your Nagios host.

If you are running other Java based middleware components in addition to JBoss, then the check_jmx plugin provided at the NagiosExchange site seems to be a working solution. I haven't checked if the fact that the colon is missing from the log message is posing any real problems, but I assume that it hasn't. Unfortunately, source code is not available.

Alternatively, you could start to think about adding other adapters to your Java systems. Nagios appears to have built-in support for SNMP. That way you could monitor your Java systems without having to rely on unsupported solutions. With the right plugins, WS-Management based connectors could provide another solution.