Cacti (home)ForumsRepositoryDocumentation
Cacti: offical forums and support
It is currently Wed Jul 23, 2014 2:51 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: [INFO] Graph Tomcat JMX mbeans
PostPosted: Thu Aug 12, 2010 7:37 pm 
Offline

Joined: Mon Aug 02, 2010 2:09 pm
Posts: 8
Hey guys,

I just finished working on graphing information provided by Tomcat's JMX connector. This post is a high-level write-up that will take you to the point of getting the data.

Ingredients:

    Tomcat with a JMX connector open
    Jmx4perl
    Bunch of Perl scripts
    Cacti


Note: all of this is done in a Linux environment

1. Open a JMX port on Tomcat

Edit catalina.sh and add this:

Code:
JAVA_OPTS="${JAVA_OPTS} \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=${REPLACE_WITH_PORT_NUM_HIGHER_THAN_1024} \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.password.file=${PATH_THAT_EXISTS}/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=${PATH_THAT_EXISTS}/jmxremote.access"


What this does: tells tomcat to open a non-ssl jmx connector on a specified port and use authentication.

For the authentication to work create the files mentioned in the $JAVA_OPTS:

Code:
host#echo 'guest readonly' > PATH_THAT_EXISTS/jmxremote.access
host#echo 'username pAssw0rd' > PATH_THAT_EXISTS/jmxremote.password


I'll let you figure out what the above does ;)

Note that the permissions on the above files should be 600 otherwise things will break (see catalina.out for exact error messages).


Start tomcat, wait for it to fire up completely and test that the JMX connector is up and running:

Code:
host#telnet localhost ${PORT_NUM_YOU_CONFIGURED_IN_JAVA_OPTS}
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.


Great, the port is open! Hold down Control-] and type quit to quit telnet.


2. Get Jmx4Perl from here: http://search.cpan.org/~roland/jmx4perl/ - and by the way, huge-huge-huge kudos to Roland, this Perl module is so great I was able to (mostly) ignore all of things for which I hate Perl while I was coding in Perl.

Install the dependencies. If you're on CentOS-5 here's a cookie:

Code:
CentOS5Host# yum install -y perl-Module-Build.noarch perl-JSON.noarch perl-Module-Find.noarch perl-Test-Deep.noarch perl-Config-General perl-Nagios-Plugin


Now expand the tarball and install Jmx4Perl (hint: look at the README file). Once you build it, find the j4p.war file in the source directory, it should live in the agent subdir.

Copy this file to ${TOMCAT_HOME}/webapps

WARNING: You just installed a jp4.war application into your tomcat. If you have an apache proxy that sends all of the requests to tomcat via mod_proxy then you just exposed http://yourhost/j4p to the world! This means that someone could potentially do nasty things with your JVM. Take the necessary steps to prevent this from happening before you continue!

Test it all:

Code:
[root@test01 ~]# jmx4perl http://localhost:8080/j4p info
Name:      Apache Tomcat
Vendor:    Apache
Version:   6.0.26
--------------------------------------------------------------------------------
Memory:
   Heap-Memory used    : 3 MB
   Heap-Memory alloc   : 10 MB
   Heap-Memory max     : 63 MB
   NonHeap-Memory max  : 116 MB
Classes:
   Classes loaded      : 1972
   Classes total       : 57083
Threads:
   Threads current     : 23
   Threads peak        : 35
OS:
   CPU Arch            : i386
   CPU OS              : Linux 2.6.18-8.el5xen
   Memory total        : 20 MB
   Memory free         : 20 MB
   Swap total          : 1055 MB
   Swap free           : 698 MB
   FileDesc Open       : 25
   FileDesc Max        : 1024
Runtime:
   Name                : 16283@test01
   JVM                 : 1.5.0_17-b04 Java HotSpot(TM) Client VM Sun Microsystems Inc.
   Uptime              : 8 d, 7 h, 32 m, 46 s
   Starttime           : Wed Aug  4 06:26:40 2010
[root@nagiostest01 ~]#


Got this far and everything is working? Great.

To see what information you just got access to do this:

Code:
host#jmx4perl http://localhost:8080/j4p list >mbeans
host#less mbeans


3. Now. Lets suppose we want to graph Heap Memory Usage. There are a couple of way to get the required metrics. We could just wrap around jmx4perl executable like so:
Code:
[root@test01 ~]# echo "heap_mem_committed: $(jmx4perl http://localhost:8080/j4p read MEMORY_HEAP_COMITTED)"
heap_mem_committed: 10694656
[root@test01 ~]#

From here on you create a data input method and all that jazz, plenty of tutorials out there on how to do that.

However, the above method is slow if you're going for many of mbeans (which, if you know why you're graphing tomcat metrics, you are) especially if you have 6-7 tomcats. That sit in a datacenter in California. And your Cacti box is in Arizona. Connected to the tomcat servers via a VPN. Getting my drift here?

The nice thing about Jmx4Perl is that it lets you write Perl scripts that can connect to your j4p agent and get the data much faster. Here's a (pretty ugly) example:

Code:
#!/usr/bin/env perl

## WARNING - THIS SCRIPT HAS BEEN PROVIDED AS AN EXAMPLE ONLY.
use strict;
use warnings;
use JMX::Jmx4Perl;
use JMX::Jmx4Perl::Alias;   # Import certains aliases for MBeans
use Getopt::Long;
use Data::Dumper;

my $url='http://localhost:8080/j4p';

my %opts = ();
my $result = GetOptions(\%opts,
                        'url=s' =>\$url,
                        "user|u=s","password|p=s",
                        "user=s","password=s",
                        "filter|f=s",
                        "verbose|v!",
                        "help|h!" => sub { &Getopt::Long::HelpMessage() }
                       );


my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});

my $OS=$jmx->get_attribute('java.lang:type=OperatingSystem',
                                ['AvailableProcessors',
                                'ProcessCpuTime',]
                                );
dumpVar($OS);

sub dumpVar{
               my $mbean = $_[0];
                while( my ($k, $v) = each %$mbean ) {
                print "$k:$v " ;
    }
}


The original version of the above produces something like this:
Code:
ProcessCpuTime:1741800000000 AvailableProcessors:4


Good times indeed. So all of the aforementioned should get you going. Hope this helps someone out there. :)


A very important note for the guys with tons of tomcats:
1. You will HAVE to roll your own data collection scripts
2. You're better off using the J4P proxy mode on one dedicated tomcat that talks to the JMX connectors on all of the tomcats that you want to monitor.


Further (and required) reading:

J4P Aliases: http://search.cpan.org/~roland/jmx4perl ... l/Alias.pm <-- this page has all of the aliases J4P knows of.
J4P Module doc: http://search.cpan.org/~roland/jmx4perl ... mx4Perl.pm <-- this page has _specific_ examples that show you how to get what you want from the J4P agents. Before you ask any questions about the Perl part - read this. Seriously.


Questions and comments are welcome.

Regards,
Ztron


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 13, 2010 2:57 am 
Offline

Joined: Mon Jul 06, 2009 2:37 pm
Posts: 7
Hi  Ztron,

very nice howto (and thanks for chosing jmx4perl ;-)

A minor comment: If you don't need the proxy mode (which is the case in this example) you don't need to setup a JMX remote connector at all. Simply deploy the agent, and everything's fine.

When using the proxy mode, JMX needs to be exposed on the target platforms but not on the proxy itself. However, if I would have the choice, I would always use the agent on the target platform, even for many tomcats, since this has some intrinsic advantages over the proxy mode. In other words, the proxy mode has some limitations (no real bulk requests, problems with custom data types, RMI serialization issues, ...). For more on this, see http://labs.consol.de/blog/jmx4perl/agentless-jmx4perl/

For enhanced security you could also create a policy file which allows access only to certain MBeans and/or from a dedicated IP/subnet. For the moment you have to repackage j4p.war to include this policy file, though. See http://search.cpan.org/~roland/jmx4perl-0.70/lib/JMX/Jmx4Perl/Manual.pod#ACCESS_POLICY. Setting up security will get easier in one of the next releases.

A final tip: With 0.70 there comes a jmx4perl shell (j4psh), which allows for interactive exploration of MBeans with a readline interface and context sensitive command completion (even on MBean names). This is often more comfortable to use than 'jmx4perl list'. There is no much documentation for this shell right now, though (but there is a small online help when running j4psh).

...roland


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Fri Aug 13, 2010 3:06 am 
Offline

Joined: Mon Jul 06, 2009 2:37 pm
Posts: 7
Another remark:

Quote:
Code:
...
my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});
...



The user and password given here are not used for authenticating against the JMX remote connector but for accessing the webapp j4p. Since in this example no security has been configured for j4p.war, the user and password here are simply ignored. The agent can be accessed by everyone.


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Fri Aug 13, 2010 5:02 pm 
Offline

Joined: Mon Aug 02, 2010 2:09 pm
Posts: 8
roland wrote:
Another remark:

Quote:
Code:
...
my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});
...



The user and password given here are not used for authenticating against the JMX remote connector but for accessing the webapp j4p. Since in this example no security has been configured for j4p.war, the user and password here are simply ignored. The agent can be accessed by everyone.



Absolutely right. This is just an excerpt from a script in my environment (I use the proxy mode because of unrelated considerations).

On another note, Roland, (and again, thank you for this wonderful module), can you please add a feature (or maybe an alias would be more appropriate?) to jmx4perl that would output mbean info in a format that Cacti can use? For example:

Code:
host#~jmx4perl http://myhost/j4p read --cacti 'memory pool whatever Usage'
HeapMemoryCommitted: 200000 HeapMemoryUsed: 200300 HeapMemoryMax: 500000
host#~


or..

Code:
host#~jmx4perl http://myhost/j4p  read HEAP_MEMORY_USAGE_CACTI
HeapMemoryCommitted: 200000 HeapMemoryUsed: 200300 HeapMemoryMax: 500000
host#~


That would be incredibly super-awesome.

Regars,
Z


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Sat Aug 14, 2010 12:42 am 
Offline

Joined: Mon Jul 06, 2009 2:37 pm
Posts: 7
ztron wrote:
On another note, Roland, (and again, thank you for this wonderful module), can you please add a feature (or maybe an alias would be more appropriate?) to jmx4perl that would output mbean info in a format that Cacti can use?


That's a fine idea, I will put it on my list. Probably an own command or integration into check_jmx4perl is more appropriate. The latter (although Nagios and Cacti are only loosely related) would have the advantage that I could use parameterized checks for Cacti out of the box. Parameterized checks are the successors of aliases in 0.70 and allows for providing one or more parameters for querying an MBean-Template (e.g. for answering the question: "HTTP request throughput of Webapplication 'bla'" where 'bla' is the parameter).


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Wed Mar 09, 2011 9:13 pm 
Offline

Joined: Wed Mar 09, 2011 8:07 pm
Posts: 1
There seems to be a growing need for historical data from JMX stuffed into a Cacti graph, but still without a 'good' way to do it. After poking around on the web it seems "jmx4perl -> Cacti" is the most promising.

Roland, have you decided yet if/when jmx4perl will natively support a data format Cacti understands?

Thanks for all of your hard work. You have really made the lives of countless ops people better, and sometimes, fun too.


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Thu Mar 10, 2011 7:59 am 
Offline

Joined: Mon Jul 06, 2009 2:37 pm
Posts: 7
I just uploaded a developer version jmx4perl 0.90_2 to CPAN which contains a new script 'cacti_jmx4perl'.
It is a stripped down version of 'check_jmx4perl' with all of its configuration features, but without threshold checking.

Please give it a try and let me know, whether you have any suggestions or remarks (or bug reports).
I expect to release 0.90 final in about one or two weeks.

For example:

Code:
$ cacti_jmx4perl --url http://localhost:8080/jolokia --alias MEMORY_HEAP_USED
15308376

$ cacti_jmx4perl -u http://localhost:8080/jolokia --config config/memory.cfg --check memory
Heap:15341168 Non-Heap:19450312


The later sample uses a configuration file for a multi-check, which comes with the jmx4perl distribution.

enjoy it ...
...roland


Top
 Profile  
 
 Post subject: Re: [INFO] Graph Tomcat JMX mbeans
PostPosted: Thu Mar 10, 2011 11:03 am 
Offline

Joined: Mon Jul 06, 2009 2:37 pm
Posts: 7
FYI, Jmx4Perl 0.90_3 has been pushed now to the CPAN Mirrors: http://search.cpan.org/~roland/jmx4perl-0.90_3


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group