Alpha Testing Poller.php Performance

TheWitness · Posted: Fri Sep 15, 2006 10:12 am Post subject:

Ok, I would like you to see if the following has any "positive" impact. I am thinking that it should not, but if you can test, I would appreciate it:

Files:
lib/poller.php
lib/rrd.php

Regards,

Larry

fmangeant · Posted: Fri Sep 15, 2006 10:18 am Post subject:

I'll test this on monday morning and let you know.

N3NCY · Posted: Sun Sep 17, 2006 6:04 pm Post subject:

I gave this a try on my production FreeBSD 5.4 box running Cacti 0.8.6h.
This box is severly overloaded and it will be swapped out with a newer FreeBSD 6.1 server this Monday 9/18/2006.

Anyway, just for the heck of it I ran your test poller and rrd php files!

I am not sure if this will give you any insight into my heavily overloaded server:

09/17/2006 06:51:48 PM - SYSTEM STATS: Time:708.0388 Method:cactid Processes:8 Threads:128 Hosts:219 HostsPerProcess:28 DataSources:31303 RRDsProcessed:12050
09/17/2006 06:51:48 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
09/17/2006 06:50:20 PM - CACTID: Poller[0] FATAL: Cactid Fork process encountered a FATAL error and must exit
09/17/2006 06:50:20 PM - CACTID: Poller[0] MYSQL: Connection Failed: Too many connections
09/17/2006 06:50:20 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/17/2006 06:50:20 PM - CACTID: Poller[0] MYSQL: Connection Failed: Too many connections Warning: mysql_pcon...
09/17/2006 06:50:20 PM - CACTID: Poller[0] Host[83] DS[6904] WARNING: Result from SCRIPT not valid. Partial Result: Warning: mysql_pcon...
09/17/2006 06:50:19 PM - CACTID: Poller[0] FATAL: Cactid Fork process encountered a FATAL error and must exit
09/17/2006 06:50:19 PM - CACTID: Poller[0] MYSQL: Connection Failed: #08004Too many connections

Well, tomorrow brings a new day and a new server!

TheWitness · Posted: Sun Sep 17, 2006 6:42 pm Post subject:

N3NCY,

You are your own worst enemy:

Concurrent Processes = 8
Threads = 128
Script Server Processes = n
Pollers = 1

MySQL Connections Required = Pollers + (Concurrent Processes * (Scripts_Servers Processes + Threads))

In your case:

1 + (8 * (128 + n))

Assuming: Script Servers Processes = 1

1 + (8 * 129) aka (Way too many)

Regards,

Larry

N3NCY · Posted: Sun Sep 17, 2006 6:48 pm Post subject:

What settings can you recommend that I try?
Please be specific since this box will be replaced in the AM.

I would love to put correct settings, but do not quite get your meaning with the formula.

TheWitness · Posted: Sun Sep 17, 2006 6:49 pm Post subject:

Ernie,

Run the following from a sql command shell:

select count(action), action from poller_items group by action order by action;

Also,

How many processors? How much memory?

Larry

N3NCY · Posted: Sun Sep 17, 2006 7:02 pm Post subject:

mysql> use cacti
Database changed
mysql> select count(action), action from poller_item group by action order by action;
+---------------+--------+
| count(action) | action |
+---------------+--------+
| 31043 | 0 |
| 255 | 1 |
| 5 | 2 |
+---------------+--------+
3 rows in set (0.00 sec)

This box is FreeBSD 5.4 with 3.0Ghz P4 and 1GB RAM.

TheWitness · Posted: Sun Sep 17, 2006 7:05 pm Post subject:

1) What is your Max OID's Per Get Request set too? Increase it until you start to loose OID's. Most modern devices handle 50 ok, and not much more.
2) Upgrade to the Beta2 of Cactid
3) Concurrent Processes 2, Threads 30, Script Servers 1

TheWitness

N3NCY · Posted: Sun Sep 17, 2006 7:35 pm Post subject:

My max OID is 60.

I am still having issues compiling cactid beta 2.

I did dos2unix on every file.

I did try to compile similar to the non-beta cactid which works for me:

aclocal19
autoheader259
cp CHANGELOG ChangeLog
automake19
autoconf259
LD_LIBRARY_PATH=/usr/local/lib:/opt/net-snmp/lib/:/opt/mysql/lib/mysql/
export LD_LIBRARY_PATH
./configure --prefix=/opt/apache/htdocs/cacti --with-mysql=/opt/mysql/ --with-snmp=/opt/net-snmp/

<snip>
checking for mysql_thread_init in -lmysqlclient... yes
checking if UCD-SNMP needs crypto support... no
checking if Net-SNMP needs crypto support... no
checking for snmp_timeout in -lnetsnmp... no
Cannot find NET-SNMP libraries(snmp)... checking UCD-SNMP
checking for snmp_timeout in -lsnmp... no
configure: error: Cannot find UCD-SNMP libraries(snmp)

I cant configure so cant compile.

TheWitness · Posted: Sun Sep 17, 2006 7:37 pm Post subject:

Try this:

./configure --prefix=/opt/apache/htdocs/cacti --with-mysql=/opt/mysql --with-snmp=/opt/net-snmp

Larry

N3NCY · Posted: Mon Sep 18, 2006 12:48 am Post subject:

Hmm same deal awith the configure line you gave me (like mine but more correct without trailing slashes) I still get:
Cannot find NET-SNMP libraries(snmp)... checking UCD-SNMP
checking for snmp_timeout in -lsnmp... no
configure: error: Cannot find UCD-SNMP libraries(snmp)

I wish I could figure out what the configure script was looking for on net-snmp. Previous versions of cactid compile and find net-snmp no problem.

fmangeant · Posted: Mon Sep 18, 2006 3:10 am Post subject:

kwabbernoot · Posted: Mon Sep 18, 2006 4:50 am Post subject: Updated Poller

Hi,

I migrated the poller and rrd php scripts of my cacti-0.8.6i svn version (downloaded 18/09/2006) to the alpha versions.

Polling still works fine, but the performance increase is very limited. I added the stats of the cacti log file.

TheWitness · Posted: Mon Sep 18, 2006 6:02 am Post subject:

Yea, not enough to justify a change at this point. After reviewing the code in more detail, I noted that the rrdupdates would only "not" be optimal if you were running either cactid from quite a few releases ago or cmd.php.

If you are running the latest release of Cactid, which writes most, if not all data from Cactid to the poller_output at once, you would not see any performance gain. It is assumed that if you are running cmd.php, you don't care.

So, the net of this is that the performance gain opportunity does not substantiate the change.

TheWitness

TheWitness · Posted: Mon Sep 18, 2006 6:03 am Post subject:

Ernie,

Can you please ls your directory structure under /opt/net-snmp? Can you list it.

Larry