cmd.php dying in 0.8.6j

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Author
Message
luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

cmd.php dying in 0.8.6j

#1 Post by luxten » Fri Jan 19, 2007 12:56 pm

Hi all

Here I have CentOS 4.4, php 4.3.9, net-snmp 5.1.2.
I just upgraded to 0.8.6j to fix the recent security issues.
Immediately after it, my drawings ceased being updated, and I found that poller.php is timing out.
Here's my log:

01/19/2007 06:38:28 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:38:28 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
01/19/2007 06:38:29 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:38:29 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
...
LOTS of these lines...
...
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"
01/19/2007 06:39:51 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.
01/19/2007 06:39:51 PM - SYSTEM STATS: Time:293.0723 Method:cmd.php Processes:1 Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:97 RRDsProcessed:5
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:293.0723 Method:cmd.php Processes:1 Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:97 RRDsProcessed:5')"
01/19/2007 06:39:51 PM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_command"

I investigated the cause of this behaviur, and found that:
1) cmd.php dies without any warning soon after being run by poller.php.
2) poller.php tries forever to read poller_time, which cmd.php should update, but timeouts because cmd.php is just dead.
3) cmd.php dies when trying to ping a host (in this case localhost), more precisely it dies exactly when when it calls @snmpgetnext() in cacti_snmp_getnext() in snmp.php. In fact, the call to @snmpgetnext has been added in 0.8.6j.
4) @snmpgetnext is only available in PHP5, not in 4.
5) reverting snmpgetnext to snmpget as it used to be, solved the problem.

I think that in the code we should at least track the case of PHP4 and use snmpget instead of snmpgetnext;
and maybe poller.php should look for cmd.php dying abnormally, and report it in the log.
If cmd.php could die with some sort of debug message, I would appreciate it, but I really don't know if this can be done in PHP.

Comments are welcome.

Thank you for your attention
Luigi

cbcinnovations
Posts: 1
Joined: Fri Jan 19, 2007 6:06 pm
Location: Heath, OH
Contact:

#2 Post by cbcinnovations » Fri Jan 19, 2007 6:06 pm

Having the same issue... I am running CentOS 4.4, PHP 4.3.,NET-SNMP 5.3.1

-CBC-

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#3 Post by luxten » Sat Jan 20, 2007 2:57 am

I wrote that changing @snmpgetnext in @snmpget solved the problem. I had been optimistic. It only stops cmd.php to suddenly die.
I also had to change the line in ping.php saying:
$oid = ".1";
into:
$oid = ".1.3.6.1.2.1.1.3.0";
because if I do snmpget instead of snmpgetnext, I have to point to an available oid. I noticed I could have the same result using a retry count higher than 1, since the code switches to the ".1.3.6.1.2.1.1.3.0" oid when it retries.

So, a feature request is added to the ones in my previous post:
can't we have a new property for the host, saying what oid we should ask for pinging it, just as we can modify the SNMP port of the host?
In my opinion, it could be also helpful to have a similar property for the UDP port we use to ping the host, instead of the hard coded 33439. I have cacti in a firewalled environment, and could not use UDP ping because the firewall blocks this port.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#4 Post by TheWitness » Sat Jan 20, 2007 1:07 pm

1) Please verify if php_snmp is installed. Post your findings.
2) Please verify that you have a valid path the the binary snmpgetnext. Post your findings.
3) Please revert to ".1" and sniff utilizing Wireshark (Ethereal) and PM me the capture file.

This works perfectly fine on other systems.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#5 Post by luxten » Sat Jan 20, 2007 5:12 pm

TheWitness wrote:1) Please verify if php_snmp is installed. Post your findings.
It's installed. Sorry for not metioning it explicitly. The fact that with 0.8.6i (and with 0.8.6j, with the little mod I was talking in my previous post) I had no problems should nearly show that the problem is in some change between cacti 0.8.6i and 0.8.6j.
The thing (I think) is that in php-snmp version 4.3, snmpgetnext is not implemented. I had a look at php source code and, althought I'm not a programmer, I found no mention about it in the snmp module. Instead, in php version 5, snmpgetnext is implemented.
In 0.8.6i we did not use snmpgetnext, so there was no problem even using php 4.3. Now in 0.8.6j, since bug#0000886, we use snmpgetnext. But it is not implemented in php-snmp v4.3, and my cmd.php crashes.
TheWitness wrote:2) Please verify that you have a valid path the the binary snmpgetnext. Post your findings.
It's in /usr/bin, and it works. But I'm using snmp v1 so in lib/snmp.php I'm using SNMP_METHOD_PHP, and not SNMP_METHOD_BINARY.
TheWitness wrote:3) Please revert to ".1" and sniff utilizing Wireshark (Ethereal) and PM me the capture file.
I sniffed using the filter 'port 33439 or port 161'. The first being the UDP port used for UDP pinging, the second the SNMP port. Absolutely no packet was captured.
Not surprinsing for me, because of the way cmd.php works: it first pings the host, then it retrieves the data it needs. My cmd.php dies as soon as it tries to ping the host (the first is localhost) with a call to snmpgetnext (which is not implemented in php 4.3, and cmd.php suddenly dies).
If I should sniff using a different filter please ask me. My system is in production, so sniffing without a filter produces a lot of unuseful, and potentially private, infos.
<opinion>
I love network analyzers very much, but unfortunately in this case this kind of tool is of little help, since cmd.php dies just before sending its first packet.
</opinion>
TheWitness wrote:This works perfectly fine on other systems.
Hmm.. with cacti = 0.8.6j and PHP < 5? Because only in this situation this problem arises.
Another info: I tried to use both the options "Ping and SNMP" and "SNMP" for "Downed Host Detection" in the poller options. Since the SNMP ping is tried in both cases, cmd.php dies in both cases.

On my side, things return to work perfectly if I only apply the following patch that reverts to the old-style (pre-0.8.6j) SNMP ping style if we are on php < 5:

Code: Select all

--- lib/ping.php.lux    2007-01-20 08:49:25.000000000 -0500
+++ lib/ping.php        2007-01-20 12:21:05.000000000 -0500
@@ -289,6 +289,7 @@
                                return false;
                        }
 
+                       if (PHP_VERSION >= 5) {
                        $output = cacti_snmp_getnext($this->host["hostname"],
                                $this->host["snmp_community"],
                                $oid,
@@ -298,6 +299,18 @@
                                $this->host["snmp_port"],
                                $this->host["snmp_timeout"],
                                SNMP_CMDPHP);
+                       }else{
+                               /* no snmpgetnext here :( */
+                               $output = cacti_snmp_get($this->host["hostname"],
+                                       $this->host["snmp_community"],
+                                       ".1.3.6.1.2.1.1.3.0" ,
+                                       $this->host["snmp_version"],
+                                       $this->host["snmp_username"],
+                                       $this->host["snmp_password"],
+                                       $this->host["snmp_port"],
+                                       $this->host["snmp_timeout"],
+                                       SNMP_CMDPHP);
+                       }
 
                        /* determine total time +- ~10% */
                        $this->time = $this->get_time($this->precision);
Thank you.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#6 Post by TheWitness » Sat Jan 20, 2007 5:47 pm

Looking at the 4.4.4 source, it's there. I will go back to 4.3.9 and check as well.

TheWitness

Code: Select all

		} else if (st >= 2) {
			pdu = snmp_pdu_create(SNMP_MSG_GETNEXT);
			snmp_add_null_var(pdu, name, name_length);
		}
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#7 Post by TheWitness » Sat Jan 20, 2007 5:49 pm

It's also in php 4.3.9. Going back even further...
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#8 Post by TheWitness » Sat Jan 20, 2007 5:51 pm

It's in 4.3.0

Try the following. Create a and run a test script:

Code: Select all

<?php

$get_next = function_exists("snmpgetnext");

print "The getnext function '" . $get_next . "'\n";

?>
TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#9 Post by TheWitness » Sat Jan 20, 2007 5:52 pm

Hmm, even though it's in the source code, it does not appear to be in use.. :(
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#10 Post by TheWitness » Sat Jan 20, 2007 6:02 pm

Ok, try this...

TheWitness
Attachments
ping.zip
(3.94 KiB) Downloaded 558 times
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#11 Post by luxten » Sat Jan 20, 2007 6:30 pm

TheWitness wrote:It's in 4.3.0

Try the following. Create a and run a test script:

Code: Select all

<?php

$get_next = function_exists("snmpgetnext");

print "The getnext function '" . $get_next . "'\n";

?>
TheWitness
The output is:

Content-type: text/html
X-Powered-By: PHP/4.3.9

The getnext function ''




If I change snmpgetnext in snmpget I obtain:

Content-type: text/html
X-Powered-By: PHP/4.3.9

The getnext function '1'

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#12 Post by luxten » Sat Jan 20, 2007 6:35 pm

TheWitness wrote:Hmm, even though it's in the source code, it does not appear to be in use.. :(
I noticed that in the php source package, is the file ext/snmp/snmp.c. There I found the array snmp_functions. In this array, looking in php 5, I found snmpgetnext; in php 4.3.9, I did not.
A look to the function php_snmp and to the documentation seems to confirm that snmpgetnext support began in v5 of php.

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#13 Post by luxten » Sat Jan 20, 2007 6:56 pm

TheWitness wrote:Ok, try this...

TheWitness
This does not work as is, but maybe with
(version_compare("5.0", phpversion(), ">="))
you wanted to say
(version_compare("5.0", phpversion(), "<="))

With this change, the patch works. Similar to my little patch, seems we had the same idea :wink:

User avatar
TheWitness
Developer
Posts: 14804
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#14 Post by TheWitness » Sat Jan 20, 2007 7:47 pm

How does this work?

Larry
Attachments
ping.zip
(3.94 KiB) Downloaded 608 times
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

luxten
Posts: 7
Joined: Fri Jan 19, 2007 12:00 pm

#15 Post by luxten » Sun Jan 21, 2007 1:12 am

TheWitness wrote:How does this work?

Larry
Now it works.
Any hope to have this included in the mainstream package?

Thanks.

Post Reply