graph gaps

Morgan-guest · Guest

still getting graph gaps.

running latest cacti 086 and cactid from the 17th

Guest · Posted: Fri Aug 20, 2004 11:54 am Post subject: more logs

TheWitness · Posted: Fri Aug 20, 2004 6:33 pm Post subject:

08/20/2004 05:35 PM - CACTID: Poller[0] MYSQL: Connection Failed: Lost connection to MySQL server during query

There is a clue. Take the popen calls out of your beta (Perl or PHP Queries or Script Queries) and see if the problem corrects itself.

TheWitness

Morgan · Cacti User Joined: 25 Feb 2004 Posts: 187

i've removed the only script i had running
scripts/unix_processes.pl

i'm adding all my hosts into the db today, we'll see how it behaves.

i'm goign to switch to icmp pings as some of my routers don't like udp pings

TheWitness · Posted: Sat Aug 21, 2004 7:47 am Post subject:

Morgan,

Please insure that you are not running PHP 4.3.4. There is a problem with that version of PHP that is causing issues with poller.php. I will be uploading a temporary workaround today.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Is there still a problem with php 3.4.3? I am getting graph gaps that i was hoping would be solved bu upgrading to 8.6b. Similar errors:
11/19/2004 04:51:53 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/19/2004 04:51:53 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/19/2004 06:02:15 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

TheWitness · Posted: Fri Nov 19, 2004 8:32 pm Post subject:

I have posted a new cactid.c file to CVS that does correct an issue identified by a Solaris tester. When you are browsing CVS, make sure you choose the BRANCH_0_8_6. Otherwise, the MAIN cactid.c will break your installation. With the new cactid.c remake Cactid and let me know if you fair any better.

In addition, please post 1 pass of debug output to the message board.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I am not noticing anymore graphs, however it looks like there are some hanging processes after poller.php is finished:

# ps -eaf | grep cacti
cacti 29240 29237 0 12:40:01 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 21703 21326 0 Nov 18 pts/1 0:00 -sh
root 501 15978 0 17:26:24 pts/2 0:00 grep cacti
cacti 5149 1 0 17:50:01 ? 0:02 /usr/local/cactid/bin/cactid 0 98
cacti 825 1 0 Nov 22 ? 0:02 /usr/local/cactid/bin/cactid 0 98
cacti 29320 1 0 07:55:01 ? 0:02 /usr/local/cactid/bin/cactid 0 98
cacti 4418 22079 0 13:28:20 pts/1 0:00 tail -f cacti.log
cacti 29237 1 0 12:40:01 ? 0:02 /usr/local/cactid/bin/cactid 0 98
cacti 9184 1 0 23:35:00 ? 0:02 /usr/local/cactid/bin/cactid 0 98
cacti 828 825 0 Nov 22 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 22079 21703 0 Nov 18 pts/1 0:01 bash
cacti 5152 5149 0 17:50:01 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 9187 9184 0 23:35:00 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 29323 29320 0 07:55:01 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 20777 20774 0 16:00:01 ? 0:00 /usr/local/bin/php /data/cacti86/script_server.php cactid 0
cacti 20774 1 0 16:00:01 ? 0:02 /usr/local/cactid/bin/cactid 0 98
# date
Tue Nov 23 17:26:47 EST 2004

TheWitness · Posted: Tue Nov 23, 2004 5:59 pm Post subject:

What is your PHP version again?

Larry

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Its php 4.3.4 on Solaris8. I have not noticed any more graph gaps, but i do continue to see the hanging processes. Let me know if you need any more info.

TheWitness · Posted: Tue Nov 30, 2004 6:16 pm Post subject:

Did you apply the cactid.c patch? You may have to experiement with different versions of PHP in Solaris. I would not recommend PHP 4.3.4 due to numerous problems during development.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I have applied the cactid patch, that did not help, now I have moved from php3.4.3 to PHP 4.3.10RC1 and i am still getting failures. Here is some of the stuff I am seeing in the logs. Let me know if you want me to extract more info, or try something.
Thanks

grep POLL cacti.log:

12/04/2004 09:14:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 10:05:49 AM - POLLER: Poller[0] Host[55] WARNING: Recache Event Detected for Host
12/04/2004 10:14:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 10:24:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 11:14:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 11:34:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 12:09:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 12:15:52 PM - POLLER: Poller[0] Host[54] WARNING: Recache Event Detected for Host
12/04/2004 12:39:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 12:44:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 01:54:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 01:59:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 02:29:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 02:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 03:19:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 04:19:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 05:29:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 05:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 05:54:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 06:44:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 07:09:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 08:29:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 10:49:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 11:14:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/04/2004 11:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 01:19:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 02:14:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 03:39:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 03:44:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 04:09:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 05:59:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 06:44:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 07:29:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 08:04:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 08:34:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 08:49:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 10:59:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 11:00:50 AM - POLLER: Poller[0] Host[56] WARNING: Recache Event Detected for Host
12/05/2004 12:09:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 12:34:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 01:09:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 03:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 03:34:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 03:59:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 05:54:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 06:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 07:09:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 07:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 08:39:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 08:59:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/05/2004 09:45:52 PM - POLLER: Poller[0] Host[91] WARNING: Recache Event Detected for Host
12/05/2004 10:24:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 01:09:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 02:04:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 02:14:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 02:34:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 03:05:51 AM - POLLER: Poller[0] Host[46] WARNING: Recache Event Detected for Host
12/06/2004 03:29:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 03:55:49 AM - POLLER: Poller[0] Host[46] WARNING: Recache Event Detected for Host
12/06/2004 04:44:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 05:34:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 07:49:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 07:54:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 08:24:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 08:34:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 09:09:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 11:19:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 11:39:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 11:54:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 12:10:50 PM - POLLER: Poller[0] Host[29] WARNING: Recache Event Detected for Host
12/06/2004 12:24:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 12:34:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
12/06/2004 12:55:50 PM - POLLER: Poller[0] Host[7] WARNING: Recache Event Detected for Host

grep PHP cacti.log | grep ERROR

12/06/2004 01:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 02:43:01 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 02:43:01 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 02:43:01 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 03:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 04:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 05:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 08:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 08:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 08:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 08:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 09:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 11:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 11:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 12:43:00 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 12:43:00 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
12/06/2004 12:43:00 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

TheWitness · Posted: Mon Dec 06, 2004 2:23 pm Post subject:

Looks like Cactid is segfaulting/crashing.

TheWitness

TheWitness · Posted: Mon Dec 06, 2004 2:25 pm Post subject:

Can you please run cactid from the command line in debug for a few polls to verify output and where the app is crashing. Please post results.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

There seem to be less gaps, however, i find they seem to occur at different parts of the run, but always similar to this:

12/08/2004 03:30:06 AM - POLLER: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.0.42/bin/rrdtool update /data/cacti86/rra/intgw2_lga2_ds0_296.rrd --template ds0:ds1 1102494600:478350:0
12/08/2004 03:34:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

It seems to hang radomly during rrdtool updates. I then pkill these once an hour, and this line appears in the log:

12/07/2004 11:43:00 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

I attached a file of a good run, then a bad run, the bad run craps out at 3:15.