graph gaps

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Witness,
I have moved to the latest version of cactid 86d. I am running 4.3.9 and rrdtool 1.0.49 on Solaris 8. I am still getting these errors, they seem to randomly occur durning the cactid run:

01/27/2005 09:24:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 09:29:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 10:24:58 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 11:59:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 12:29:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 01:14:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 01:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 01:39:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 01:44:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 03:19:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
01/27/2005 04:19:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

Any other ideas??
Let me know if you need more info.

TheWitness · Posted: Thu Jan 27, 2005 5:16 pm Post subject:

Do you have YIM, MSM, AIM or access to NetMeeting?

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

try to reach me on aim- yrdbone
thanks

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I have run the gdb a number of times, each time, it runs successfully:

CACTID: Execution Time: 65.6162 s, Threads: 2, Hosts: 100

Program exited normally.

Cactid only bombs about 8-10 times out of the 288 runs per day. Is there a way for me to cron the run with gdb for debugging purposes?

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

After using the new version you gave me, I have seen a few errors:
02/10/2005 04:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
02/10/2005 04:54:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

It keeps hanging at different parts of the poll on the new cactid86e:

02/17/2005 01:24:57 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

i have gotten it to hang while running it in gdb, but gdb does not provide any output.

Here is what the ps looks like of the hanging PIDs:

bash-2.03$ ps -eaf | grep cactid
cacti 8694 4944 0 13:33:32 pts/4 0:00 grep cactid
cacti 6772 1 0 13:20:00 ? 0:03 /usr/local/cactid/bin/cactid 0 130
cacti 6775 6772 0 13:20:00 ? 0:00 /usr/local/bin/php /data/cacti-0.8.6c/script_server.php cactid 0

bash-2.03$ truss -p 6775
read(10, 0x00420648, 8192) (sleeping...)

bash-2.03$ truss -p 6772
Received signal #14, SIGALRM, in sigsuspend() [caught]
sigsuspend(0xFFBEF2D0) Err#4 EINTR
sigprocmask(SIG_SETMASK, 0xFEFEEFE8, 0x00000000) = 0
sigprocmask(SIG_SETMASK, 0xFEFFAD68, 0x00000000) = 0
setcontext(0xFFBEEC40)
sigprocmask(SIG_SETMASK, 0xFFBEF260, 0x00000000) = 0
sigaction(SIGALRM, 0xFFBEF1D0, 0x00000000) = 0
setitimer(ITIMER_REAL, 0xFFBEF340, 0x00000000) = 0
alarm(0) = 0
setitimer(ITIMER_REAL, 0xFFBEF350, 0xFFBEF340) = 0
sigaction(SIGALRM, 0xFFBEF1D0, 0xFFBEF300) = 0
setitimer(ITIMER_REAL, 0xFFBEF350, 0x00000000) = 0
sigprocmask(SIG_SETMASK, 0xFFBEF250, 0x00000000) = 0
Received signal #14, SIGALRM, in sigsuspend() [caught]
sigsuspend(0xFFBEF2D0) Err#4 EINTR
sigprocmask(SIG_SETMASK, 0xFEFEEFE8, 0x00000000) = 0
sigprocmask(SIG_SETMASK, 0xFEFFAD68, 0x00000000) = 0
setcontext(0xFFBEEC40)
sigprocmask(SIG_SETMASK, 0xFFBEF260, 0x00000000) = 0
sigaction(SIGALRM, 0xFFBEF1D0, 0x00000000) = 0
setitimer(ITIMER_REAL, 0xFFBEF340, 0x00000000) = 0
alarm(0) = 0
setitimer(ITIMER_REAL, 0xFFBEF350, 0xFFBEF340) = 0
sigaction(SIGALRM, 0xFFBEF1D0, 0xFFBEF300) = 0
setitimer(ITIMER_REAL, 0xFFBEF350, 0x00000000) = 0
sigprocmask(SIG_SETMASK, 0xFFBEF250, 0x00000000) = 0

....keeps looping

Any other ideas??

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Larry,
I am still seeing similar problems after that new version of cactid you gave me last night:

02/18/2005 11:44:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
02/18/2005 11:49:57 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
02/18/2005 12:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
02/18/2005 01:49:58 PM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.

bash-2.03$ ps -eaf | grep cactid
cacti 18266 1 0 13:45:01 ? 0:00 /usr/local/cactid/bin/cactid 0 131
cacti 25042 4944 0 14:33:10 pts/4 0:00 grep cactid
cacti 18269 18266 0 13:45:01 ? 0:00 /usr/local/bin/php /data/cacti-0.8.6c/script_server.php cactid 0
bash-2.03$ date
Fri Feb 18 14:33:12 EST 2005
bash-2.03$ truss -p 18266
lwp_sema_wait(0x00036190) (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xFE909E30) (sleeping...)
lwp_sema_wait(0xFE807E30) (sleeping...)
lwp_cond_wait(0xFEFF5548, 0xFEFF5558, 0xFEFEEDB0) (sleeping...)
door_return(0x00000000, 0, 0x00000000, 0) (sleeping...)
^Cbash-2.03$ truss -p 18269
read(10, 0x00420648, 8192) (sleeping...)

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I ran the cactid procces with a truss, and logged the output over the weekend. I have the logs of failed polls showing where they start to sleep. I dont understand the truss output, let me know if you want to take a look at it.
Thanks

TheWitness · Posted: Mon Mar 14, 2005 9:10 pm Post subject:

Yard,

I saw a post that the user that runs cactid needs to have it's home directory be the binary for cactid. Also, I have a new version of Cactid ready for testing at the following location:

http://forums.cacti.net/viewtopic.php?p=26992#26992

If you find it helps, please post there.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I am using the new version, only the error message that is causing the gaps has changed:

# ls -ltr /tmp/mysql.sock
srwxrwxrwx 1 mysql other 0 Sep 3 2004 /tmp/mysql.sock

03/17/2005 10:50:38 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
03/17/2005 10:55:44 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
03/18/2005 12:00:43 AM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
03/17/2005 11:55:29 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
03/18/2005 12:00:02 AM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
03/18/2005 12:35:18 AM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock'

TheWitness · Posted: Sat Mar 19, 2005 12:42 pm Post subject:

Looks like a permissions problem in MySQL, or you are running from windows and don't have the IP Address specified. In Windows, you must specify the IP Address.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

This is still my Solaris box. I am seeing graph gaps with the same amount of frequency as on the older cactid version (d). As far as I can tell the sql database has been up since september.

TheWitness · Posted: Mon Mar 21, 2005 3:08 pm Post subject:

Right, you have hung Cactid processes. Can you check, are you using the latest version? Please let me know.

Larry

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Yeah, I downloaded and am using that 86e version you posted last week. The error message in this version looks like a sql problem, is it possible this was the problem the whole time only the new version has more detailed logging?

TheWitness · Posted: Mon Mar 21, 2005 8:10 pm Post subject:

I think so. I removed the sql call today and reposted under Announcements. Give it a try.

The SQL call was associated with a timeout. My thoughts are that MySQL connections are closing during the middle of a subsequent poll and causing a segfault.

Larry