Broken graphs

Post support questions that relate to the Windows 2003/2000/XP operating systems.

Moderators: Moderators, Developers

Author
Message
shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Broken graphs

#1 Post by shahin » Mon May 28, 2018 7:38 am

Hi Guys,

We have setup cacti for sometimes now on one of our server 2008 R2 to monitor the network. untill last two weeks everything was working great. But now it looks like that we get some empty spaces in the graphs. Nothing has been changed on the Cacti server and the logs show no error at all. Any ide why we suddenly see broken graphs.

This is one of the graphs and we see the same for all of the servers that we monitor (see attachment)

Thaks
Attachments
Demosrv.JPG
Demosrv.JPG (40.89 KiB) Viewed 1338 times

cigamit
Developer
Posts: 2779
Joined: Thu Apr 07, 2005 3:29 pm
Location: B/CS Texas
Contact:

Re: Broken graphs

#2 Post by cigamit » Mon May 28, 2018 12:03 pm

Without seeing the logs for that time period, its difficult to tell. You really don't give us anything to go off of.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#3 Post by shahin » Tue May 29, 2018 2:35 am

Thaks for the tpdate,
Do you mean the IIS logs or logs of Cacti itself?
in the cacti logs only see this warning:
05/29/2018 05:10:01 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 516 seconds have passed since the last poll!

If you want I can also upload the portion of the IIS logs as well

Thanks

cigamit
Developer
Posts: 2779
Joined: Thu Apr 07, 2005 3:29 pm
Location: B/CS Texas
Contact:

Re: Broken graphs

#4 Post by cigamit » Tue May 29, 2018 11:23 pm

Just the Cacti logs is all that is needed. That one particular warning you have would most likely be the problem. Either your scheduled task is not kicking off in time, or your clock time is out of sync. Could be a few other issues, but those are the 1st 2 I would check.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#5 Post by shahin » Wed May 30, 2018 4:12 am

Thanks for your update,

I did check the graphs and it looks like that we again see an break in the graphs between 00:00 AM and 02:00 AM.
at the same time see some warnings in the cacti logs.
Also ren the php c:/cacti_web_root/cacti/poller.php and it shows everything is ok.
can you please tell us how can we first correct these issues?

Thanks
Attachments
cactipoller.JPG
cactipoller.JPG (37.99 KiB) Viewed 1310 times
cactilog1.JPG
cactilog1.JPG (261.24 KiB) Viewed 1315 times
graph1.JPG
graph1.JPG (42.12 KiB) Viewed 1315 times

cigamit
Developer
Posts: 2779
Joined: Thu Apr 07, 2005 3:29 pm
Location: B/CS Texas
Contact:

Re: Broken graphs

#6 Post by cigamit » Wed May 30, 2018 10:01 am

Your maximum run time is going over. If it always happens during a specific time frame, do you have backups or something else running at that time? You are going from normally taking 10 seconds to taking 5 minutes, so something is definitely slowing your server down during that period.

First step is to go into Settings and up the process count of cmd.php. Having just 1 process makes it really slow. cmd.php is super slow compared to spine, but you can deal with that later. I would up it to say, 10 for now.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#7 Post by shahin » Thu May 31, 2018 4:19 am

Thank you for the update,

sure at the time that we see the blanck space in graphs mostlly is back-up is running
I did change the Number of PHP Script Servers under settings-Poller to 10.
I will give it a try and because this does happen randomly, when I see the issue again I will let you know.

Thanks

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#8 Post by shahin » Mon Jun 11, 2018 3:46 am

Hi,

It looks like that increasing the Poller to 10 did not solve the issue.

Any sugestion?

Thanks

netniV
Cacti Guru User
Posts: 2784
Joined: Sun Aug 27, 2017 12:05 am

Re: Broken graphs

#9 Post by netniV » Mon Jun 11, 2018 9:44 am

There is an error saying that your poller task is out of sync with the scheduled task. This suggests that your tasks are still overrunning above or becoming hung for some reason (more than likely when trying to communicate with other systems).

Try turning up the logging levels to see if any more information is produced that might make it clearer.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#10 Post by shahin » Tue Jun 12, 2018 5:11 am

Hi,

I will enable the DeBug, statistics,Errors,results and I/O and will post the result here.

Thanks again.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#11 Post by shahin » Mon Jun 18, 2018 6:32 am

Hi,

I did enable the debuge logs but did not find any usefull information, or I dont know what I should look for!
Any suggestion on how to get rid of the wranning regarding the poller interval?

netniV
Cacti Guru User
Posts: 2784
Joined: Sun Aug 27, 2017 12:05 am

Re: Broken graphs

#12 Post by netniV » Tue Jun 19, 2018 2:34 am

There is an option within the Task Scheduler's task properties to say only run one instance (which is used by default). You can change that to be run another copy (or the language equivalent) but I think that would mean you have multiple processes running and polling at the same time.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#13 Post by shahin » Tue Jun 19, 2018 3:18 am

Hi,

This is what I could find in the logs:
Host[51] ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] RECACHE DQ[1] OID: .1.3.6.1.2.1.1.3.0, output: 23505264
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] DS[52] SNMP: v2: 10.0.10.232, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.12, output: 2042039928
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Host[54] DS[52] SNMP: v2: 10.0.10.232, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.12, output: 1045884375
06/16/2018 10:10:11 PM - CMDPHP: Poller[0] Time: 1.1433


CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (69, 'traffic_out', '2018-06-13 01:47:26', '1753281276')'
06/13/2018 01:55:01 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 481 seconds have passed since the last poll!
06/13/2018 01:55:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 6, Data Sources: traffic_in(DS[55]), traffic_out(DS[55]), traffic_in(DS[68]), traffic_out(DS[68]), traffic_in(DS[69]), traffic_out(DS[69])
06/13/2018 01:55:10 AM - SYSTEM STATS: Time:8.4062 Method:cmd.php Processes:10 Threads:N/A Hosts:34 HostsPerProcess:4 DataSources:68 RRDsProcessed:34
06/13/2018 02:00:19 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 318 seconds have passed since the last poll!
06/13/2018 02:00:30 AM

netniV
Cacti Guru User
Posts: 2784
Joined: Sun Aug 27, 2017 12:05 am

Re: Broken graphs

#14 Post by netniV » Fri Jun 22, 2018 2:50 am

shahin wrote:CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (69, 'traffic_out', '2018-06-13 01:47:26', '1753281276')'
This bit worries me. It would suggest to me a problem with your MySQL installation. Unless I'm missing something there, the command looks perfectly valid.

shahin
Posts: 11
Joined: Mon May 28, 2018 7:28 am

Re: Broken graphs

#15 Post by shahin » Mon Jun 25, 2018 8:04 am

Thank you for your update,

We are running the same Cacti installation for over 2 years now and we never had any problem with it. and now suddenly every day at the same time (backup time) we get this issue.
Any suggestion on how to findout if there is a problem with mysql?

Thanks

Post Reply