Lots of cuts in graph!!!

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Moderators, Developers

Author
Message
charan89
Posts: 1
Joined: Thu Jan 23, 2014 1:33 pm

Lots of cuts in graph!!!

#1 Post by charan89 » Thu Jan 23, 2014 1:39 pm

hi all..

Can someone help me.. i have almost 200host in my cacti and i find for few clients the graph cuts a lot... but actually there is no cut...

What should i do..?? The problem is not for all the host only for few..

Another problem is for a single host cacti doesnt record at all... Please help..

Thanks in advance...

idle
Cacti User
Posts: 76
Joined: Wed May 26, 2004 10:49 am
Location: Barcelona
Contact:

Re: Lots of cuts in graph!!!

#2 Post by idle » Fri Jan 24, 2014 2:49 pm

Same here.
Started at 21th.
Attachments
cuts.png
cuts.png (47.15 KiB) Viewed 2529 times

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#3 Post by winni » Mon Jan 27, 2014 5:46 am

Welcome to the club. We've been having problems like this for years now. It gets a bit better when you use five minute polling instead of one minute polling, but there will still be white gaps and the "bar codes" don't entirely disappear.

I don't know what the problem is or how to fix it. And when I see how many times people post similar issues here on the forum - and don't get an answer that would fix the problem - I'm getting the feeling that this is something that one simply has to live with when you use Cacti.

What would interest me at this point is if running Cacti on, let's say, Windows Server would improve the situation or make it even worse. We're still running Cacti on two Ubuntu 12.04 LTS x64 servers here, and I'd like to rule out the operating system as a probable cause.

So is anybody running Cacti in a rather large, global SATELLITE BASED network and does NOT have gaps or "bar code" graphs? And if so, what hard- and software are you using?

idle
Cacti User
Posts: 76
Joined: Wed May 26, 2004 10:49 am
Location: Barcelona
Contact:

Re: Lots of cuts in graph!!!

#4 Post by idle » Mon Jan 27, 2014 6:48 am

winni wrote:Welcome to the club. We've been having problems like this for years now.
Actually I'm using cacti for decade now, and never faced this one.

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#5 Post by winni » Mon Jan 27, 2014 7:45 am

Actually I'm using cacti for decade now, and never faced this one.
That is reassuring to hear!

May I ask how many hosts/devices you are monitoring and how many data sources do you query?
Do you use one minute or five minute polling?
What operating do you use? CentOS? Ubuntu? Windows? Any other?
Do you use Apache or nginx? Or IIS?

How many CPU cores and RAM does your server have?
Do you use SSDs or hard disks?

Any of this information might be very helpful for others to also get proper graphs. For reference, we're monitoring 247 hosts with 2251 graphs on one of our servers, which is an Octo-Core Xeon machine 16 GB RAM and an OCZ 256GB SSD. It's connected via One-Gig-Ethernet directly to one of our backbone switches. It runs on one minute polling and, according to the logs, Spine finishes a polling cycle in 21 seconds. Cacti's web interface is awfully slow on that server and although the graphs don't look as bad the one's that the original poster published here, there are white gaps every 9.5 hours that last for three minutes:

Image

As you can see, there are not only white gaps, but also spikes that should not be there.

We also occasionally have "bar code"-style graphs as the OP, but this can usually be fixed by deleting and re-creating a host.

Our second Cacti server only uses a Quad Core Xeon CPU with only 4 GB RAM and an HDD instead of an SSD, but it uses five minute polling and interestingly enough does not show the gaps. But it also shows drops in the graph that shouldn't be there:

Image

Frankly speaking, I no longer know what screws I can turn to fix this thing. I'd really appreciate any more detailed information about your working setup, maybe it will help us to fix our system.

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#6 Post by winni » Wed Jan 29, 2014 9:16 am

Just as an update, all of a sudden three of our switches began to produce "bar code" graphs last night:

Image

Lowb
Posts: 5
Joined: Tue Apr 16, 2013 5:04 am

Re: Lots of cuts in graph!!!

#7 Post by Lowb » Thu Jan 30, 2014 10:30 am

Im in the same basket, having cuts in the last graphs added to my system

User avatar
phalek
Cacti Guru User
Posts: 2739
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Lots of cuts in graph!!!

#8 Post by phalek » Thu Jan 30, 2014 10:42 am

I honestly urge everyone running a cacti system to install the Cacti poller template:

http://docs.cacti.net/usertemplate:host:cacti:poller

Gaps can have multiple causes, including:
- High load on a polled System ( i have that when there's backups runing on linux systems -> SNMP has a very low prio on these )
- Poller time exceeds the 300/60 seconds max runtime ( The load is distributed so some graphs may just see this causally, some other will always be hitting this )

The later one can be tracked down with that template. And some scripts causing a high runtme can actually be heavily optimized by caching mechanisms.

E.g. why getting data points from a system one-by-one instead of going in, getting everything at once and then just process the data locally ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#9 Post by winni » Fri Jan 31, 2014 6:52 am

Hi phalek,

Thanks for the suggestions - I've installed the poller template on both servers and will keep an eye on the result over the next hours.
E.g. why getting data points from a system one-by-one instead of going in, getting everything at once and then just process the data locally ?
Because, unfortunately, a bunch of the devices that we're monitoring, e.g. certain satellite modems, will only return one value at a time and don't respond to bulk requests.

However, I'm not convinced that the poller itself is the cause of the problems - according to the log files, both machines finish the polling cycle in less than a minute.

I'm beginning to suspect that our Juniper firewall cluster might cause some of the "bar code" style graphs, but at this point it's more of a gut feeling than something that I can actually prove.

In any case, that still doesn't fix the other problem that both of our servers have - the ultra slow response times of the web interface. The installation on the SSD server initially was very fast, but then rapidly degraded over the weeks.

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#10 Post by winni » Fri Jan 31, 2014 7:06 am

What is the unit behind "Export Duration"? Seconds? Milliseconds? The server currently shows an export duration of 792.55, but I don't know what this number is supposed to tell me.

User avatar
phalek
Cacti Guru User
Posts: 2739
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Lots of cuts in graph!!!

#11 Post by phalek » Fri Jan 31, 2014 8:14 am

According to the PHP code, this would be seconds.

list($micro,$seconds) = split(" ", microtime());
$start = $seconds + $micro;
list($micro,$seconds) = split(" ", microtime());
$end = $seconds + $micro;

ExportDuration = round($end - $start,4):
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061

User avatar
phalek
Cacti Guru User
Posts: 2739
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Lots of cuts in graph!!!

#12 Post by phalek » Fri Jan 31, 2014 8:15 am

Is this a pure SSD installation ?

Did you start a "top" command and look at it through the poller run ?

Is your web-interface always slow, or just when the poller runs ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061

User avatar
winni
Posts: 24
Joined: Wed Aug 22, 2012 6:35 am
Location: Germany
Contact:

Re: Lots of cuts in graph!!!

#13 Post by winni » Fri Jan 31, 2014 9:01 am

Oh, okay. So exporting 18.06k graphs in almost 12 minutes is something I should be concerned about, is that right?

User avatar
phalek
Cacti Guru User
Posts: 2739
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Lots of cuts in graph!!!

#14 Post by phalek » Fri Jan 31, 2014 9:27 am

not if your web-interface is responding fine and you don't have gaps ...

or this is being done just once a day or at least hourly.
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061

User avatar
Tolaris
Posts: 17
Joined: Mon Nov 15, 2010 3:46 pm
Location: London, UK
Contact:

Re: Lots of cuts in graph!!!

#15 Post by Tolaris » Fri Jan 31, 2014 11:47 am

Winni, are you by chance using Boost? I have found Boost actually reduces web interface performance when used with SSDs. The extra caching doesn't help much when you have fast write cycles.

Post Reply