Graph, partial failure

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Post Reply
Author
Message
akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

Graph, partial failure

#1 Post by akai » Thu Aug 03, 2006 6:51 am

I have some short 5min legs in the graph, but the host was always online in this time! So I think I have a config problem in cacti! The rrds files flapping see below, but I don't make anything in this time!?

HW:
ProLiant Xeon DL380R03 X2.8/400 512 1GB
6x36,4GB/15k Ultra 320 SCSI
So I think this is not the problem! ;-)

Cacti:
cacti Version 0.8.6h
Poller Cache [1508 Items]
SNMP Cache [30752 Items]
rrd: [1235 graphs]

cacti Version 0.8.6h
Cacti Settings (Poller)
Enabled: (yes) Enabled
Poller Type: cactid
Maximum Concurrent Poller Processes: 1
Maximum Threads per Process: 90
Number of PHP Script Servers: 2
Script and Script Server Timeout Value: 150
The Maximum SNMP OID's Per SNMP Get Request: 55
Downed Host Detection: Ping- Faster Option with Risk
Ping Type:[/b] UDP Ping
Ping Timeout Value: 1000
Ping Retry Count: 3
Failure Count: 2
Recovery Count: 2

Any solutions? :-?
Attachments
Local - Poller Statistics.jpg
Local - Poller Statistics.jpg (40.5 KiB) Viewed 4002 times
router.png
router.png (18.51 KiB) Viewed 4002 times
switches.png
switches.png (12.93 KiB) Viewed 4002 times

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#2 Post by gandalf » Thu Aug 03, 2006 1:59 pm

The runtime is quite insane. This is very near the 300 sec limit and may sometimes not come to an end. Also, the poller_output table should always be empty after a pollers's run. You may want to increase the number of processes to 2 (is it a multi processor system? Sorry, not that hardware guy), but reduce threads to about 15-20.
Are there other processes consuming CPU, memory or disk I/O? With cactid, I've seen loads of systems filling more that 4-5 of the numbers of rrd files in about a quarter of time ...
Reinhard

akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

#3 Post by akai » Fri Aug 04, 2006 1:26 am

The runtime is quite insane.
Some HD graphs locks insane!

This is very near the 300 sec limit and may sometimes not come to an end.
I know, but I have think that is normal by so many graphs...

but reduce threads to about 15-20.
When I go down, so I loss directly about 1/2 of the rrd graphs!

Are there other processes consuming CPU, memory or disk I/O?
No special thing that I know.
Attachments
hd.png
hd.png (17.09 KiB) Viewed 3966 times
localhost.png
localhost.png (36.75 KiB) Viewed 3966 times
rrd.jpg
rrd.jpg (18.36 KiB) Viewed 3966 times

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#4 Post by TheWitness » Fri Aug 04, 2006 5:52 am

Goto mysql and run the following:

select count(action) from poller_items group by action;

post the results. Also, reduce your script/script server timeout to some thing like 10 seconds. 150 is not prudent. If you have a script that takes that long to complete, you need to look at an alternate data collection method. Please start a separate thread related to "How" that is accomplished.

LVM, having asynchronous processes writing to the Poller Output table would be a great "How to". Also, having setting up a script/file to contain results that a poller script can simply read is another. I have written a few of these, I would be glad to share if you to the writeup.

Regards,

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

#5 Post by akai » Fri Aug 04, 2006 8:14 am

mysql> select count(action) from poller_item group by action;
+---------------+
| count(action) |
+---------------+
| 384 |
| 755 |
| 371 |
+---------------+
3 rows in set (0.01 sec)

I'ts ok. or bad?


How can I check the poller output table exactly?

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#6 Post by TheWitness » Fri Aug 04, 2006 12:04 pm

Sorry, can you rerun, my bad:

select action, count(action) from poller_items group by action order by action;

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#7 Post by gandalf » Sun Aug 06, 2006 9:47 am

akai wrote:The runtime is quite insane.
Some HD graphs locks insane!

This is very near the 300 sec limit and may sometimes not come to an end.
I know, but I have think that is normal by so many graphs...

but reduce threads to about 15-20.
When I go down, so I loss directly about 1/2 of the rrd graphs!

Are there other processes consuming CPU, memory or disk I/O?
No special thing that I know.
Did you first increase the number of processes? Recommended settings is 1-2 times the number of processors (well, HT processors count 2, do they?). What Larry is aiming at (I suppose) is to check, whether you're running a lot of scripts (they can increase runtime to a great extend)
Reinhard

akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

#8 Post by akai » Mon Aug 07, 2006 1:50 am

Processes: I have changed to 1 (1 processor server).

You are meaning Cacti scripts? Yes, the most of the graphs they are running, with support from scripts (ping, hd, poller....)!

Is it the only solution, to change the poller windows, from 5 to 10 or 15min running time, for more performance without any interrupts?

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#9 Post by TheWitness » Mon Aug 07, 2006 5:33 am

Akai,

This is your problem:
Script and Script Server Timeout Value: 150
Make it like 5 seconds or so. Also, could you run that second query and post.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

#10 Post by akai » Mon Aug 07, 2006 6:09 am

I have changed friday to 10s on your post, with no different.
The biggest different was, when I have "Maximum Threads per Process" to about 20!
50% of the graph I have loosed in 5min.

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#11 Post by TheWitness » Mon Aug 07, 2006 6:48 am

Can you dump and send me your DB?

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

akai
Posts: 7
Joined: Tue Feb 01, 2005 3:59 am

#12 Post by akai » Tue Aug 08, 2006 5:32 am

TheWitness:
You have a Email.

Post Reply