Submit Your CMD.PHP vs. SPINE Metrics Here

Important information about Cacti developments that all users should be interested in.

Moderators: Moderators, Developers

Post Reply
Author
Message
melchandra
Cacti User
Posts: 311
Joined: Tue Jun 29, 2004 12:52 pm
Location: Indiana
Contact:

#91 Post by melchandra » Fri Apr 21, 2006 8:32 am

knobdy wrote:
Amen - thanks! :)

oops, think it should be: grep -rHn <pattern> *
HAHA, you're entirely right. Oops.
Dave

knobdy
Cacti User
Posts: 495
Joined: Wed Sep 28, 2005 1:39 pm

#92 Post by knobdy » Tue Apr 25, 2006 10:14 am

I thought I posted this already (and even admitted to my idiotic misconfiguration of cactid the last time I tried to convert), but if I did it must have been the wrong forum area. Having run with cactid over night, here are the current results:

Code: Select all

04/25/2006 09:05:19 AM - SYSTEM STATS: Time:16.9793 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1453
04/25/2006 09:10:19 AM - SYSTEM STATS: Time:17.9754 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1510
04/25/2006 09:15:20 AM - SYSTEM STATS: Time:19.0551 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1569
04/25/2006 09:20:21 AM - SYSTEM STATS: Time:20.1951 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1569
04/25/2006 09:25:19 AM - SYSTEM STATS: Time:17.8080 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:30:20 AM - SYSTEM STATS: Time:18.3497 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:35:22 AM - SYSTEM STATS: Time:20.5984 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1525
04/25/2006 09:40:21 AM - SYSTEM STATS: Time:20.2469 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1554
04/25/2006 09:45:20 AM - SYSTEM STATS: Time:19.1965 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1512
04/25/2006 09:50:19 AM - SYSTEM STATS: Time:18.1279 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1558
04/25/2006 09:55:20 AM - SYSTEM STATS: Time:18.9790 Method:cactid Processes:2 Threads:8 Hosts:59 HostsPerProcess:30 DataSources:5309 RRDsProcessed:1554
How am I doing?

Here are some just prior to and after the change-over:

Code: Select all

04/24/2006 12:04:53 AM - SYSTEM MACTRACK STATS: Time:227.1363 ConcurrentProcesses:20 Devices:35
04/24/2006 04:01:00 PM - SYSTEM STATS: Time:57.8296 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:06:02 PM - SYSTEM STATS: Time:59.8644 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:11:03 PM - SYSTEM STATS: Time:61.5958 Method:cmd.php Processes:12 Threads:N/A Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:2638
04/24/2006 04:15:26 PM - SYSTEM STATS: Time:24.8374 Method:cactid Processes:12 Threads:8 Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:1569
04/24/2006 04:20:27 PM - SYSTEM STATS: Time:25.8656 Method:cactid Processes:12 Threads:8 Hosts:59 HostsPerProcess:5 DataSources:5309 RRDsProcessed:1569
04/19/2006 05:54:47 AM - SYSTEM MACTRACK STATS: Time:229.7355 ConcurrentProcesses:20 Devices:35
Oops, looking at this, it would seem I've got a problem still - over a thousand problems actually... http://forums.cacti.net/viewtopic.php?p=64338#64338

KaiB
Posts: 14
Joined: Sun Mar 19, 2006 11:04 am
Location: Norway

#93 Post by KaiB » Thu Apr 27, 2006 8:45 am

My results:

04/26/2006 12:01:28 PM - SYSTEM STATS: Time:27.8974 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 12:00:26 PM - SYSTEM STATS: Time:25.6139 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:59:29 AM - SYSTEM STATS: Time:29.1076 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:58:27 AM - SYSTEM STATS: Time:26.3604 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:57:26 AM - SYSTEM STATS: Time:26.1172 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

04/26/2006 11:07:40 AM - SYSTEM STATS: Time:39.0367 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:06:43 AM - SYSTEM STATS: Time:42.6524 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:05:34 AM - SYSTEM STATS: Time:33.5517 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:04:39 AM - SYSTEM STATS: Time:38.4495 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:03:35 AM - SYSTEM STATS: Time:35.2310 Method:cmd.php Processes:2 Threads:N/A Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

System:
2003 Server SP1
Dell PowerEdge SC 420
2.53GHz P4 Celeron
1GB RAM
Regards

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#94 Post by gandalf » Thu Apr 27, 2006 2:18 pm

KaiB wrote:My results:

04/26/2006 12:01:28 PM - SYSTEM STATS: Time:27.8974 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 12:00:26 PM - SYSTEM STATS: Time:25.6139 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:59:29 AM - SYSTEM STATS: Time:29.1076 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:58:27 AM - SYSTEM STATS: Time:26.3604 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148
04/26/2006 11:57:26 AM - SYSTEM STATS: Time:26.1172 Method:cactid Processes:2 Threads:2 Hosts:65 HostsPerProcess:33 DataSources:196 RRDsProcessed:148

System:
2003 Server SP1
Dell PowerEdge SC 420
2.53GHz P4 Celeron
1GB RAM
For such a decent machine, this seems to be slow (for me). Are you running lots of scripts?
Reinhard

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#95 Post by TheWitness » Thu Apr 27, 2006 3:56 pm

He needs to kick up the threads to somewhere from between 15-20.

Larry
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

KaiB
Posts: 14
Joined: Sun Mar 19, 2006 11:04 am
Location: Norway

#96 Post by KaiB » Fri Apr 28, 2006 1:55 am

I did try to increase the threads to 3, it got a couple of seconds faster but i experienced result loss for some reason (gaps in graphs to stable hosts). Not sure if something else interupted the polling at the same time. Il give it another go today. Woud be nice if I coud get it faster as the polling cycles overlap from the stress of the nightly backup.

When it comes to scripts, I run the Advanced Ping to almost all hosts and a couple of others to special hosts.
Regards

KaiB
Posts: 14
Joined: Sun Mar 19, 2006 11:04 am
Location: Norway

#97 Post by KaiB » Fri Apr 28, 2006 6:12 am

These where the highest stable results I achived.

04/28/2006 01:07:26 PM - SYSTEM STATS: Time:25.5384 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:06:26 PM - SYSTEM STATS: Time:25.5731 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:05:25 PM - SYSTEM STATS: Time:25.0394 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:04:25 PM - SYSTEM STATS: Time:25.0031 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154
04/28/2006 01:03:26 PM - SYSTEM STATS: Time:25.6107 Method:cactid Processes:4 Threads:3 Hosts:65 HostsPerProcess:17 DataSources:205 RRDsProcessed:154

More threads then this with any combination with processes resulted in cactid missbehaving. (POPEN timeouts and stable hosts marked as down.)

I can PM the log file if there is interest.
Regards

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#98 Post by TheWitness » Fri Apr 28, 2006 7:33 am

That is very interesting. What is your os and libc version? It appears that you are running many scripts. What type of scripts are they? If you have large quantities of scripts, you will have to watch load.

You can run 1 script per thread per process, so if you are running a perl environment that has a lot of overhead, it could cause timing issues as everything has to reload per script. Also, if you have concurrent processes to 3 and threads to 17, you could conceivably have 51 jobs in the run queue all attempting to reach a binary at the same time.

I am thinking that maybe I should limit the number of concurrent scripts to some level. Comments from long time listeners would be welcome.

This is the primary reason for the script server. In PHP5, I can load perl modules and run perl scripts much faster from PHP, without all the overhead, but it has never been a priority for me.

Thanks,

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#99 Post by gandalf » Sat Apr 29, 2006 6:45 am

KaiB wrote:I can PM the log file if there is interest.
More to check pollers performance to be found at
http://forums.cacti.net/viewtopic.php?t=9272 and
http://forums.cacti.net/viewtopic.php?t=12369
Reinhard

KaiB
Posts: 14
Joined: Sun Mar 19, 2006 11:04 am
Location: Norway

#100 Post by KaiB » Tue May 09, 2006 7:59 am

Sorry for the late reply, been very busy at work lately.
What is your os and libc version? It appears that you are running many scripts. What type of scripts are they? If you have large quantities of scripts, you will have to watch load.
The OS is 2003 SP1
Got no idea how check the "libc" version number :)

Atm im running about 50 php scripts and 10 perl scripts and when it comes to load its 100% while the poller runs.
More to check pollers performance to be found at
Got the poller graph installed but not the performance debug log analyzer, didnt get it to work a while ago, dont remember why.
Attachments
Cacti stats.png
Here's a week's graph:
Cacti stats.png (56.66 KiB) Viewed 5903 times
Regards

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#101 Post by TheWitness » Tue May 09, 2006 6:36 pm

Interesting how it climbs slowly and then drops again. Could that be at about the same time as a reboot. Do you have any correlation to the increased numbers and something going on on the system.

Have you looked at open TCP ports or concurrent logins? There are scripts for both in the scripts section for those.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#102 Post by gandalf » Wed May 10, 2006 11:27 am

Is it possible, that there are some scripts running on e.g. some log files that increase in size and are e.g. logrotated every day?
Reinhard

User avatar
cdukes
Cacti User
Posts: 61
Joined: Tue Mar 26, 2002 1:25 pm
Location: Morrisville, NC
Contact:

My Stats

#103 Post by cdukes » Wed May 10, 2006 8:43 pm

Hey guys,
Just trying to make this as efficient as possible, here are a couple samples...do you have any tuning recommendations for me?

Code: Select all

05/10/2006 08:37:23 PM - SYSTEM STATS: Time:440.0827 Method:cactid Processes:1 Threads:15 Hosts:774 HostsPerProcess:774 DataSources:5211 RRDsProcessed:3708

Code: Select all

05/10/2006 08:39:59 PM - SYSTEM STATS: Time:296.9150 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5649 RRDsProcessed:4023

Code: Select all

5/10/2006 08:42:55 PM - SYSTEM STATS: Time:171.9149 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5650 RRDsProcessed:4024
Thanks!

User avatar
cdukes
Cacti User
Posts: 61
Joined: Tue Mar 26, 2002 1:25 pm
Location: Morrisville, NC
Contact:

Re: My Stats

#104 Post by cdukes » Wed May 10, 2006 8:52 pm

Seems to have gone down considerably now that I changed to 3 processes...

Code: Select all

Time:59.5912 Method:cactid Processes:3 Threads:20 Hosts:774 HostsPerProcess:258 DataSources:5650 RRDsProcessed:4024

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#105 Post by gandalf » Thu May 11, 2006 1:39 pm

Recommendation is to choose about 2* (numbers of CPUs) as a starting point for #processes. Threads in the low 2-digit area should do.
Reinhard

Post Reply