Submit Your CMD.PHP vs. SPINE Metrics Here

Important information about Cacti developments that all users should be interested in.

Moderators: Moderators, Developers

Post Reply
Author
Message
User avatar
zorrosam
Cacti User
Posts: 244
Joined: Thu May 03, 2007 3:17 pm
Location: Italy

Re: advice on my poor performance?!

#271 Post by zorrosam » Mon Jun 22, 2009 2:56 am

zorrosam wrote:Hi guys,

any advice for my situation?

06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065

DELL Poweredge 1950
2xQUADCORE 2.00 GHZ XEON
8GB RAM
2xHDD 250GB RAID1
MYSQL ACTIVE ON ANOTHER SERVER TUNED

I don't know if will be better activate boost on my side ... the system is on production and fundamental for all my bussiness. There's the way for do that without downtime?

bye

sam
the version of cacti is 0.87b!

User avatar
zorrosam
Cacti User
Posts: 244
Joined: Thu May 03, 2007 3:17 pm
Location: Italy

Re: advice on my poor performance?!

#272 Post by zorrosam » Mon Jun 22, 2009 3:09 am

zorrosam wrote:
zorrosam wrote:Hi guys,

any advice for my situation?

06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065

DELL Poweredge 1950
2xQUADCORE 2.00 GHZ XEON
8GB RAM
2xHDD 250GB RAID1
MYSQL ACTIVE ON ANOTHER SERVER TUNED

I don't know if will be better activate boost on my side ... the system is on production and fundamental for all my bussiness. There's the way for do that without downtime?

bye

sam
the version of cacti is 0.87b!

06/22/2009 10:04:31 AM - SYSTEM STATS: Time:269.2874 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 10:04:28 AM - RECACHE STATS: RecacheTime:299.7883 HostsRecached:66
06/22/2009 09:59:31 AM - RECACHE STATS: RecacheTime:304.3112 HostsRecached:71
06/22/2009 09:59:28 AM - SYSTEM STATS: Time:266.6671 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:54:27 AM - SYSTEM STATS: Time:266.1210 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065
06/22/2009 09:49:22 AM - RECACHE STATS: RecacheTime:297.9598 HostsRecached:70
06/22/2009 09:44:24 AM - SYSTEM STATS: Time:262.2032 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:39:25 AM - SYSTEM STATS: Time:263.7265 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:39:20 AM - RECACHE STATS: RecacheTime:297.6952 HostsRecached:71
06/22/2009 09:34:22 AM - SYSTEM STATS: Time:261.4651 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:29:22 AM - SYSTEM STATS: Time:261.6119 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:29:22 AM - RECACHE STATS: RecacheTime:297.0967 HostsRecached:71
06/22/2009 09:24:24 AM - SYSTEM STATS: Time:262.7349 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:19:26 AM - SYSTEM STATS: Time:264.5988 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:19:24 AM - RECACHE STATS: RecacheTime:297.9909 HostsRecached:71
06/22/2009 09:14:26 AM - SYSTEM STATS: Time:264.5102 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:09:20 AM - SYSTEM STATS: Time:259.1723 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:09:15 AM - RECACHE STATS: RecacheTime:292.1965 HostsRecached:74
06/22/2009 09:04:22 AM - SYSTEM STATS: Time:260.7924 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 08:59:23 AM - SYSTEM STATS: Time:261.6565 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 08:59:13 AM - RECACHE STATS: RecacheTime:291.8565 HostsRecached:73
06/22/2009 08:54:22 AM - RECACHE STATS: RecacheTime:300.2356 HostsRecached:71
06/22/2009 08:54:21 AM - SYSTEM STATS: Time:259.7655 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:49:22 AM - SYSTEM STATS: Time:260.2591 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:44:21 AM - SYSTEM STATS: Time:260.2246 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:44:20 AM - RECACHE STATS: RecacheTime:299.8413 HostsRecached:74

The IOS is RedHat 5.4 Arch 64 BIT

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: advice on my poor performance?!

#273 Post by TheWitness » Mon Jun 22, 2009 10:24 am

zorrosam wrote:
zorrosam wrote:
zorrosam wrote:Hi guys,

any advice for my situation?

06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065

DELL Poweredge 1950
2xQUADCORE 2.00 GHZ XEON
8GB RAM
2xHDD 250GB RAID1
MYSQL ACTIVE ON ANOTHER SERVER TUNED

I don't know if will be better activate boost on my side ... the system is on production and fundamental for all my bussiness. There's the way for do that without downtime?

bye

sam
the version of cacti is 0.87b!

06/22/2009 10:04:31 AM - SYSTEM STATS: Time:269.2874 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 10:04:28 AM - RECACHE STATS: RecacheTime:299.7883 HostsRecached:66
06/22/2009 09:59:31 AM - RECACHE STATS: RecacheTime:304.3112 HostsRecached:71
06/22/2009 09:59:28 AM - SYSTEM STATS: Time:266.6671 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:54:27 AM - SYSTEM STATS: Time:266.1210 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:49:30 AM - SYSTEM STATS: Time:269.0361 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7065
06/22/2009 09:49:22 AM - RECACHE STATS: RecacheTime:297.9598 HostsRecached:70
06/22/2009 09:44:24 AM - SYSTEM STATS: Time:262.2032 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:39:25 AM - SYSTEM STATS: Time:263.7265 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:39:20 AM - RECACHE STATS: RecacheTime:297.6952 HostsRecached:71
06/22/2009 09:34:22 AM - SYSTEM STATS: Time:261.4651 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:29:22 AM - SYSTEM STATS: Time:261.6119 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:29:22 AM - RECACHE STATS: RecacheTime:297.0967 HostsRecached:71
06/22/2009 09:24:24 AM - SYSTEM STATS: Time:262.7349 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:19:26 AM - SYSTEM STATS: Time:264.5988 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:19:24 AM - RECACHE STATS: RecacheTime:297.9909 HostsRecached:71
06/22/2009 09:14:26 AM - SYSTEM STATS: Time:264.5102 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:09:20 AM - SYSTEM STATS: Time:259.1723 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 09:09:15 AM - RECACHE STATS: RecacheTime:292.1965 HostsRecached:74
06/22/2009 09:04:22 AM - SYSTEM STATS: Time:260.7924 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 08:59:23 AM - SYSTEM STATS: Time:261.6565 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7055
06/22/2009 08:59:13 AM - RECACHE STATS: RecacheTime:291.8565 HostsRecached:73
06/22/2009 08:54:22 AM - RECACHE STATS: RecacheTime:300.2356 HostsRecached:71
06/22/2009 08:54:21 AM - SYSTEM STATS: Time:259.7655 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:49:22 AM - SYSTEM STATS: Time:260.2591 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:44:21 AM - SYSTEM STATS: Time:260.2246 Method:spine Processes:6 Threads:15 Hosts:179 HostsPerProcess:30 DataSources:14340 RRDsProcessed:7083
06/22/2009 08:44:20 AM - RECACHE STATS: RecacheTime:299.8413 HostsRecached:74

The IOS is RedHat 5.4 Arch 64 BIT
This should be a separate thread. The big thing to understand is this, how long does it take spine to complete. If you issue a watch command for spine processes, and time them. You should be able to determine if the problem is spine (some script or snmp call haning on too long), or RRDtool updates.

Another thing to note is that if the following command (+ 1 GByte) is more than your system ram, you issue is likely the RRDupdate process, of which Boost can help with.

du -s /var/www/html/cacti/rra

Boost can be deployed with no downtime, but you need to know what you are doing. Having a test system to "play" with is essential.

Regards,

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:

#274 Post by jbarbieri » Mon Jul 20, 2009 11:21 am

Fully migrated to new server:

Code: Select all

07/20/2009 03:30:11 PM - SYSTEM STATS: Time:9.6051 Method:spine Processes:15 Threads:25 Hosts:53 HostsPerProcess:4 DataSources:10547 RRDsProcessed:4406
07/20/2009 03:35:10 PM - SYSTEM STATS: Time:9.0917 Method:spine Processes:15 Threads:25 Hosts:53 HostsPerProcess:4 DataSources:10547 RRDsProcessed:4406
Not writing to an NFS helps alot!

Quad Xeon 3.4Ghz with 2GB RAM.
Cacti 0.8.7e and Spine 0.8.7d


Attached is a picture of my poller stats....nice dropoff! Those "blank" spots are when the process took longer then 5 minutes.
Attachments
poller.png
Poller Stats
poller.png (24.59 KiB) Viewed 18947 times

frdrk
Cacti User
Posts: 55
Joined: Fri Jul 06, 2007 2:27 pm

#275 Post by frdrk » Sun Jul 26, 2009 3:08 pm

jbarbieri wrote: Attached is a picture of my poller stats....nice dropoff! Those "blank" spots are when the process took longer then 5 minutes.
that's a nice fixed font, whats it called?

jbarbieri
Cacti User
Posts: 101
Joined: Tue May 24, 2005 8:46 am
Location: Rochester, New Hampshire
Contact:

#276 Post by jbarbieri » Mon Jul 27, 2009 7:30 am

To be honest, I don't know. I am running RRDTool 1.3.7, but my paths for fonts are all blank.

kmhendress
Posts: 33
Joined: Wed Aug 27, 2008 2:49 pm

#277 Post by kmhendress » Wed Jul 29, 2009 3:48 pm

I'm very happy with the stats!
Cacti Version - 0.8.7e
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.9-67.0.1.ELsmp
Web Server - Apache/2.0.52 (Red Hat)
PHP - 5.2.9
PHP Extensions - libxml, xml, tokenizer, session, pcre, SPL, PDO, SQLite, sockets, standard, Reflection, posix, pdo_sqlite, mysql, ldap, json, iconv, hash, gd, filter, date, ctype, zlib, apache2handler
MySQL - 4.1.20
RRDTool - 1.2.23
SNMP - 5.1.2
Plugins
  • Thresholds (thold - v0.4.1)
    Global Plugin Settings (settings - v0.5)
    Host Info (hostinfo - v0.2)
    Update Checker (update - v0.4)
    Device Monitoring (monitor - v0.8.2)
    PHP Network Weathermap (weathermap - v0.96a)

07/22/2009 03:05:12 PM - SYSTEM STATS: Time:298.4995 Method:cmd.php Processes:30 Threads:N/A Hosts:18 HostsPerProcess:1 DataSources:5731 RRDsProcessed:2238

07/28/2009 09:20:57 AM - SYSTEM STATS: Time:54.9774 Method:spine Processes:20 Threads:5 Hosts:18 HostsPerProcess:1 DataSources:6184 RRDsProcessed:3439

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#278 Post by gandalf » Thu Jul 30, 2009 12:59 pm

kmhendress wrote: 07/22/2009 03:05:12 PM - SYSTEM STATS: Time:298.4995 Method:cmd.php Processes:30 Threads:N/A Hosts:18 HostsPerProcess:1 DataSources:5731 RRDsProcessed:2238

07/28/2009 09:20:57 AM - SYSTEM STATS: Time:54.9774 Method:spine Processes:20 Threads:5 Hosts:18 HostsPerProcess:1 DataSources:6184 RRDsProcessed:3439
I'd do using 5 processes and 20 threads unless you have a Quad-Quadcore or the like
Reinhard

kmhendress
Posts: 33
Joined: Wed Aug 27, 2008 2:49 pm

#279 Post by kmhendress » Thu Jul 30, 2009 3:21 pm

Thanks, I have made the reccomended change. It is a VM box, with two CPU's tied to it.

Here are the new stats, even much better!

07/30/2009 03:20:38 PM - SYSTEM STATS: Time:35.9677 Method:spine Processes:5 Threads:20 Hosts:18 HostsPerProcess:4 DataSources:6186 RRDsProcessed:1474

kmhendress
Posts: 33
Joined: Wed Aug 27, 2008 2:49 pm

#280 Post by kmhendress » Thu Jul 30, 2009 3:44 pm

kmhendress wrote:Thanks, I have made the reccomended change. It is a VM box, with two CPU's tied to it.

Here are the new stats, even much better!

07/30/2009 03:20:38 PM - SYSTEM STATS: Time:35.9677 Method:spine Processes:5 Threads:20 Hosts:18 HostsPerProcess:4 DataSources:6186 RRDsProcessed:1474
Went back to about ~60 seconds
07/30/2009 03:40:58 PM - SYSTEM STATS: Time:56.8005 Method:spine Processes:5 Threads:20 Hosts:18 HostsPerProcess:4 DataSources:6186 RRDsProcessed:3478

User avatar
Howie
Cacti Guru User
Posts: 5333
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

#281 Post by Howie » Fri Jul 31, 2009 9:13 am

Not spine vs cmd, but spine vs cactid :-)

I just upgraded a customer's Cacti install from 0.8.7a+cactid to 0.8.7e+spine and the polling time nearly halved!

07/31/2009 03:13:26 PM - SYSTEM STATS: Time:7.4273 Method:spine Processes:1 Threads:10 Hosts:10 HostsPerProcess:10 DataSources:958 RRDsProcessed:390

Nice work!
Attachments
graph_image.php.png
graph_image.php.png (28.15 KiB) Viewed 18774 times
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)

kmhendress
Posts: 33
Joined: Wed Aug 27, 2008 2:49 pm

#282 Post by kmhendress » Fri Jul 31, 2009 9:40 am

Howie wrote:Not spine vs cmd, but spine vs cactid :-)

I just upgraded a customer's Cacti install from 0.8.7a+cactid to 0.8.7e+spine and the polling time nearly halved!

07/31/2009 03:13:26 PM - SYSTEM STATS: Time:7.4273 Method:spine Processes:1 Threads:10 Hosts:10 HostsPerProcess:10 DataSources:958 RRDsProcessed:390

Nice work!
I like your graph, what template did you use to get that working? I'd love to have it :)

User avatar
Howie
Cacti Guru User
Posts: 5333
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

#283 Post by Howie » Fri Jul 31, 2009 9:50 am

kmhendress wrote:
I like your graph, what template did you use to get that working? I'd love to have it :)
It's Gandalf's Poller Statistics template. Very handy for figuring out whether you are about to hit a wall, and what the impact is of any tuning.
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)

kmhendress
Posts: 33
Joined: Wed Aug 27, 2008 2:49 pm

#284 Post by kmhendress » Fri Jul 31, 2009 10:22 am

Awesome - Thanks!

ikorzha758
Cacti User
Posts: 162
Joined: Thu Jan 21, 2010 8:41 pm

#285 Post by ikorzha758 » Thu Jan 28, 2010 1:17 pm

Here are my results so far:
01/28/2010 01:17:29 PM - SYSTEM STATS: Time:28.0138 Method:spine Processes:6 Threads:4 Hosts:60 HostsPerProcess:10 DataSources:16713 RRDsProcessed:8516
01/28/2010 01:16:29 PM - SYSTEM STATS: Time:28.0694 Method:spine Processes:6 Threads:4 Hosts:60 HostsPerProcess:10 DataSources:16713 RRDsProcessed:8516
01/28/2010 01:15:32 PM - SYSTEM STATS: Time:29.8361 Method:spine Processes:6 Threads:4 Hosts:60 HostsPerProcess:10 DataSources:16713 RRDsProcessed:8516

1 Minute polling.
Cacti 0.8.7e and Spine 0.8.7e
2 Xeons 2.0Ghz each 6 Gigs of RAM Linux 5.2 Enterprise.

Post Reply