I have noticed that with my setup where I store RRD file on NFS mount point polling performance was very bad.
After some investigation I have found out that SNMP/Script polling was performing very well (within 10s) but rrdtool would take very long time (thousands of seconds) to finish writing data.
The problem here was that rrdtool will write one RRD file at a time and over NFS opening files is slow. With about 7k RRD files this would not scale.
I have solved the problem by parallelizing RRD file write with rrdtool wrapper script:
Code: Select all
#!/usr/bin/python import zlib import subprocess import sys import time import os rrd = "/usr/bin/rrdtool" process_no = 16 processes =  def spawn(): processes.append(subprocess.Popen([rrd, "-"], stdin=subprocess.PIPE)) def pipe(process, data): processes[process].stdin.write(data) if len(sys.argv) > 1 and sys.argv == "-": pno = process_no while pno: spawn() pno -= 1 while True: line = sys.stdin.readline() if not line: break args = line.lstrip().split(" ") if len(args) < 2 or args != "update": pipe(0, line) else: cmd = args file = args hash = zlib.crc32(file) process = hash % process_no #print "cmd: ", cmd, "file: ", file, "hash: ", hash, "process no: ", process pipe(process, line) else: os.execlp(rrd, *([rrd] + sys.argv[1:]))
With this script used instead of rrdtool (path in settings page) I got my poller run times down to about 90s.
I beleve that it should be easy to rewrite this in PHP and include in Cacti release.