|
|
| Author |
Message |
johnrembo
Cacti User
Joined: 24 Apr 2006
Posts: 198
|
 Posted: Tue Jul 29, 2008 7:01 am Post subject: Memorize: image caching plugin |
 |
|
hi all,
I've recently got stuck on a small problem. I use aggregate plugin which generates huge images from variety of data-sources. Those images are continously being watched by service-desk team, moreover - several applications are fetching them as well.
I've wrote a small experimental plugin which caches images in shared-memory after they were watched. It also adds a small informational legend on the bottom of the image.
without memorize plug-in generating one image on quad-core machine with 4gigs of ram takes from 2.5 to 6 seconds (depending on poller is running or not). With memorize plugin installed, cached images are returned within 1 microsecond and less (from 4000 to 7000 times faster).
Browser renders images on the fly and pooler gets extra cpu resources, while RAM consumption is relatively small.
moreover, png images are compressed up to the max, so average image size drops ~10%:
without memorize: 91.73 KB (93927 bytes)
with memorize: 78.96 KB (80853 bytes)
here are some examples:
when poller isn't running:
when poller is running:
after simple refresh memorize starts doing it's job:
poller isn't running:
poller is running:
memorize statistics when poller isn't running:
| Code: |
displayed graphs count: 4
rendered from rrd: 2
rendered from cache: 2
avg rrd render time: 2.600489
avg cache render time: 0.000336
benefit(times): 7739.550595
used cache memory size: 361947
|
memorize statistics when poller is running:
| Code: |
displayed graphs count: 4
rendered from rrd: 2
rendered from cache: 2
avg rrd render time: 6.138474
avg cache render time: 0.001444
benefit(times): 4251.020776
used cache memory size: 361982
|
one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.
after downloading it, please see "README". Most likely you will have to tune some kernel parameters to cache more images.
Also, make sure you have the following extensions enabled in php.ini (or statically compiled-in):
php-gd2 with FreeType lib - to see statistical information on image
shmop - to cache images in memory
sysvsem
sysvshm
this plugin was tested with cacti 0.8.7b + PIA 2.1
on freebsd-7 and debian-4 systems (+reported working on gentoo)
running on php5.2 + mysql 5.1+apache 2.2
whithout any aditional kernel tunig
| Description: |
|
Download |
| Filename: |
memorize_v03.tar.gz |
| Filesize: |
46.76 KB |
| Downloaded: |
48 Time(s) |
| Description: |
|
Download |
| Filename: |
memorize_v02.tar.gz |
| Filesize: |
46.68 KB |
| Downloaded: |
44 Time(s) |
Last edited by johnrembo on Fri Aug 08, 2008 5:47 am; edited 6 times in total |
|
| Back to top |
|
 |
streaker69
Cacti Pro User
Joined: 27 Mar 2006
Posts: 647
Location: Psychic Amish Network Administrator
|
| Posted: Tue Jul 29, 2008 7:39 am Post subject: |
|
|
I'm getting an error:
| Quote: |
Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117
|
I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.
Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"
I think that's normally off by default when PHP is installed.
|
|
| Back to top |
|
|
Howie
Cacti Guru User
Joined: 16 Sep 2004
Posts: 2135
Location: United Kingdom
|
| Posted: Tue Jul 29, 2008 7:54 am Post subject: Re: Memorize: image caching plugin |
|
|
| johnrembo wrote: | | one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os. |
It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.
|
|
| Back to top |
|
|
streaker69
Cacti Pro User
Joined: 27 Mar 2006
Posts: 647
Location: Psychic Amish Network Administrator
|
| Posted: Tue Jul 29, 2008 8:03 am Post subject: Re: Memorize: image caching plugin |
|
|
| Howie wrote: | | johnrembo wrote: | | one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os. |
It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does. |
Does Weathermap currently do anything like this? (hint hint).
|
|
| Back to top |
|
|
Howie
Cacti Guru User
Joined: 16 Sep 2004
Posts: 2135
Location: United Kingdom
|
| Posted: Tue Jul 29, 2008 8:07 am Post subject: Re: Memorize: image caching plugin |
|
|
| streaker69 wrote: | | Howie wrote: | | johnrembo wrote: | | one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os. |
It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does. |
Does Weathermap currently do anything like this? (hint hint). |
Yes 
It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.
What are you hinting at? It's too subtle for me... 
|
|
| Back to top |
|
|
streaker69
Cacti Pro User
Joined: 27 Mar 2006
Posts: 647
Location: Psychic Amish Network Administrator
|
| Posted: Tue Jul 29, 2008 8:10 am Post subject: Re: Memorize: image caching plugin |
|
|
| Howie wrote: |
Yes
It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.
What are you hinting at? It's too subtle for me... |
Ahh, I didn't realize it was caching, must just be my slow Cacti box that makes it seem like it's not.
|
|
| Back to top |
|
|
johnrembo
Cacti User
Joined: 24 Apr 2006
Posts: 198
|
| Posted: Tue Jul 29, 2008 8:32 am Post subject: |
|
|
| streaker69 wrote: | I'm getting an error:
| Quote: |
Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117
|
I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.
Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"
I think that's normally off by default when PHP is installed. |
damn! i've forgotten to revert php settings to their defaults while testing :\
it whould be enough to remove ampersand on line 117 
anyway, thanx for the tip - i've reuploaded the file (no need for you to redownload).
|
|
| Back to top |
|
|
johnrembo
Cacti User
Joined: 24 Apr 2006
Posts: 198
|
| Posted: Tue Jul 29, 2008 8:38 am Post subject: Re: Memorize: image caching plugin |
|
|
| Howie wrote: | | johnrembo wrote: | | one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os. |
It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does. |
hmm...now i'm thinking maybe I was wrong?
| Quote: | | Note: Versions of Windows previous to Windows 2000 do not support shared memory. Under Windows, Shmop will only work when PHP is running as a web server module, such as Apache or IIS (CLI and CGI will not work). |
is there anyone running windows installation? it would be nice to see what this plugin vomits about it. maybe there still is a hope...
|
|
| Back to top |
|
|
Howie
Cacti Guru User
Joined: 16 Sep 2004
Posts: 2135
Location: United Kingdom
|
| Posted: Tue Jul 29, 2008 8:44 am Post subject: |
|
|
My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.
Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ) - Vera and DeJa Vu are free to distribute with open source software.
I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now.
|
|
| Back to top |
|
|
johnrembo
Cacti User
Joined: 24 Apr 2006
Posts: 198
|
| Posted: Tue Jul 29, 2008 9:02 am Post subject: |
|
|
| Howie wrote: | My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.
Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ) - Vera and DeJa Vu are free to distribute with open source software.
I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now. |
thanx for the tips Howie. I'll note that.
this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now
ps: this plugin was originaly developed on debian-linux machine and now seems to work fine on 5 different servers. I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap.
|
|
| Back to top |
|
|
Howie
Cacti Guru User
Joined: 16 Sep 2004
Posts: 2135
Location: United Kingdom
|
| Posted: Tue Jul 29, 2008 9:18 am Post subject: |
|
|
| johnrembo wrote: | this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now
|
Hehe. See what you have done?
| Quote: | | I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap. |
I'm happy to test on FreeBSD for you - just not on my mail server I have a spare FreeBSD system to play with.
|
|
| Back to top |
|
|
frdrk
Joined: 06 Jul 2007
Posts: 41
|
| Posted: Tue Jul 29, 2008 11:33 am Post subject: |
|
|
Hi, this seems like great plugin . I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.
edit:
Hm, no.. still no images.
| Code: | gd
GD Support enabled
GD Version 2.0 or higher
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.3.7
T1Lib Support enabled
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
|
That should work, right?
When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."
Cache statistics shows nothing;
| Code: | displayed graphs count: 0
rendered from rrd: 0
rendered from cache: 0
avg rrd render time: 0
avg cache render time: 0
benefit(times): 0
used cache memory size: 0 |
|
|
| Back to top |
|
|
johnrembo
Cacti User
Joined: 24 Apr 2006
Posts: 198
|
| Posted: Tue Jul 29, 2008 3:25 pm Post subject: |
|
|
| frdrk wrote: | Hi, this seems like great plugin . I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.
edit:
Hm, no.. still no images.
| Code: | gd
GD Support enabled
GD Version 2.0 or higher
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.3.7
T1Lib Support enabled
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
|
That should work, right?
When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."
Cache statistics shows nothing;
| Code: | displayed graphs count: 0
rendered from rrd: 0
rendered from cache: 0
avg rrd render time: 0
avg cache render time: 0
benefit(times): 0
used cache memory size: 0 |
|
i've upgraded my post. forgot to mension the most important extension - shmop 
|
|
| Back to top |
|
|
frdrk
Joined: 06 Jul 2007
Posts: 41
|
| Posted: Tue Jul 29, 2008 3:31 pm Post subject: |
|
|
| johnrembo wrote: |
i've upgraded my post. forgot to mension the most important extension - shmop |
Haha  . Remerging php now with the +sharedmem use-flag
edit
ok, there we go. Niice
| Code: | displayed graphs count: 18
rendered from rrd: 9
rendered from cache: 9
avg rrd render time: 1.300332
avg cache render time: 0.000818
benefit(times): 1589.647922
used cache memory size: 518870 |
To get this running on gentoo one has to emerge php with the sharedmem and sysvipc useflags.
|
|
| Back to top |
|
|
bodo.bach
Joined: 02 Jul 2008
Posts: 9
|
| Posted: Fri Aug 01, 2008 2:16 am Post subject: PLEASE HELP ! |
|
|
Hi,
thanx for your nice plugin.
I am using cacti 0.87b und PA 2.0 on cactiEZ 0.4
1. in Settings/Memorize if pressing SHOW or FLUSH buttom gives in the Browser-Statusline "ERROR on page." and nothing is happend. Seemed to be a Java compatibility problem, i am using IE7 & IE6 with JRE 1.607 and JRE 1.5
2. It speeds up the graphs, but sometimes, calling graphs for a device the first time, it shows no graphs (placeholder is shown), then i press the refresh buttom in the browser .... now the graphs are shown......
You have a idea to fix it ?
|
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Register
Profile
Log in to check your private messages
Log in
