Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sun May 19, 2019 9:35 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 7:01 am 
Offline
Cacti User
User avatar

Joined: Mon Apr 24, 2006 3:33 am
Posts: 216
hi all,

I've recently got stuck on a small problem. I use aggregate plugin which generates huge images from variety of data-sources. Those images are continously being watched by service-desk team, moreover - several applications are fetching them as well.

I've wrote a small experimental plugin which caches images in shared-memory after they were watched. It also adds a small informational legend on the bottom of the image.

without memorize plug-in generating one image on quad-core machine with 4gigs of ram takes from 2.5 to 6 seconds (depending on poller is running or not). With memorize plugin installed, cached images are returned within 1 microsecond and less (from 4000 to 7000 times faster).
Browser renders images on the fly and pooler gets extra cpu resources, while RAM consumption is relatively small.

moreover, png images are compressed up to the max, so average image size drops ~10%:
without memorize: 91.73 KB (93927 bytes)
with memorize: 78.96 KB (80853 bytes)

here are some examples:

when poller isn't running:
Image
when poller is running:
Image

after simple refresh memorize starts doing it's job:

poller isn't running:
Image
poller is running:
Image

memorize statistics when poller isn't running:
Code:
displayed graphs count:   4
rendered from rrd:   2
rendered from cache:   2
avg rrd render time:   2.600489
avg cache render time:   0.000336
benefit(times):    7739.550595
used cache memory size:   361947


memorize statistics when poller is running:
Code:
displayed graphs count:   4
rendered from rrd:   2
rendered from cache:   2
avg rrd render time:   6.138474
avg cache render time:   0.001444
benefit(times):    4251.020776
used cache memory size:   361982


one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

after downloading it, please see "README". Most likely you will have to tune some kernel parameters to cache more images.

Also, make sure you have the following extensions enabled in php.ini (or statically compiled-in):
php-gd2 with FreeType lib - to see statistical information on image
shmop - to cache images in memory
sysvsem
sysvshm

this plugin was tested with cacti 0.8.7b + PIA 2.1
on freebsd-7 and debian-4 systems (+reported working on gentoo)
running on php5.2 + mysql 5.1+apache 2.2
whithout any aditional kernel tunig


Attachments:
File comment: Memorize plugin v0.3
memorize_v03.tar.gz [46.76 KiB]
Downloaded 748 times
File comment: Memorize plugin v0.2
memorize_v02.tar.gz [46.68 KiB]
Downloaded 411 times

_________________
zond plugin
memorize plugin
transporter plugin


Last edited by johnrembo on Fri Aug 08, 2008 5:47 am, edited 6 times in total.
Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 7:39 am 
Offline
Cacti Pro User
User avatar

Joined: Mon Mar 27, 2006 10:35 am
Posts: 711
Location: Psychic Amish Network Administrator
I'm getting an error:

Quote:
Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117


I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.

Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"

I think that's normally off by default when PHP is installed.

_________________
Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-128.1.6.el5
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.2.9
MySQL - 5.0.45-log
RRDTool - 1.3.0
SNMP - 5.3.2.2
PluginsPHP Network Managing v0.6.1, Global Plugin Settings v0.6,thold v0.4.1,XMLPort v0.3.5,CactiCam v0.1.5,NetTools v0.1.5,pollperf v0.32,RRD Cleaner v1.1,sqlqueries v0.2,superlinks v0.8,syslog v0.5.2,update v0.4,discovery v0.9,zond v0.34a,hostinfo v0.2,Bloom v0.6.5,mactrack v1.1,weathermap v0.96a,mobile v0.1


Top
 Profile  
 
 Post subject: Re: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 7:54 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5311
Location: United Kingdom
johnrembo wrote:
one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.


It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.

_________________
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject: Re: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 8:03 am 
Offline
Cacti Pro User
User avatar

Joined: Mon Mar 27, 2006 10:35 am
Posts: 711
Location: Psychic Amish Network Administrator
Howie wrote:
johnrembo wrote:
one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.


It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.


Does Weathermap currently do anything like this? (hint hint).

_________________
Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-128.1.6.el5
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.2.9
MySQL - 5.0.45-log
RRDTool - 1.3.0
SNMP - 5.3.2.2
PluginsPHP Network Managing v0.6.1, Global Plugin Settings v0.6,thold v0.4.1,XMLPort v0.3.5,CactiCam v0.1.5,NetTools v0.1.5,pollperf v0.32,RRD Cleaner v1.1,sqlqueries v0.2,superlinks v0.8,syslog v0.5.2,update v0.4,discovery v0.9,zond v0.34a,hostinfo v0.2,Bloom v0.6.5,mactrack v1.1,weathermap v0.96a,mobile v0.1


Top
 Profile  
 
 Post subject: Re: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 8:07 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5311
Location: United Kingdom
streaker69 wrote:
Howie wrote:
johnrembo wrote:
one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.


It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.


Does Weathermap currently do anything like this? (hint hint).


Yes :P

It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.

What are you hinting at? It's too subtle for me... :-)

_________________
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject: Re: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 8:10 am 
Offline
Cacti Pro User
User avatar

Joined: Mon Mar 27, 2006 10:35 am
Posts: 711
Location: Psychic Amish Network Administrator
Howie wrote:

Yes :P

It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.

What are you hinting at? It's too subtle for me... :-)


Ahh, I didn't realize it was caching, must just be my slow Cacti box that makes it seem like it's not.

_________________
Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-128.1.6.el5
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.2.9
MySQL - 5.0.45-log
RRDTool - 1.3.0
SNMP - 5.3.2.2
PluginsPHP Network Managing v0.6.1, Global Plugin Settings v0.6,thold v0.4.1,XMLPort v0.3.5,CactiCam v0.1.5,NetTools v0.1.5,pollperf v0.32,RRD Cleaner v1.1,sqlqueries v0.2,superlinks v0.8,syslog v0.5.2,update v0.4,discovery v0.9,zond v0.34a,hostinfo v0.2,Bloom v0.6.5,mactrack v1.1,weathermap v0.96a,mobile v0.1


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 8:32 am 
Offline
Cacti User
User avatar

Joined: Mon Apr 24, 2006 3:33 am
Posts: 216
streaker69 wrote:
I'm getting an error:

Quote:
Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117


I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.

Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"

I think that's normally off by default when PHP is installed.


damn! i've forgotten to revert php settings to their defaults while testing :\
it whould be enough to remove ampersand on line 117 ;)

anyway, thanx for the tip - i've reuploaded the file (no need for you to redownload).


Top
 Profile  
 
 Post subject: Re: Memorize: image caching plugin
PostPosted: Tue Jul 29, 2008 8:38 am 
Offline
Cacti User
User avatar

Joined: Mon Apr 24, 2006 3:33 am
Posts: 216
Howie wrote:
johnrembo wrote:
one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.


It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.


hmm...now i'm thinking maybe I was wrong?
Quote:
Note: Versions of Windows previous to Windows 2000 do not support shared memory. Under Windows, Shmop will only work when PHP is running as a web server module, such as Apache or IIS (CLI and CGI will not work).


is there anyone running windows installation? it would be nice to see what this plugin vomits about it. maybe there still is a hope...


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 8:44 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5311
Location: United Kingdom
My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.

Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ;-) ) - Vera and DeJa Vu are free to distribute with open source software.

I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now.

_________________
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 9:02 am 
Offline
Cacti User
User avatar

Joined: Mon Apr 24, 2006 3:33 am
Posts: 216
Howie wrote:
My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.

Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ;-) ) - Vera and DeJa Vu are free to distribute with open source software.

I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now.



thanx for the tips Howie. I'll note that.

this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now ;)

ps: this plugin was originaly developed on debian-linux machine and now seems to work fine on 5 different servers. I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 9:18 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5311
Location: United Kingdom
johnrembo wrote:
this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now ;)


Hehe. See what you have done? :-)

Quote:
I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap.


I'm happy to test on FreeBSD for you - just not on my mail server :-) I have a spare FreeBSD system to play with.

_________________
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 11:33 am 
Offline
Cacti User

Joined: Fri Jul 06, 2007 2:27 pm
Posts: 55
Hi, this seems like great plugin :). I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.

edit:

Hm, no.. still no images.

Code:
gd
GD Support    enabled
GD Version    2.0 or higher
FreeType Support    enabled
FreeType Linkage    with freetype
FreeType Version    2.3.7
T1Lib Support    enabled
GIF Read Support    enabled
GIF Create Support    enabled
JPG Support    enabled
PNG Support    enabled
WBMP Support    enabled


That should work, right?

When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."

Cache statistics shows nothing;
Code:
displayed graphs count:   0
rendered from rrd:   0
rendered from cache:   0
avg rrd render time:   0
avg cache render time:   0
benefit(times):    0
used cache memory size:   0


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 3:25 pm 
Offline
Cacti User
User avatar

Joined: Mon Apr 24, 2006 3:33 am
Posts: 216
frdrk wrote:
Hi, this seems like great plugin :). I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.

edit:

Hm, no.. still no images.

Code:
gd
GD Support    enabled
GD Version    2.0 or higher
FreeType Support    enabled
FreeType Linkage    with freetype
FreeType Version    2.3.7
T1Lib Support    enabled
GIF Read Support    enabled
GIF Create Support    enabled
JPG Support    enabled
PNG Support    enabled
WBMP Support    enabled


That should work, right?

When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."

Cache statistics shows nothing;
Code:
displayed graphs count:   0
rendered from rrd:   0
rendered from cache:   0
avg rrd render time:   0
avg cache render time:   0
benefit(times):    0
used cache memory size:   0

i've upgraded my post. forgot to mension the most important extension - shmop :oops:


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 29, 2008 3:31 pm 
Offline
Cacti User

Joined: Fri Jul 06, 2007 2:27 pm
Posts: 55
johnrembo wrote:
i've upgraded my post. forgot to mension the most important extension - shmop :oops:



Haha :D. Remerging php now with the +sharedmem use-flag :)

edit

ok, there we go. Niice :D
Code:
displayed graphs count:   18
rendered from rrd:   9
rendered from cache:   9
avg rrd render time:   1.300332
avg cache render time:   0.000818
benefit(times):    1589.647922
used cache memory size:   518870


To get this running on gentoo one has to emerge php with the sharedmem and sysvipc useflags.


Top
 Profile  
 
 Post subject: PLEASE HELP !
PostPosted: Fri Aug 01, 2008 2:16 am 
Offline

Joined: Wed Jul 02, 2008 7:35 am
Posts: 9
Hi,

thanx for your nice plugin.

I am using cacti 0.87b und PA 2.0 on cactiEZ 0.4

1. in Settings/Memorize if pressing SHOW or FLUSH buttom gives in the Browser-Statusline "ERROR on page." and nothing is happend. Seemed to be a Java compatibility problem, i am using IE7 & IE6 with JRE 1.607 and JRE 1.5

2. It speeds up the graphs, but sometimes, calling graphs for a device the first time, it shows no graphs (placeholder is shown), then i press the refresh buttom in the browser .... now the graphs are shown......

You have a idea to fix it ?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group