Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Thu Jul 20, 2017 5:57 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 10 posts ] 
Author Message
 Post subject: boost_server.php forgets config
PostPosted: Thu Feb 07, 2008 8:59 am 
Offline

Joined: Tue Sep 04, 2007 4:48 am
Posts: 2
Location: Tallinn, Estonia
I'm trying to set up and test a kind of Cacti cluster. My live setup contains Cacti 0.8.7a+plugininfra, 27k datasources, dual dualcore opterons, 8G mem, SAS disk array and so on. Main problem is iowait during rrd updates.

In first phase I separated mysql, but impact to main server load was next to nothing.

Next, I made a test environment where Boost_server, boost_poller and RRD files are on a separate server and sharing RRD directory to main (web+poller) server via NFS. This setup seems very good. Test env contains 2k RRD-s, boost poller runs by crontab in every 15 min and run duration is less than 2 seconds. But boost_server on demand update gets broken after about 20 minutes of run. It starts to reject connections from main server:

Code:
2008:02:07 12:55:29 - Host Connected '192.168.25.120'
02/07/2008 01:36:07 PM - BOOST SERVER: Poller[0] WARNING: Host '192.168.25.120' Not Authorized.


I made some debugging and added variable dumps into this authorization code and found that boost_server.php just misses configuration value for boost_server_clients.

$config before:
Code:
array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '192.168.25.120',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)


$config after:
Code:
array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '127.0.0.1',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)


And another bad thing is that the main server loses also the data from poller_output_boost after unsuccessful rrd update against boost_server.

I can handle this issue by some custom hacks but it is good to know if someone else had done similar setup and problems too ;)


Top
 Profile  
 
 Post subject:
PostPosted: Fri Feb 08, 2008 10:35 am 
Offline

Joined: Tue Sep 04, 2007 4:48 am
Posts: 2
Location: Tallinn, Estonia
From other logs I found that boost_server goes really mad after that and also found the root cause - mysql connection broke by idle timeout. I had mysql variable wait_timeout set to 900 seconds.

I suggest to add automatic reconnect to boost server because when it's used only for on demand rrd updates, it may get timeout for any timeout value if users do not use cacti wui that long time.

Some examples what happened after timeout:

boost_server.log (boost_server.php -d -f > $LOG):
Code:
2008:02:08 12:05:51 - Host Connected '192.168.25.120'
2008:02:08 12:05:51 - Host Validated '192.168.25.120'
2008:02:08 12:05:51 - RRD Command 'update /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:6
4956675963:59680022319'
02/08/2008 12:05:51 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: plugins/boost/boost_rrdupdate.php 5
18 /usr/bin/rrdupdate /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:64956675963:596800223
19]
02/08/2008 12:06:01 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '192.168.25.120'


cacti.log@boost_server:
Code:
02/08/2008 12:05:51 PM - CMDPHP: Poller[0] ERROR: SQL Row Failed!, Error:'2006', SQL:"select value from settings where name='poller_verbosity'"
02/08/2008 12:05:51 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'pclose(): 57 is not a valid stream resource' IN FILE:'/hom
e/cacti/cacti_test/lib/rrd.php' LINE NO:'57'
... and thousands of such rows until disk gets full


and cacti.log@frontend server:
Code:
02/08/2008 12:06:01 PM - BOOST: Poller[0] ERROR: Timeout detected.  Boost server is down.  Contact support immediately!!
02/08/2008 12:06:01 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '27529'

02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "INSERT INTO `rcaston_1202472351` (local_data_id, rrd_name, time) VALUES ('27529',
'rb_traf_rx', '2008-02-08 12:05:16'), ('27529', 'rb_traf_tx', '2008-02-08 12:05:16') ON DUPLICATE KEY UPDATE local_data_id=VALUES(local_data_id
)"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DELETE FROM `poller_output_boost`  USING `poller_output_boost`, `rcaston_120247235
1`  WHERE rcaston_1202472351.local_data_id=poller_output_boost.local_data_id  AND rcaston_1202472351.rrd_name=poller_output_boost.rrd_name  AND
rcaston_1202472351.time=poller_output_boost.time"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DROP table `rcaston_1202472351`"
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_write(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1193'
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_close(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1194'


Top
 Profile  
 
 Post subject:
PostPosted: Fri Feb 08, 2008 7:09 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Holly crap, nasty dependencies. This is one of my "projects" to get this thing working again.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: Fri Feb 08 2008 12 09 pm
PostPosted: Tue Feb 02, 2010 2:43 pm 
Offline
User avatar

Joined: Sun Jan 31, 2010 6:38 pm
Posts: 1
Location: Holland
how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks

_________________
---------------------
Things that make you go....


Top
 Profile  
 
 Post subject: Re: Fri Feb 08 2008 12 09 pm
PostPosted: Tue Feb 02, 2010 3:53 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
pooreiple wrote:
how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks


Your post makes no sense.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 21, 2010 5:21 am 
Offline

Joined: Tue Sep 21, 2010 4:54 am
Posts: 1
Hi,

Thanks for the great work on boost! It really has changed my life!

I have also had this issue and found it disturbing to find a gap in my graphs.
I think boost should only remove data from the poller_output_boost table once the rrd file has been successfully updated. I guess a rrd update is currently considered a success even if the update is a nan, which it shouldn't be as the data is correct in the DB, but the client is not authorized. If the client it not authorized it should not perform any processing at all. Disconnect before fetching any data maybe?

Just a quick note on the reconnect check you do in boost server. You are probably way ahead of me but it may be a better idea to use something like http://php.net/manual/en/function.mysql-ping.php .

Thanks again for the great work on boost. It is really appreciated!
All the best!


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 21, 2010 8:04 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Boost 4.2 already does this.

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: Re: boost_server.php forgets config
PostPosted: Wed Nov 10, 2010 4:55 pm 
Offline
Cacti User

Joined: Sun Aug 28, 2005 12:51 pm
Posts: 137
Unfortunately i still don't see this, and got an error when processing 'On Demand'.
Quote:
PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'


MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex


Top
 Profile  
 
 Post subject: Re: boost_server.php forgets config
PostPosted: Fri Mar 04, 2011 9:49 am 
Offline
Cacti User

Joined: Mon Jan 17, 2011 12:15 pm
Posts: 155
kanada wrote:
Unfortunately i still don't see this, and got an error when processing 'On Demand'.
Quote:
PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'


MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex


Hi,

I've also this with :

MySQL 5.1.49, PHP 5.3.3-7, Cacti 0.8.7g && Spine && Boost 4.3.

I've "solve" this by change Boost Server Timeout 2s -> 20s

And I've never this alarm and no gap on graph.

But I've some alarms when I want to see graphs.


Top
 Profile  
 
 Post subject: Re: boost_server.php forgets config
PostPosted: Sat Oct 01, 2011 12:47 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Use boost 5.0

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group