boost_server.php forgets config

Discussions on developing plugins for the Cacti Plugin Architecture

Moderators: Moderators, Developers

Post Reply
Author
Message
aldobert
Posts: 2
Joined: Tue Sep 04, 2007 4:48 am
Location: Tallinn, Estonia

boost_server.php forgets config

#1 Post by aldobert » Thu Feb 07, 2008 8:59 am

I'm trying to set up and test a kind of Cacti cluster. My live setup contains Cacti 0.8.7a+plugininfra, 27k datasources, dual dualcore opterons, 8G mem, SAS disk array and so on. Main problem is iowait during rrd updates.

In first phase I separated mysql, but impact to main server load was next to nothing.

Next, I made a test environment where Boost_server, boost_poller and RRD files are on a separate server and sharing RRD directory to main (web+poller) server via NFS. This setup seems very good. Test env contains 2k RRD-s, boost poller runs by crontab in every 15 min and run duration is less than 2 seconds. But boost_server on demand update gets broken after about 20 minutes of run. It starts to reject connections from main server:

Code: Select all

2008:02:07 12:55:29 - Host Connected '192.168.25.120'
02/07/2008 01:36:07 PM - BOOST SERVER: Poller[0] WARNING: Host '192.168.25.120' Not Authorized. 
I made some debugging and added variable dumps into this authorization code and found that boost_server.php just misses configuration value for boost_server_clients.

$config before:

Code: Select all

array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '192.168.25.120',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)
$config after:

Code: Select all

array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '127.0.0.1',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)
And another bad thing is that the main server loses also the data from poller_output_boost after unsuccessful rrd update against boost_server.

I can handle this issue by some custom hacks but it is good to know if someone else had done similar setup and problems too ;)

aldobert
Posts: 2
Joined: Tue Sep 04, 2007 4:48 am
Location: Tallinn, Estonia

#2 Post by aldobert » Fri Feb 08, 2008 10:35 am

From other logs I found that boost_server goes really mad after that and also found the root cause - mysql connection broke by idle timeout. I had mysql variable wait_timeout set to 900 seconds.

I suggest to add automatic reconnect to boost server because when it's used only for on demand rrd updates, it may get timeout for any timeout value if users do not use cacti wui that long time.

Some examples what happened after timeout:

boost_server.log (boost_server.php -d -f > $LOG):

Code: Select all

2008:02:08 12:05:51 - Host Connected '192.168.25.120'
2008:02:08 12:05:51 - Host Validated '192.168.25.120'
2008:02:08 12:05:51 - RRD Command 'update /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:6
4956675963:59680022319'
02/08/2008 12:05:51 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: plugins/boost/boost_rrdupdate.php 5
18 /usr/bin/rrdupdate /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:64956675963:596800223
19]
02/08/2008 12:06:01 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '192.168.25.120'
[email protected]_server:

Code: Select all

02/08/2008 12:05:51 PM - CMDPHP: Poller[0] ERROR: SQL Row Failed!, Error:'2006', SQL:"select value from settings where name='poller_verbosity'"
02/08/2008 12:05:51 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'pclose(): 57 is not a valid stream resource' IN FILE:'/hom
e/cacti/cacti_test/lib/rrd.php' LINE NO:'57'
... and thousands of such rows until disk gets full
and [email protected] server:

Code: Select all

02/08/2008 12:06:01 PM - BOOST: Poller[0] ERROR: Timeout detected.  Boost server is down.  Contact support immediately!!
02/08/2008 12:06:01 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '27529'

02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "INSERT INTO `rcaston_1202472351` (local_data_id, rrd_name, time) VALUES ('27529',
'rb_traf_rx', '2008-02-08 12:05:16'), ('27529', 'rb_traf_tx', '2008-02-08 12:05:16') ON DUPLICATE KEY UPDATE local_data_id=VALUES(local_data_id
)"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DELETE FROM `poller_output_boost`  USING `poller_output_boost`, `rcaston_120247235
1`  WHERE rcaston_1202472351.local_data_id=poller_output_boost.local_data_id  AND rcaston_1202472351.rrd_name=poller_output_boost.rrd_name  AND
 rcaston_1202472351.time=poller_output_boost.time"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DROP table `rcaston_1202472351`"
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_write(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1193'
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_close(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1194'

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#3 Post by TheWitness » Fri Feb 08, 2008 7:09 pm

Holly crap, nasty dependencies. This is one of my "projects" to get this thing working again.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
pooreiple
Posts: 1
Joined: Sun Jan 31, 2010 6:38 pm
Location: Holland
Contact:

Fri Feb 08 2008 12 09 pm

#4 Post by pooreiple » Tue Feb 02, 2010 2:43 pm

how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks
---------------------
Things that make you go....

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: Fri Feb 08 2008 12 09 pm

#5 Post by TheWitness » Tue Feb 02, 2010 3:53 pm

pooreiple wrote:how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks
Your post makes no sense.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

simonjones80
Posts: 1
Joined: Tue Sep 21, 2010 4:54 am

#6 Post by simonjones80 » Tue Sep 21, 2010 5:21 am

Hi,

Thanks for the great work on boost! It really has changed my life!

I have also had this issue and found it disturbing to find a gap in my graphs.
I think boost should only remove data from the poller_output_boost table once the rrd file has been successfully updated. I guess a rrd update is currently considered a success even if the update is a nan, which it shouldn't be as the data is correct in the DB, but the client is not authorized. If the client it not authorized it should not perform any processing at all. Disconnect before fetching any data maybe?

Just a quick note on the reconnect check you do in boost server. You are probably way ahead of me but it may be a better idea to use something like http://php.net/manual/en/function.mysql-ping.php.

Thanks again for the great work on boost. It is really appreciated!
All the best!

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#7 Post by TheWitness » Tue Sep 21, 2010 8:04 am

Boost 4.2 already does this.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

kanada
Cacti User
Posts: 137
Joined: Sun Aug 28, 2005 12:51 pm

Re: boost_server.php forgets config

#8 Post by kanada » Wed Nov 10, 2010 4:55 pm

Unfortunately i still don't see this, and got an error when processing 'On Demand'.
PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'
MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex

c3226026
Cacti User
Posts: 87
Joined: Mon Jan 17, 2011 12:15 pm

Re: boost_server.php forgets config

#9 Post by c3226026 » Fri Mar 04, 2011 9:49 am

kanada wrote:Unfortunately i still don't see this, and got an error when processing 'On Demand'.
PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'
MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex
Hi,

I've also this with :

MySQL 5.1.49, PHP 5.3.3-7, Cacti 0.8.7g && Spine && Boost 4.3.

I've "solve" this by change Boost Server Timeout 2s -> 20s

And I've never this alarm and no gap on graph.

But I've some alarms when I want to see graphs.

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: boost_server.php forgets config

#10 Post by TheWitness » Sat Oct 01, 2011 12:47 pm

Use boost 5.0
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

Post Reply