Cacti (home)ForumsRepositoryDocumentation
Cacti: offical forums and support
It is currently Sat Oct 25, 2014 5:12 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 11 posts ] 
Author Message
 Post subject: [SOLVED] Poller crash - MySQL error
PostPosted: Tue Apr 08, 2008 6:37 am 
Offline
Cacti User

Joined: Thu Nov 02, 2006 6:02 am
Posts: 92
Location: Bulgaria
For some reason CACTID gives me the message:

CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'
and I didn't get any updates in my graphs.

After mysql --autorepair cacti, it said that poller_output table has been repaired.After that I rebuilt the poller cache and error messages disapeared, but for a while.
After two poll intervals Cacti gives me again the same error:

CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'

Any suggestions how can I fix this ?

_________________
Cacti user


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 08, 2008 6:38 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Your database is damaged. Goto the cli directory and run:

Code:
php -q database_repair.php


TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 08, 2008 2:32 pm 
Offline
Cacti User

Joined: Thu Nov 02, 2006 6:02 am
Posts: 92
Location: Bulgaria
Thank you for your quick answer. Unfotunately I am using Cacti 0.8.6j and I do not have CLI folder at all. I have managed to fix the problem with deleting and then recreating poller_output table by sql commands from cacti.sql file.

_________________
Cacti user


Top
 Profile  
 
 Post subject:
PostPosted: Wed May 07, 2008 4:00 pm 
Offline
Cacti User
User avatar

Joined: Tue May 03, 2005 4:29 pm
Posts: 142
Location: MD, US
Sorry to hijack this thread but I started experiencing the "CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'" error this morning after adding some new graph templates. I ran the repair_database.php file and this did seem to at least make my database usable again. Unfortunately, stuff is not quite normal yet. CPU usage on my cacti server is much higher than normal and using the Cacti site is much slower now. My logs are showing this every poller cycle:

Quote:
05/07/2008 04:46:54 PM - SYSTEM STATS: Time:112.9826 Method:spine Processes:1 Threads:5 Hosts:72 HostsPerProcess:72 DataSources:3839 RRDsProcessed:1682
05/07/2008 04:45:03 PM - CACTID: Poller[0] Host[1] DS[2112] WARNING: Result from SCRIPT not valid. Partial Result: ...
05/07/2008 04:45:03 PM - CACTID: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
05/07/2008 04:45:03 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into settings values ('syslog_last_incoming','')'


I am not exactly sure what this means but it seems to be my biggest clue at the moment. Can anyone suggest what I should try next? I appreciate the help.
Thanks!
-Chris


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 08, 2008 7:40 am 
Offline
Cacti User
User avatar

Joined: Tue May 03, 2005 4:29 pm
Posts: 142
Location: MD, US
OK, I have another clue. I came into work this morning and opened Cacti like I usually do and was told that the database could not be found. When I attempt to open the cacti database through webmin, I am told, "DBI connect failed : Too many connections". I rebooted the cacti box and then was able to log on. Upon logging on, I found that nothing has been working since around 7:00 PM last night. All graphs are empty after that point.

I'm not sure exactly what the repair_database.php did, but it at least allowed me to get back in to the database. Does anyone know what I should look at/what I should do to get back up and running?

Thanks much.
-Chris


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 08, 2008 8:45 am 
Offline
Cacti User
User avatar

Joined: Tue May 03, 2005 4:29 pm
Posts: 142
Location: MD, US
Whoa. I just noticed that the hard drive usage on my Cacti box has been steadily increasing since yesterday when all this started. Looks like I now have only about 2 GB free as opposed to the 40 GB free I had 2 days ago. Being a Linux novice, I'm not too sure how to tell what the heck is taking up all the space. Could it be my database that slowly growing?

The future looks bleak. I am sad.


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 08, 2008 10:39 am 
Offline
Cacti User
User avatar

Joined: Tue May 03, 2005 4:29 pm
Posts: 142
Location: MD, US
It was my /var/log/mysqld.log file that was taking up all the space. It had grown to 48 GB and I had only 300 MB free on the drive before I figured it out. I deleted the file and rebooted the server. I can now log on to the cacti site again and I re-enabled the poller cron job since I had shut it down earlier to prevent further growage.

Now that I can log back on, I went in and had a loog at the mysql log. All that's in it is this, logged about 3 times a second:

Quote:
080508 11:33:01 [ERROR] /usr/libexec/mysqld: Can't open file: 'syslog_incoming.MYI' (errno: 145)


Can anyone tell me what this means?

Thanks.
-Chris


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 08, 2008 11:42 am 
Offline
Cacti User
User avatar

Joined: Tue May 03, 2005 4:29 pm
Posts: 142
Location: MD, US
I found a post on cactiusers.org describing this same issue, that recommended running this:

Quote:
mysqlcheck --auto-repair --databases syslog
mysqlcheck --auto-repair --databases cacti


I did that and received this:

Quote:
[root@hbicacti ~]# mysqlcheck --auto-repair --databases syslog
syslog.syslog
warning : 2 clients are using or haven't closed the table properly
status : OK
syslog.syslog_alert OK
syslog.syslog_incoming
warning : Table is marked as crashed
warning : 2 clients are using or haven't closed the table properly
error : Record at pos: 1062960 is not remove-marked
error : record delete-link-chain corrupted
error : Corrupt
syslog.syslog_remove OK

Repairing tables
syslog.syslog_incoming
warning : Number of rows changed from 5 to 6
status : OK
[root@hbicacti ~]# mysqlcheck --auto-repair --databases cacti
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_data_rra OK
cacti.data_template_rrd OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.mac_track_device_types OK
cacti.mac_track_devices OK
cacti.mac_track_ips OK
cacti.mac_track_ports OK
cacti.mac_track_processes OK
cacti.mac_track_scan_dates OK
cacti.mac_track_scanning_functions OK
cacti.mac_track_sites OK
cacti.mac_track_temp_ports OK
cacti.manage_alerts OK
cacti.manage_device_type OK
cacti.manage_groups OK
cacti.manage_host OK
cacti.manage_method OK
cacti.manage_poller_output OK
cacti.manage_process OK
cacti.manage_services OK
cacti.manage_sites OK
cacti.manage_tcp OK
cacti.manage_templates OK
cacti.plugin_discover_hosts OK
cacti.plugin_discover_os OK
cacti.plugin_flowview_dnscache
note : The storage engine for the table doesn't support check
cacti.plugin_report_definitions OK
cacti.plugin_report_items OK
cacti.plugin_report_last_values OK
cacti.plugin_reports OK
cacti.plugin_thold_contacts OK
cacti.plugin_thold_log OK
cacti.plugin_thold_template_contact OK
cacti.plugin_thold_threshold_contact OK
cacti.plugin_update_info OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item OK
cacti.poller_output OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.thold OK
cacti.thold_data OK
cacti.thold_template OK
cacti.tholdset OK
cacti.uptime OK
cacti.uptime_data OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_data OK
cacti.weathermap_maps OK


The sql log continues to grow, filling up with the same info I mentioned above and my cacti log is still full of this:
Quote:
05/08/2008 12:36:47 PM - SYSTEM STATS: Time:104.8778 Method:spine Processes:1 Threads:5 Hosts:72 HostsPerProcess:72 DataSources:3839 RRDsProcessed:1686
05/08/2008 12:35:02 PM - CACTID: Poller[0] Host[1] DS[2112] WARNING: Result from SCRIPT not valid. Partial Result: ...
05/08/2008 12:35:02 PM - CACTID: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
05/08/2008 12:35:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into settings values ('syslog_last_incoming','1877252')'


Graphs are now being updated and CPU usage on the cacti box is lower but stuff still ain't right.

Thanks
-Chris


Top
 Profile  
 
 Post subject: Same as yours I did. But my server hangs.
PostPosted: Fri Jun 27, 2008 1:20 am 
Offline

Joined: Mon Feb 05, 2007 8:03 am
Posts: 38
Hi Chris,.....

I ran the following two commands you mentioned boss. But the first command for 'syslog' says " select database", I am affraid syslog is not a database in my case.

However second command run succesfully and got same result as you got.
But I m sad to see my server CPU &Memory utilization going high and I am unable to login the server at all............

Well then I rebooted it and was fine for some time, but again started to hang.

Could you please let me know anything else you did to over come this.

Thanks & regards,
S.toufeeq


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jun 27, 2008 2:16 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22461
Location: Muenster, Germany
I answered to your other thread already. Please do not crosspost
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: Re:
PostPosted: Wed Jan 19, 2011 6:43 am 
Offline
Cacti User

Joined: Wed May 27, 2009 5:10 pm
Posts: 120
Diggit2001 wrote:
I found a post on cactiusers.org describing this same issue, that recommended running this:

Quote:
mysqlcheck --auto-repair --databases syslog
mysqlcheck --auto-repair --databases cacti

-Chris


Thanks Chris,
After searching for hours I finally found this post of yours which fixed my problem. The issue for me was similar cacti logs and no syslogs in the plugin at all - even though these were working fine and no changes had been made.
I'm still getting the "Poller[0] ERROR: A DB Exec Failed!, Error:'1062'" in the cacti log which I cannot explain, though. I give up on that.

Thanks!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group