Cacti (home)ForumsRepositoryDocumentation
Cacti: offical forums and support  

 FAQFAQ   SearchSearch   MemberlistMemberlist    RegisterRegister   ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in    


Syslog Plugin Very Slow

 
Post new topic   Reply to topic    Cacti Forum Index -> Plugin General
Author Message
chrisgapske
Cacti User


Joined: 22 May 2007
Posts: 264
Location: Padacuh, Ky-Alpena, MI-Gulf Shores,AL
PostPosted: Tue Feb 19, 2008 9:46 am    Post subject: Syslog Plugin Very Slow Reply with quote
WHen I am using the syslog plugin it can take up to two min to come back to me with the data results.
Has anybody done any work out there to speed this process up.

Any Ideas.
Back to top
warnesj
Cacti User


Joined: 29 May 2005
Posts: 163
PostPosted: Wed Feb 20, 2008 3:26 pm    Post subject: Reply with quote
Yes and no.

If I understand your question, and how the syslog plugin works, the reason it takes a couple of minutes (or up to 5 minutes) for the syslog messages to show up is because of the polling period of Cacti. The syslog messages are actually stored in a temporary table of your syslog database as they are recieved (the syslog_incoming table). Then when the main Cacti poller runs the syslog poller, those messages are examined and any filters you've defined are applied. After that the remaining messages are checked against any alerts you've defined. Once all the filters and alerts have been run, the filtered messages are then stored in the another table which is viewable in Cacti (the syslog table).

So is there a way to make those come in faster? Yes, run your poller at more frequent intervals. Though I don't know what this will do to the rest of your stuff.
Back to top
chrisgapske
Cacti User


Joined: 22 May 2007
Posts: 264
Location: Padacuh, Ky-Alpena, MI-Gulf Shores,AL
PostPosted: Wed Feb 20, 2008 10:44 pm    Post subject: Reply with quote
warnesj wrote:
Yes and no.

If I understand your question, and how the syslog plugin works, the reason it takes a couple of minutes (or up to 5 minutes) for the syslog messages to show up is because of the polling period of Cacti. The syslog messages are actually stored in a temporary table of your syslog database as they are recieved (the syslog_incoming table). Then when the main Cacti poller runs the syslog poller, those messages are examined and any filters you've defined are applied. After that the remaining messages are checked against any alerts you've defined. Once all the filters and alerts have been run, the filtered messages are then stored in the another table which is viewable in Cacti (the syslog table).

So is there a way to make those come in faster? Yes, run your poller at more frequent intervals. Though I don't know what this will do to the rest of your stuff.



OK Almost but not not the problem I am having.

I am talking about using syslog plugin when I use the plugin it is very slow for me. Not the input of data from syslog sources but using the plugin from selecting to clicking opjects takes a very long time. I even cut my data storage to 2 days to somewhat shrink the database. It is just super slow.
Back to top
eternal
Cacti User


Joined: 14 Dec 2006
Posts: 57
Location: Kingsport TN
PostPosted: Thu Feb 21, 2008 9:01 am    Post subject: Reply with quote
Try to optimize the syslog mysql table.

OPTIMIZE TABLE table_name[,table_name]
Back to top
chrisgapske
Cacti User


Joined: 22 May 2007
Posts: 264
Location: Padacuh, Ky-Alpena, MI-Gulf Shores,AL
PostPosted: Thu Feb 21, 2008 9:16 am    Post subject: Solved Reply with quote
Thank you that helped. I am still a nebie DBA.
Back to top
sterpstra



Joined: 27 May 2008
Posts: 33
Location: So Cal
PostPosted: Wed Jun 25, 2008 12:09 pm    Post subject: Table Search Reply with quote
Hey guys...

Love this plugin!! It fits well into the system but I'm having some performance issues and wanted to get some opinions... I have several systems dumping their syslogs to this server. I originally had the archive set at 30 days and after the first 3 days, I was at about 20 million rows... I have since changed to 3 days but still average 8-10 million rows. My problem is two-fold:

1. It takes between 8-12 minutes to even bring up the plugin
2. It takes more than 15 minutes to search by Host/Text.

This is currently on a box with 2GB memory and an Intel Duo-core (I think 2.0Ghz). I understand there are obviously some general hardware issues when searching and I've just ordered a new Dell server with 4GB mem, Xeon Quad-Core and 10K drives. I'm curious how much this will affect the queries in SYSLOG. I would like to increase both the amount of hosts sending to this server as well as the days (would like 7 at least) before deletion.

Anyone had similar issues?
Back to top
TheWitness
Developer


Joined: 14 May 2002
Posts: 9671
Location: MI, USA
PostPosted: Thu Jun 26, 2008 2:04 pm    Post subject: Reply with quote
Depending on when you installed, you may have missed some very critical indexes. Do the following:

Code:
mysql syslog
> show create table syslog;


From there, you should see the following:

Code:
CREATE TABLE  `syslog` (
  `facility` varchar(10) default NULL,
  `priority` varchar(10) default NULL,
  `date` date default NULL,
  `time` time default NULL,
  `host` varchar(128) default NULL,
  `message` text,
  `seq` bigint(20) unsigned NOT NULL auto_increment,
  PRIMARY KEY  (`seq`),
  KEY `date` (`date`),
  KEY `time` (`time`),
  KEY `host` (`host`),
  KEY `priority` (`priority`),
  KEY `facility` (`facility`)
) ENGINE=MyISAM AUTO_INCREMENT=6498155 DEFAULT CHARSET=latin1;


You will note the "KEY" statements. If you are missing some, issue the following per index:

Code:
ALTER TABLE syslog ADD INDEX `field` (`field`);


You can combine multiple ADD's into one statement. I believe that you must separate with comma though.

TheWitness
Back to top
sterpstra



Joined: 27 May 2008
Posts: 33
Location: So Cal
PostPosted: Thu Jun 26, 2008 2:49 pm    Post subject: Reply with quote
Quote:
mysql> show create table syslog;
syslog | CREATE TABLE `syslog` (
`facility` varchar(10) default NULL,
`priority` varchar(10) default NULL,
`date` date default NULL,
`time` time default NULL,
`host` varchar(128) default NULL,
`message` text,
`seq` int(10) unsigned NOT NULL auto_increment,
PRIMARY KEY (`seq`),
KEY `date` (`date`),
KEY `time` (`time`),
KEY `host` (`host`),
KEY `priority` (`priority`),
KEY `facility` (`facility`)
) ENGINE=MyISAM AUTO_INCREMENT=28197952 DEFAULT CHARSET=latin1 |

1 row in set (0.02 sec)


Looks like they are all there
Back to top
TheWitness
Developer


Joined: 14 May 2002
Posts: 9671
Location: MI, USA
PostPosted: Fri Jun 27, 2008 8:38 pm    Post subject: Reply with quote
Looks like you need to tweak your MySQL Settings for syslog. Tell me about your system:

Quote:
OS
CPU's
Core's
Physical Memory
Disk Number on MySQL mount
du -kh /var/www/html/cacti/rra
cat > /etc/my.cnf


Please provide that information and I will comment further. However, you need to optimize your system for MySQL. Syslog can become quite big, to the point of requiring your Cacti installation to have 1) fast disk (raid0 stripped, and 2) lot's of memory) or moving Syslog itself to it's own database. If you have lot's of RRA's "du -kh" is more than 60% of your physical memory, you need to split the web site and database into separate servers.

I have been working with one of the Cacti developers on a Syslog update that will resolve "some" of the issues.

TheWitness
Back to top
sterpstra



Joined: 27 May 2008
Posts: 33
Location: So Cal
PostPosted: Fri Jun 27, 2008 9:24 pm    Post subject: Reply with quote
Quote:
OS
CPU's
Core's
Physical Memory
Disk Number on MySQL mount
du -kh /var/www/html/cacti/rra
cat > /etc/my.cnf


Thanks for the help... Here ya go:

Ubuntu 8.04/Hardy [Linux 2.6.24-16-server]
1 CPU x Intel Core 2 3.9Ghz
2GB RAM

207M /var/lib/cacti/rra
CAT Results:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 8
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
skip-bdb
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
Back to top
TheWitness
Developer


Joined: 14 May 2002
Posts: 9671
Location: MI, USA
PostPosted: Sat Jun 28, 2008 10:04 pm    Post subject: Reply with quote
Increase your key bugger to 768MB. That should help a bit. Then restart MySQL.

TheWitness
Back to top
streaker69
Cacti Pro User


Joined: 27 Mar 2006
Posts: 647
Location: Psychic Amish Network Administrator
PostPosted: Sat Jun 28, 2008 10:27 pm    Post subject: Reply with quote
TheWitness wrote:
Increase your key bugger to 768MB. That should help a bit. Then restart MySQL.

TheWitness


You do mean 'buffer', right?
Back to top
TheWitness
Developer


Joined: 14 May 2002
Posts: 9671
Location: MI, USA
PostPosted: Sun Jun 29, 2008 8:55 am    Post subject: Reply with quote
lol. Yes. Just got back from Japan. I guess I needed sleep more than I thought

TheWitness
Back to top
spoonman
Cacti User


Joined: 03 May 2005
Posts: 243
PostPosted: Tue Jul 08, 2008 2:40 pm    Post subject: Reply with quote
Hello all....I seem to have the same issue and here are my thoughts on the plugin slowness to view...which btw I love and its so helpful especially having it as a plugin in cacti. I have about 30 or so devices i receive syslog data from...it seems to me that the reason syslog tab seems slow is the amount of data its trying to read but in the end it only shows the top 100 or whatever entries on the page....to me could the plugin only pull those top 100 or 200 entries from the database to present to the viewer?? Can the viewer only view the top rows?? instead of it seems to pull the whole table before it presents the view?? just my opinion...by no means do I know how the code is or am i a dba///

Thanks guys...esp Witness...whose helped me numerous times..

Spoon
Back to top
cigamit
Developer


Joined: 07 Apr 2005
Posts: 945
Location: B/CS Texas
PostPosted: Sun Sep 07, 2008 2:04 pm    Post subject: Reply with quote
spoonman wrote:
Hello all....I seem to have the same issue and here are my thoughts on the plugin slowness to view...which btw I love and its so helpful especially having it as a plugin in cacti. I have about 30 or so devices i receive syslog data from...it seems to me that the reason syslog tab seems slow is the amount of data its trying to read but in the end it only shows the top 100 or whatever entries on the page....to me could the plugin only pull those top 100 or 200 entries from the database to present to the viewer?? Can the viewer only view the top rows?? instead of it seems to pull the whole table before it presents the view?? just my opinion...by no means do I know how the code is or am i a dba///

Thanks guys...esp Witness...whose helped me numerous times..

Spoon


It should only be pulling the exact number of messages it needs. The biggest issue is when there are no indexes on the table, so it is extremely slow to query which results it needs.
Back to top
Display posts from previous:   
Post new topic   Reply to topic    Cacti Forum Index -> Plugin General All times are GMT - 5 Hours
Page 1 of 1

 



Powered by phpBB © 2001, 2005 phpBB Group