graph gaps

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I used that latest version you recommended, and am still gettting the same error.

04/04/2005 01:00:04 PM - CACTID: Poller[0] ERROR: MySQL Connection failed aft
er 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysq
l.sock' (4)
04/04/2005 01:00:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Serv
er Terminating

I changed my cacti threads from 2 to 1, and restarted mysql. Is cacti trying to make too many connections to mysql so that mysql is denying them?

TheWitness · Posted: Mon Apr 04, 2005 2:34 pm Post subject:

- You should check for hung processes first. "ps -ef | grep cactid"

- If you don't have any, then check to see how many concurrent connections mysql is configured for. You can check that in my.conf/my.ini.

- If then you don't get happy with those two checks, my question would be: "does cactid only run intermittantly or not at all?"

- If it does not run at all, check your mysql library version vs. the installed version only if you are running MySQL 4.1 or higher.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

- You should check for hung processes first. "ps -ef | grep cactid"

Everytime the cactid fails it gets hung. I wrote a script to kill the hung processes once an hour.

- If you don't have any, then check to see how many concurrent connections mysql is configured for. You can check that in my.conf/my.ini.

I was running mysql in safe mode with no .cnf. I have now changed this and am running it with the 'my_huge.cnf" as my /etc/my.cnf file.

- If then you don't get happy with those two checks, my question would be: "does cactid only run intermittantly or not at all?"

Cactid fails about 10 percent of the times it runs, at various points during the poll.

- If it does not run at all, check your mysql library version vs. the installed version only if you are running MySQL 4.1 or higher.

Ill see if my changes help out.

TheWitness · Posted: Tue Apr 05, 2005 5:11 pm Post subject:

I'm working with a user who was having problems and he is ok now and had to perform an upgrade to net-snmp 5.1.1 to stop the segfaults.

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I changed the threading from 2 to 1, and it reduced the number of MYSQL errors from a few per hour, to 1 or 2 per day. I let it run like this for a few days, then upgraded from UCD-SNMP 4.2.3 to NET-SNMP-5.1.1. I am still seeing the MYSQL Errors occuring frequently when I have 2 threads enabled:

04/12/2005 04:13:29 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
04/12/2005 04:30:49 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)
04/12/2005 04:50:03 PM - CACTID: Poller[0] ERROR: MySQL Connection failed after 10 attempts: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (4)

The only difference I see is that my perfomance acutally slowed from 130 sec, to 140 sec.

How many connections is cactid attempting to make to MYSQL during the polling period? Is it possible MYSQL is not allowing the connections? Here is some stuff from my my.cnf file:

port = 3306
socket = /tmp/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
#set-variable = key_buffer=384M
set-variable = key_buffer=128M
set-variable = max_allowed_packet=1M
set-variable = max_connection=200
set-variable = table_cache=512
set-variable = sort_buffer=2M
set-variable = record_buffer=2M
set-variable = thread_cache=8
# Try number of CPU's*2 for thread_concurrency
set-variable = thread_concurrency=8
set-variable = myisam_sort_buffer_size=64M
#log-bin
server-id = 1

TheWitness · Posted: Tue Apr 12, 2005 4:32 pm Post subject:

The maximum connections that Cacti will use is (Processes*Treads)+1, but likely less. Could it be that there are issues with the MySQL client libraries and the release of connections after a process ends. Can you look at MySQL Admin and see if you are simply hanging on to old connections and if you can see what they were doing before hanging?

TheWitness

TheWitness · Posted: Tue Apr 12, 2005 4:35 pm Post subject:

Also, when you see these errors, are the processes still locking up?

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

Yes. I will see some hung rrdtool updates and php scripts.

TheWitness · Posted: Tue Apr 12, 2005 5:15 pm Post subject:

Which scripts? Which updates?

TheWitness

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

The failures occur at different places during the poll, I was not able to find any sort of trend. See my previous posts on this topic from Nov 23 and Feb17/18 for an example of what hangs.

Since I changed to 1 thread yesterday, I have not had a MySQL Connection failure, however, the poll is taking as long as 150 sec. I will run with 1 thread for a while, and then switch back to 2 to see the difference in performance and rate of failure.

TheWitness · Posted: Wed Apr 13, 2005 5:30 pm Post subject:

This then sounds like a pthread issue with Solaris. Can you look up bugs for your version of the OS relative to pthread support on the Sun web site? In addition, review the net-snmp bug pages on sourceforge.net for references to memory leaks/segfaults. My PC died yesterday, and I don't know when I will be back upto full speed.

TheWitness

gandalf · Posted: Thu Apr 14, 2005 9:17 am Post subject: mysql threads approach

To all cacti fans!
Yes, we also suffered from those gaps. But meanwhile, all graphs are clean. We have about 5580 poller_items in about 2375 rrds. The solution seemd to be the "tuning" of some mysql parms:

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I have made the mysql changes recommended above. Some of those features are not in our current version.

bash-2.03$ /usr/local/mysql/bin/mysql --version
/usr/local/mysql/bin/mysql Ver 11.18 Distrib 3.23.53, for sun-solaris2.8 (sparc)

We are able to get rid of the gaps, but the key seems to be the threading. With 2 threads enabled, I was getting 1-5 per MySQL connection failures per hour. After switching to 1 thread, I have only had 3 failures in 5 days. Any idea why threading would cause this problem? The machine is not under alot of load.

TheWitness · Posted: Tue Apr 19, 2005 3:52 pm Post subject:

Poor implementation of threading in solaris. Don't forget to re-maike php_snmp per my other thread. You can simply download the PHP CVS version for snmp.c and remake the entire package.

Larry

lard · Posted: Sat Dec 03, 2005 6:10 pm Post subject:

Hi Yardus,

Did you ever get this resolved? I ask as I am having the timeout messages and also the PHP crash?

Many thanks,

Larry