Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sat Apr 29, 2017 4:35 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 33 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject:
PostPosted: Fri Apr 09, 2010 11:06 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
Aah, i thought you meant for us to stick with 0.8.7e or something :)


Top
 Profile  
 
 Post subject:
PostPosted: Sat Apr 10, 2010 10:41 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
That's the latest "official" release.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 11, 2010 5:27 pm 
Offline
Cacti User
User avatar

Joined: Mon Dec 13, 2004 2:55 pm
Posts: 145
Location: San Jose, CA
I have been working with Larry on this, and it looks like the latest checkin of spine.c (svn rev 5736) improves things quite a bit.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Apr 12, 2010 12:41 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
Alright, here's the stuff:
Code:
[root@cacti /usr/local/share/cacti]# php poller.php --force --debug
04/12/2010 07:27:11 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '9800-cisco_memfree-2010-04-12 07:27:07' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (9800,'cisco_memfree','2010-04-12 07:27:07','324528488'),(9801,'cisco_memused','2010-04-12 07:27:07','67030728'),(9802,'cat_systraf','2010-04-12 07:27:07','1'),(9803,'fiveMin','2010-04-12 07:27:07','1'),(9803,'oneMin','2010-04-12 07:27:07','1'),(9804,'fiveMin','2010-04-12 07:27:07','3'),(9804,'oneMin','2010-04-12 07:27:07','3'),(9805,'discards_in','2010-04-12 07:27:07','0'),(9805,'discards_out','2010-04-12 07:27:07','0'),(9805,'errors_in','2010-04-12 07:27:07','0'),(9805,'errors_out','2010-04-12 07:27:07','0'),(9806,'traffic_in','2010-04-12 07:27:07','380966458750'),(9806,'traffic_out','2010-04-12 07:27:07','1081543917436'),(9807,'errors_in','2010-04-12 07:27:07','0'),(9807,'discards_in','2010-04-12 07:27:07','0'),(9807,'discards_out','2010-04-12 07:27:07','0'),(9807,'errors_out','2010-04-12 07:27:07','0'),(9808,'traffic_in','2010-04-12 07:27:07','1462741080060'),(9808,'traffic_out','2010-04-12 07:27:07','4408169509160'),(9809,'errors_i'
04/12/2010 07:27:11 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '9800-cisco_memfree-2010-04-12 07:27:07' for key 1', SQL Fragment:'INSERT INTO poller_output_boost (local_data_id, rrd_name, time, output) VALUES (9800,'cisco_memfree','2010-04-12 07:27:07','324528488'),(9801,'cisco_memused','2010-04-12 07:27:07','67030728'),(9802,'cat_systraf','2010-04-12 07:27:07','1'),(9803,'fiveMin','2010-04-12 07:27:07','1'),(9803,'oneMin','2010-04-12 07:27:07','1'),(9804,'fiveMin','2010-04-12 07:27:07','3'),(9804,'oneMin','2010-04-12 07:27:07','3'),(9805,'discards_in','2010-04-12 07:27:07','0'),(9805,'discards_out','2010-04-12 07:27:07','0'),(9805,'errors_in','2010-04-12 07:27:07','0'),(9805,'errors_out','2010-04-12 07:27:07','0'),(9806,'traffic_in','2010-04-12 07:27:07','380966458750'),(9806,'traffic_out','2010-04-12 07:27:07','1081543917436'),(9807,'errors_in','2010-04-12 07:27:07','0'),(9807,'discards_in','2010-04-12 07:27:07','0'),(9807,'discards_out','2010-04-12 07:27:07','0'),(9807,'errors_out','2010-04-12 07:27:07','0'),(9808,'traffic_in','2010-04-12 07:27:07','1462741080060'),(9808,'traffic_out','2010-04-12 07:27:07','4408169509160'),(9809,'er'
04/12/2010 07:27:12 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '7190-cisco_memfree-2010-04-12 07:27:07' for key 1', SQL Fragment:'INSERT INTO poller_output_boost (local_data_id, rrd_name, time, output) VALUES (7190,'cisco_memfree','2010-04-12 07:27:07','324408520'),(7191,'cisco_memused','2010-04-12 07:27:07','67203048'),(7192,'cat_systraf','2010-04-12 07:27:07','3'),(7193,'fiveMin','2010-04-12 07:27:07','1'),(7193,'oneMin','2010-04-12 07:27:07','1'),(7194,'fiveMin','2010-04-12 07:27:07','3'),(7194,'oneMin','2010-04-12 07:27:07','3'),(7195,'errors_in','2010-04-12 07:27:07','0'),(7195,'discards_in','2010-04-12 07:27:07','0'),(7195,'discards_out','2010-04-12 07:27:07','0'),(7195,'errors_out','2010-04-12 07:27:07','0'),(7196,'traffic_in','2010-04-12 07:27:07','3296690199530'),(7196,'traffic_out','2010-04-12 07:27:07','5506951528252'),(7197,'errors_in','2010-04-12 07:27:07','0'),(7197,'discards_in','2010-04-12 07:27:07','0'),(7197,'discards_out','2010-04-12 07:27:07','0'),(7197,'errors_out','2010-04-12 07:27:07','0'),(7198,'traffic_in','2010-04-12 07:27:07','1012975778121'),(7198,'traffic_out','2010-04-12 07:27:07','3262527282336'),(7199,'d'
04/12/2010 07:27:15 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '16125-cisco_memfree-2010-04-12 07:27:08' for key 1', SQL Fragment:'INSERT INTO poller_output_boost (local_data_id, rrd_name, time, output) VALUES (16125,'cisco_memfree','2010-04-12 07:27:08','18212772'),(16126,'cisco_memused','2010-04-12 07:27:08','36318924'),(16127,'cisco_tempcur','2010-04-12 07:27:08','28'),(16128,'cisco_tempthr','2010-04-12 07:27:08','50'),(16129,'ping','2010-04-12 07:27:08','8.173'),(16130,'fiveMin','2010-04-12 07:27:08','4'),(16130,'oneMin','2010-04-12 07:27:08','3'),(16131,'errors_in','2010-04-12 07:27:08','0'),(16131,'discards_in','2010-04-12 07:27:08','0'),(16131,'discards_out','2010-04-12 07:27:08','19'),(16131,'errors_out','2010-04-12 07:27:08','0'),(16132,'traffic_in','2010-04-12 07:27:08','1152139869498'),(16132,'traffic_out','2010-04-12 07:27:08','369133410165'),(16133,'errors_in','2010-04-12 07:27:08','42'),(16133,'discards_in','2010-04-12 07:27:08','0'),(16133,'discards_out','2010-04-12 07:27:08','123'),(16133,'errors_out','2010-04-12 07:27:08','0'),(16134,'traffic_in','2010-04-12 07:27:08','580574637467'),(16134,'traffic_out','2010-04-12 07:'
No log handling enabled - turning on stderr logging
04/12/2010 07:28:26 AM - SPINE: Poller[0] WARNING: SS[0] The PHP Script Server did not respond in time and will therefore be restarted
04/12/2010 07:28:55 AM - SPINE: Poller[0] WARNING: SS[2] The PHP Script Server did not respond in time and will therefore be restarted
04/12/2010 07:28:56 AM - SPINE: Poller[0] WARNING: SS[1] The PHP Script Server did not respond in time and will therefore be restarted
04/12/2010 07:29:01 AM - SYSTEM STATS: Time:114.8978 Method:spine Processes:4 Threads:10 Hosts:430 HostsPerProcess:108 DataSources:42971 RRDsProcessed:0

And from rev5737:
Code:
[root@cacti ~]# spine -R -S -V 5 -H 45
SPINE: Using spine config file [/etc/cacti/spine.conf]
04/12/2010 11:05:10 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is /usr/local/share/cacti/script_server.php
04/12/2010 11:05:10 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /usr/local/share/cacti/log/cacti.log
DEBUG: The log_destination variable is 4 (STDOUT)

DEBUG: The path_php variable is /usr/local/bin/php

DEBUG: The availability_method variable is 4

DEBUG: The ping_recovery_count variable is 3

DEBUG: The ping_failure_count variable is 2

DEBUG: The ping_method variable is 2

DEBUG: The ping_retries variable is 2

DEBUG: The ping_timeout variable is 400

DEBUG: The snmp_retries variable is 3

DEBUG: The log_perror variable is 1

DEBUG: The log_pwarn variable is 0

DEBUG: The boost_redirect variable is 1

DEBUG: The log_pstats variable is 0

DEBUG: The threads variable is 10

DEBUG: The polling interval is 300 seconds

DEBUG: The number of concurrent processes is 4

DEBUG: The script timeout is 25

DEBUG: The number of php script servers to run is 4

DEBUG: Host List to be polled='45', TotalPHPScripts='1'

DEBUG: The PHP Script Server is Required

DEBUG: The Maximum SNMP OID Get Size is 10

Version 0.8.7f starting

DEBUG: MySQL is Thread Safe!

SPINE: Initializing Net-SNMP API

DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.

SPINE: Initializing PHP Script Server(s)

DEBUG: SS[0] PHP Script Server Routine Starting

DEBUG: SS[0] PHP Script Server About to FORK Child Process

DEBUG: SS[0] PHP Script Server Child FORK Success

DEBUG: SS[0] Confirmed PHP Script Server running using readfd[7], writefd[6]

DEBUG: SS[1] PHP Script Server Routine Starting

DEBUG: SS[1] PHP Script Server About to FORK Child Process

DEBUG: SS[1] PHP Script Server Child FORK Success

DEBUG: SS[1] Confirmed PHP Script Server running using readfd[9], writefd[8]

DEBUG: SS[2] PHP Script Server Routine Starting

DEBUG: SS[2] PHP Script Server About to FORK Child Process

DEBUG: SS[2] PHP Script Server Child FORK Success

DEBUG: SS[2] Confirmed PHP Script Server running using readfd[11], writefd[10]

DEBUG: SS[3] PHP Script Server Routine Starting

DEBUG: SS[3] PHP Script Server About to FORK Child Process

DEBUG: SS[3] PHP Script Server Child FORK Success

DEBUG: SS[3] Confirmed PHP Script Server running using readfd[13], writefd[12]

NOTE: Spine did not detect multithreaded device polling.

DEBUG: Initial Value of Active Threads is 0

DEBUG: Valid Thread to be Created

DEBUG: In Poller, About to Start Polling of Host

DEBUG: The Value of Active Threads is 1

DEBUG: Valid Thread to be Created

DEBUG: The Value of Active Threads is 2

DEBUG: In Poller, About to Start Polling of Host

Host[0] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

DEBUG: The Value of Active Threads is 1

Host[45] SNMP Result: Host responded to SNMP

Host[45] TH[1] RECACHE: Processing 2 items in the auto reindex cache for '<host>'

Host[45] TH[1] NOTE: There are '16' Polling Items for this Host

Host[45] TH[1] DS[2050] SS[0] SERVER: /usr/local/share/cacti/scripts/ss_cisco_cpu_usage.php ss_cisco_cpu_usage <host>:2:public:::::::161:500 get fiveMin 1, output: 56

Host[45] TH[1] DS[2050] SS[1] SERVER: /usr/local/share/cacti/scripts/ss_cisco_cpu_usage.php ss_cisco_cpu_usage <host>:2:public:::::::161:500 get oneMin 1, output: 56

Host[45] TH[1] DS[2048] SNMP: v2: <host>, dsname: cisco_memfree, oid: .1.3.6.1.4.1.9.9.48.1.1.1.6.1, value: 9306652

Host[45] TH[1] DS[2049] SNMP: v2: <host>, dsname: cisco_memused, oid: .1.3.6.1.4.1.9.9.48.1.1.1.5.1, value: 2861672

Host[45] TH[1] DS[11680] SNMP: v2: <host>, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.50, value: 414442

Host[45] TH[1] DS[11680] SNMP: v2: <host>, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.50, value: 0

Host[45] TH[1] DS[11680] SNMP: v2: <host>, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.50, value: 0

Host[45] TH[1] DS[11680] SNMP: v2: <host>, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.50, value: 0

Host[45] TH[1] DS[11681] SNMP: v2: <host>, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.50, value: 1757317390

Host[45] TH[1] DS[11681] SNMP: v2: <host>, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.50, value: 4025778552

Host[45] TH[1] DS[11682] SNMP: v2: <host>, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.51, value: 0

Host[45] TH[1] DS[11682] SNMP: v2: <host>, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.51, value: 0

Host[45] TH[1] DS[11682] SNMP: v2: <host>, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.51, value: 460430

Host[45] TH[1] DS[11682] SNMP: v2: <host>, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.51, value: 0

Host[45] TH[1] DS[11683] SNMP: v2: <host>, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.51, value: 2143588406

Host[45] TH[1] DS[11683] SNMP: v2: <host>, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.51, value: 1807432811

Host[45] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

DEBUG: The Value of Active Threads is 0

DEBUG: Thread Cleanup Complete

DEBUG: SS[0] Script Server Shutdown Started

DEBUG: SS[1] Script Server Shutdown Started

DEBUG: SS[2] Script Server Shutdown Started

DEBUG: SS[3] Script Server Shutdown Started

DEBUG: PHP Script Server Pipes Closed

DEBUG: Allocated Variable Memory Freed

DEBUG: MYSQL Free & Close Completed

Time: 1.6721 s, Threads: 10, Hosts: 2

I am getting poller timeouts more often now though:
Code:
04/12/2010 03:46:19 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
04/12/2010 03:46:19 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
04/12/2010 03:46:19 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
04/12/2010 03:46:19 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
04/12/2010 03:46:18 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine thread)

Edit: I reverted back to the one we worked on, TheWitness. I think its rev5624 (last one from Jan 18th)
Edit2: forgot to mention i saw some elevated CPU Usage and Load avg. Some 50% cpu and 4-5 more of the total load in the included load template (ucd/net)


Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 21, 2010 12:30 pm 
Offline
Cacti User
User avatar

Joined: Mon Dec 13, 2004 2:55 pm
Posts: 145
Location: San Jose, CA
I'm still seeing occasional duplicate entry errors in the logs with 10 threads and 1 poller process with spine.c svn rev 5736, but the frequency is greatly reduced (between 45 minutes to 8 hours with 5 min polling interval).


Top
 Profile  
 
 Post subject:
PostPosted: Fri Apr 23, 2010 3:28 pm 
Offline
Cacti User
User avatar

Joined: Mon Dec 13, 2004 2:55 pm
Posts: 145
Location: San Jose, CA
I have been testing with spine.c SVN rev 5787 since this morning, but still see intermittent duplicate entry errors with 1 process. Bumping the process count up to 4 makes the errors occur more frequently.


Top
 Profile  
 
 Post subject:
PostPosted: Sat Apr 24, 2010 12:42 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
5788 is all fixed now. Resolved the issue.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Sat Apr 24, 2010 2:29 am 
Offline
Cacti User
User avatar

Joined: Mon Dec 13, 2004 2:55 pm
Posts: 145
Location: San Jose, CA
TheWitness wrote:
5788 is all fixed now. Resolved the issue.

TheWitness


Thanks again Larry!

I am testing it out now, and will report back later this morning.


Top
 Profile  
 
 Post subject:
PostPosted: Sat Apr 24, 2010 12:07 pm 
Offline
Cacti User
User avatar

Joined: Mon Dec 13, 2004 2:55 pm
Posts: 145
Location: San Jose, CA
As you said, 5788 looks to have resolved the duplicate entry issue.

I tested with 1 process, then 4 process and re-enabled Balance Process Load and all appear to be working just fine.


Thanks for all your help!


Top
 Profile  
 
 Post subject:
PostPosted: Mon Apr 26, 2010 12:43 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
Works for me too!
Thanks TW :)

//David


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 27, 2010 12:49 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
Just wanted to say that rev5788 is very much more stable.
Previously we used to get a gap in basically all the graphs at least once a day, no reason at all. And on some devices random gaps at any time, ive thought them to be because of load on devices, or errors on the interfaces, and even thought about QoS:ing SNMP-traffic!

Its all fixed now!

Just have a look at the pic, this is one of those more troubled devices, i installed the new revision at ~7.30am Monday.
Edit: Forgot to say that once every 6h maybe we used to get a spike in the poller time and the poller would time out, hasnt happened since the new revision :)

Imo, this is a candidate for 0.8.8 :D <3

(tho i still have issues with the script server, but thats for another thread)


Attachments:
Screen shot 2010-04-27 at 07.37.11.png
Screen shot 2010-04-27 at 07.37.11.png [ 45.02 KiB | Viewed 6758 times ]
Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 15, 2010 9:08 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
I have to reopen this issue, i downloaded rev6054 from SVN (figured since 'g' was out, there might be some fixed bugs fixed in the svn).
And now im getting the 1062 error again:
Code:
07/15/2010 03:45:04 PM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '265-cisco_memfree-0000-00-00 00:00:00' for key 1', SQL Fragment:'INSERT INTO poller_output_boost (local_data_id, rrd_name, time, output) VALUES (265,'cisco_memfree','','325192384'),(266,'cisco_memused','','66265280') etc

07/15/2010 03:45:00 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 39786, Data Sources: hdd_total(DS[4073]), hdd_used(DS[4108]), hdd_used(DS[4109]), traffic_in(DS[4114]), traffic_out(DS[4114]), cpu(DS[4113]), cpu(DS[4112]), hdd_total(DS[4111]), hdd_used(DS[4111]), hdd_total(DS[4110]), hdd_used(DS[4110]), hdd_total(DS[4109]), hdd_total(DS[4108]), hdd_total(DS[4107]), hdd_used(DS[4107]), hdd_used(DS[4106]), hdd_total(DS[4106]), proc(DS[4102]), users(DS[4101]), (DS[4100]), hdd_used(DS[4073]), Additional Issues Remain. Only showing first 20


I am however still using Cacti 0.8.7e if theres a problem there.
Checking and unchecking 'Balance Process Load' did not do anything.

Any ideas?

I love rev6054 otherwise, went from ~170s polls to ~7s :)


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 15, 2010 4:27 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
That's pretty huge. Not enough information to tell you what's going on though.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 16, 2010 1:29 am 
Offline
Cacti User

Joined: Fri Sep 12, 2008 2:41 am
Posts: 168
Location: Sweden
Im not sure how to proceed, running poller.php with debug mode doesnt show anything about the MySQL error. However the cacti.log shows it. But its the 2nd line and nothing to show what would be causing it.

Ill see if i cant upgrade cacti aswell next week.

I think those 7s are from snmp only queries, i see script_servers still running long afterwards, when they close its about the same time elapsed as before.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 16, 2010 8:25 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
I know what the issue is. However, i am on vacation at the moment. If you put a usleep() after the pthread create, that should fix it in the near term.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 33 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group