Cacti (home)ForumsRepositoryDocumentation
Cacti: offical forums and support
It is currently Fri Apr 18, 2014 6:15 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 31 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Spine 0.8.7c Beta 2
PostPosted: Thu Oct 16, 2008 7:52 pm 
Offline
Developer/Forum Admin
User avatar

Joined: Mon Nov 17, 2003 6:35 pm
Posts: 5903
Location: Michigan, USA
This is the discussion thread for Spine 0.8.7c Beta 2 support issues.

_________________
Tony Roman
Experience is what causes a person to make new mistakes instead of old ones.
There are only 3 way to complete a project: Good, Fast or Cheap, pick two.
With age comes wisdom, what you choose to do with it determines whether or not you are wise.


Top
 Profile  
 
 Post subject: spine: free(): invalid next size (fast)
PostPosted: Fri Oct 24, 2008 10:04 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
Good day!

Just downloaded and compiled beta-2 of spine 0.8.7c.
Compiling went well, but when running spine, it aborts after about 70secs with the following message.

*** glibc detected *** ./spine: free(): invalid next size (fast): 0x00000000134e9dc0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x36a2471634]
/lib64/libc.so.6(cfree+0x8c)[0x36a2474c5c]
./spine[0x40825f]
./spine[0x409ae3]
/lib64/libpthread.so.0[0x36a30062f7]
/lib64/libc.so.6(clone+0x6d)[0x36a24d1b6d]

The full error message has been attached to this post.

Some system info:
RHEL5.2
Linux 2.6.18-8.1.14.el5 #1 SMP Tue Sep 25 11:45:55 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

Cacti info:
Hosts: 607, Datasources: 10182, RRDs: 6937

Poller info:
Maximum Threads per Process: 5 (Reduced from 10 to 5)
Number of PHP Script Servers: 3 (Reduced from 5 to 3)
The Maximum SNMP OID's Per SNMP Get Request: 20 (Red. from 40 to 20)

Any more info required, let me know.

Cheers!
tobias


Attachments:
File comment: The error message.
081024-spine-087c-beta2-error.txt [3.99 KiB]
Downloaded 448 times
Top
 Profile  
 
 Post subject:
PostPosted: Wed Nov 05, 2008 7:52 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Please do the following:

Code:
cd <spine source directory>
gdb spine
b *0x40825f
b *0x409ae3
quit


When you type in the two break points, you should get a line number printed out for some line in one of the *.c files. Please let me know the line numbers and file names.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Thu Nov 06, 2008 3:52 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
G'day!

Here we go..
Output from gdb:

This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".

(gdb) b *0x40825f
Breakpoint 1 at 0x40825f: file poller.c, line 1026.
(gdb) b *0x409ae3
Breakpoint 2 at 0x409ae3: file poller.c, line 53.

Cheers!


Top
 Profile  
 
 Post subject:
PostPosted: Thu Nov 06, 2008 6:16 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Try this patch and run again. I had a few buffers not sized quite right.

Code:
Index: poller.c
===================================================================
--- poller.c   (revision 4800)
+++ poller.c   (working copy)
@@ -97,7 +97,7 @@
   char query8[BUFSIZE];
   char query9[BUFSIZE];
   char query10[BUFSIZE];
-   char errstr[BUFSIZE];
+   char errstr[RESULTS_BUFFER];
   char sysUptime[BUFSIZE];
   char result_string[BUFSIZE];
   int  result_length;
@@ -737,7 +737,7 @@

                              /* detect erroneous non-numeric result */
                              if (!validate_result(snmp_oids[j].result)) {
-                                 snprintf(errstr, BUFSIZE, "%s", snmp_oids[j].result);
+                                 snprintf(errstr, RESULTS_BUFFER, "%s", snmp_oids[j].result);

                                 if (!STRIMATCH(snmp_oids[j].result, "Nan")) {
                                    SPINE_LOG(("Host[%i] DS[%i] WARNING: Result from SNMP not valid. Partial Result: %.100s...\n", host_id, poller_items[snmp_oids[j].array_position].local_data_id, errstr));
@@ -748,7 +748,7 @@
                           }
                        }

-                        snprintf(poller_items[snmp_oids[j].array_position].result, 254, "%s", snmp_oids[j].result);
+                        snprintf(poller_items[snmp_oids[j].array_position].result, RESULTS_BUFFER, "%s", snmp_oids[j].result);

                        SPINE_LOG_MEDIUM(("Host[%i] DS[%i] SNMP: v%i: %s, dsname: %s, oid: %s, value: %s\n", host_id, poller_items[snmp_oids[j].array_position].local_data_id, host->snmp_version, host->hostname, poller_items[snmp_oids[j].array_position].rrd_name, poller_items[snmp_oids[j].array_position].arg1, poller_items[snmp_oids[j].array_position].result));
                     }
@@ -798,7 +798,7 @@

                           /* detect erroneous non-numeric result */
                           if (!validate_result(snmp_oids[j].result)) {
-                              snprintf(errstr, BUFSIZE, "%s", snmp_oids[j].result);
+                              snprintf(errstr, RESULTS_BUFFER, "%s", snmp_oids[j].result);

                              if ((!STRIMATCH(snmp_oids[j].result, "nan")) &&
                                 (!STRIMATCH(snmp_oids[j].result, "U"))) {
@@ -810,7 +810,7 @@
                        }
                     }

-                     snprintf(poller_items[snmp_oids[j].array_position].result, BUFSIZE, "%s", snmp_oids[j].result);
+                     snprintf(poller_items[snmp_oids[j].array_position].result, RESULTS_BUFFER, "%s", snmp_oids[j].result);

                     SPINE_LOG_MEDIUM(("Host[%i] DS[%i] SNMP: v%i: %s, dsname: %s, oid: %s, value: %s\n", host_id, poller_items[snmp_oids[j].array_position].local_data_id, host->snmp_version, host->hostname, poller_items[snmp_oids[j].array_position].rrd_name, poller_items[snmp_oids[j].array_position].arg1, poller_items[snmp_oids[j].array_position].result));

@@ -847,7 +847,7 @@

                  /* detect erroneous result. can be non-numeric */
                  if (!validate_result(poller_items[i].result)) {
-                     snprintf(errstr, sizeof(errstr), "%s", poller_items[i].result);
+                     snprintf(errstr, RESULTS_BUFFER, "%s", poller_items[i].result);

                     if (!STRIMATCH(poller_items[i].result, "Nan")) {
                        SPINE_LOG(("Host[%i] DS[%i] WARNING: Result from SCRIPT not valid. Partial Result: %.20s...\n", host_id, poller_items[i].local_data_id, errstr));
@@ -886,7 +886,7 @@

                  /* detect erroneous result. can be non-numeric */
                  if (!validate_result(poller_items[i].result)) {
-                     snprintf(errstr, sizeof(errstr), "%s", poller_items[i].result);
+                     snprintf(errstr, RESULTS_BUFFER, "%s", poller_items[i].result);

                     if (!STRIMATCH(poller_items[i].result, "Nan")) {
                        SPINE_LOG(("Host[%i] DS[%i] SS[%i] WARNING: Result from SERVER not valid.  Partial Result: %.20s...\n", host_id, poller_items[i].local_data_id, php_process, errstr));
@@ -938,7 +938,7 @@

                  /* detect erroneous non-numeric result */
                  if (!validate_result(snmp_oids[j].result)) {
-                     snprintf(errstr, sizeof(errstr), "%s", snmp_oids[j].result);
+                     snprintf(errstr, RESULTS_BUFFER, "%s", snmp_oids[j].result);

                     if (!STRIMATCH(snmp_oids[j].result, "Nan")) {
                        SPINE_LOG(("Host[%i] DS[%i] WARNING: Result from SNMP not valid. Partial Result: %.20s...\n", host_id, poller_items[snmp_oids[j].array_position].local_data_id, errstr));
@@ -949,7 +949,7 @@
               }
            }

-            snprintf(poller_items[snmp_oids[j].array_position].result, 254, "%s", snmp_oids[j].result);
+            snprintf(poller_items[snmp_oids[j].array_position].result, RESULTS_BUFFER, "%s", snmp_oids[j].result);

            SPINE_LOG_MEDIUM(("Host[%i] DS[%i] SNMP: v%i: %s, dsname: %s, oid: %s, value: %s\n", host_id, poller_items[snmp_oids[j].array_position].local_data_id, host->snmp_version, host->hostname, poller_items[snmp_oids[j].array_position].rrd_name, poller_items[snmp_oids[j].array_position].arg1, poller_items[snmp_oids[j].array_position].result));

@@ -963,7 +963,7 @@
      }

      /* insert the query results into the database */
-      if (!(query3 = (char *)malloc(MAX_MYSQL_BUF_SIZE+BUFSIZE))) {
+      if (!(query3 = (char *)malloc(MAX_MYSQL_BUF_SIZE+RESULTS_BUFFER))) {
         die("ERROR: Fatal malloc error: poller.c query3 oids!");
      }
      query3[0] = '\0';
@@ -973,7 +973,7 @@

      i = 0;
      while (i < rows_processed) {
-         snprintf(result_string, BUFSIZE, " (%i,'%s','%s','%s')",
+         snprintf(result_string, RESULTS_BUFFER+SMALL_BUFSIZE, " (%i,'%s','%s','%s')",
            poller_items[i].local_data_id,
            poller_items[i].rrd_name,
            host_time,


TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Thu Nov 06, 2008 8:16 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
Patch applied and spine recompiled..
I still get the same error. (see attached file.)
GDB still points out lines 1026 and 53.?..

gdb output:
------------
(gdb) b *0x40825f
Breakpoint 1 at 0x40825f: file poller.c, line 1026.

(gdb) b *0x409ae3
Breakpoint 2 at 0x409ae3: file poller.c, line 53.


poller.c:
----------
line 53 thread_mutex_lock(LOCK_THREAD);

line 1026 free(snmp_oids);


Any advise?
Cheers!


Attachments:
File comment: The error message after patching poller.c...
081106-spine-087c-beta2-error-after-patch.txt [11.53 KiB]
Downloaded 378 times
Top
 Profile  
 
 Post subject:
PostPosted: Thu Nov 06, 2008 9:38 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Try to do this with 1 thread and see if it cracks. You can run a stress test by using the Read Only flag:

Code:
while [ 1 ]; do ./spine -R;done


TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 5:05 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
Hi!
No luck with running it in a endless loop..
It always fails at the same point and (of course) starts off again...

After that I was running it with '--verbosity=5 -R' options. Output is attached to this post.
It's interesting to see, that 186 hosts are being processed and then it fails.
As of today we've got a total of 650 hosts online.

Maybe the debug output helps?

Thanks for your time anyway... :-)

Cheers!


Attachments:
File comment: &quot;./spine --verbosity=5 -R&quot;
081107-spine-verbose-debug.txt [828.68 KiB]
Downloaded 331 times
Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 8:05 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Is it always failing at the same host?

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 8:08 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Send me your configure options and configure output.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 8:23 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Also,

Do you receive any warning messages when running spine? You did not send a complete log.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 8:49 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14861
Location: MI, USA
Please test the attached poller.c. It has some custom output that I will be interested in.

TheWitness


Attachments:
poller.c [47.05 KiB]
Downloaded 355 times

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository
Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 8:55 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
Quote:
Is it always failing at the same host?

Yes. It's failing always at the same host (Host[294]).
Disabling the host does not help, as spine is failing on the next one (Host[295])...

Quote:
Send me your configure options and configure output.

Configure output and config.log have been attached.

Quote:
Do you receive any warning messages when running spine? You did not send a complete log.

No, I don't get any warnings.
The full log was not allowed to be uploaded because of its size.. I've removed some lines in the middle of the log. (Nothing interesting was removed. No errors nor warnings..)

Quote:
Please test the attached poller.c. It has some custom output that I will be interested in.

Yep. Give me a second, please... :-) I'll add it asap..


Again.. Thank you very much for taking your time!
Cheers!


Attachments:
File comment: config.log
081107-config.txt [64.88 KiB]
Downloaded 334 times
File comment: - yum list on prerequisites
- configure output

081107-configure-output.txt [8.49 KiB]
Downloaded 344 times
Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 9:08 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
Hmm.
I've tried to compile it, but 'make' exits with an error.

poller.c: In function âpoll_hostâ:
poller.c:733: error: expected â)â before â{â token
poller.c:751: error: expected â)â before â}â token
poller.c:751: error: expected expression before â}â token
poller.c:792: error: expected â)â before â{â token
poller.c:817: error: expected â)â before â}â token
poller.c:817: error: expected expression before â}â token
poller.c:923: error: expected â)â before â{â token
poller.c:948: error: expected â)â before â}â token
poller.c:948: error: expected expression before â}â token
make: *** [poller.o] Error 1


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 07, 2008 9:23 am 
Offline
User avatar

Joined: Mon Jan 30, 2006 11:47 am
Posts: 33
Location: Zurich, Switzerland
I've added the missing brackets in poller.c and now it compiles..
Give me a couple of minutes to masquerade IP's and snmp strings... :-)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 31 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group