I don't see traffic in my graphs at intervals

[farukbostanci]

Hi,

I've got a problem that I couldn't solve yet.
Sometimes I don't see the traffic on some cacti graphs at intervals.
Not all of them. it 's changing. ( you can see screnshot of this situation it is attached. all of graphs on screnshot are scaled at same time )

I use spine for poller.. and it's complete the process in 8 second ;
I have 326 active device on cacti database and nearly 3000 interfaces that I scale.

time ./spine

real 0m8.0047

What could be the problem ? Can anyone help me ?

is it about rrd ?

[And]

I've been experiencing the same problem with Spine, though not often. Yet in the last few days the gaps started to appear every 5 minutes or kinf of, forcing me to revert to cmd.php as poller. The polling process takes arounf 8-9 seconds with spine. I couldn't relate this behaviour with any change in my configuration.
I'm more than happy to run some tests if needed.

And

[And]

More info: I run Cacti under windows, no error in the log file and still no gap when running the cmd.php as a poller.
Please let me know if more information are needed.
Thanks!

And

[farukbostanci]

Hi,

The gaps still show up randomly.
RRDprocessed : 4994
and i see a lot of same log like below ;

ERROR: illegal attempt to update using time ..... when last update time is ...... ( minimum one second step )


by the way spine process complete in 7sn but Poller output ;
POLLER: Poller [0] Maximum runtime of 298 seconds exceeded. Exiting.



Method : spine Processes:30 Threads:40 Hosts:292 HostPerProcess:10 Datasources: 5726

[TheWitness]

Ok, with 30 processes and 40 threads, you would need approximately 1200 MySQL connection. This typically does not work and will cause major crashing.

Recommendation:

Processes <= (CPUS * CORES * 2)
Threads <= 20

Also, make sure you have sufficient connections in MySQL

Max Connections = (Connections for Users) + Processes * (Threads + Concurrent Script Servers)

TheWitness

[farukbostanci]

Thank you for quick responce.

actually I was confused a lot. There is a specific error log but people have different approaches.

So i changed spine to cmd again one hour ago and it looks fine now.

But if I have to turn spine i'll consider that you told.

what do you think in my circumstances ? cmd is enough ?

my.cnf ;

#max_connections = 2000

[gandalf]

Use processes = 2-4 (max 2 * number of available cores)
Use threads = 10-15

Use connections = 500
Restart mysql

Reinhard

[farukbostanci]

thank you

I returned to spine again with combination that you specified.

And it's going well now. But there is one thing more that sould work but it doesn't.

I run " php poller.php " in cacti folder but it returns me nothing.

I want to see poller performance and system stats but i cant

[farukbostanci]

Ok i can see now.


I see the poller exceeded

do i have to reconfigure parameters of ( process, threads )


General Information
Date Fri, 14 Nov 2008 10:56:01 +0200
Cacti Version 0.8.7b
Cacti OS unix
SNMP Version net-snmp
RRDTool Version RRDTool 1.2.x
Hosts 287
Graphs 2959
Data Sources SNMP: 443
SNMP Query: 2656
Script Query - Script Server: 12
Total: 3111
Poller Information
Interval 300
Type spine
Items Action[0]: 5743
Action[2]: 20
Total: 5763
Concurrent Processes 2
Max Threads 15
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:299.0555 Method:spine Processes:2 Threads:15 Hosts:288 HostsPerProcess:144 DataSources:5763 RRDsProcessed:4921
PHP Information
PHP Version 5.2.4-2ubuntu5.3
PHP OS Linux
PHP uname Linux cacti-desktop 2.6.24-19-generic #1 SMP Wed Aug 20 22:56:21 UTC 2008 i686
PHP SNMP Installed
max_execution_time 30
memory_limit 256M

[farukbostanci]

There's a new occurred problem ;

My old graphs that belong to old hosts are still working well.

but when i want to create new graph, i can't see any graph...

debug is below ;

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="gw_m_yonca_gida_kordon.izmir.bt - 111.111.111.111" \
--rigid \
--base=1000 \
--height=120 \
--width=675 \
--upper-limit=2048000 \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:13: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/lib/cacti/rra/gw_m_yonca_gida_kordon_izmir_bt_traffic_in_5403.rrd":traffic_in:AVERAGE \
DEF:b="/var/lib/cacti/rra/gw_m_yonca_gida_kordon_izmir_bt_traffic_in_5403.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdefe=b,8,* \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s\n"

RRDTool Says:

ERROR: opening '/var/lib/cacti/rra/gw_m_yonca_gida_kordon_izmir_bt_traffic_in_5403.rrd': No such file or directory

[gandalf]

In most cases, the downed host detection selected for that host returns a failure. E.g. is you select PING tests but do not provide special permssions required to run ICMP test in php.
Reinhard

[farukbostanci]

Hi gandalf,

I create this device again without ping detection. I used snmp for detection and everything seems ok.

But i can't still see the graph.
graphs of all devices that I created before are still working well without any interruption.

But i can't see the graph of new device even device returns snmp information.

cacti logs ;

11/16/2008 10:46:00 PM - SYSTEM STATS: Time:299.3378 Method:spine Processes:2 Threads:25 Hosts:283 HostsPerProcess:142 DataSources:5779 RRDsProcessed:5792
11/16/2008 10:46:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.


should i turn to cmd ? or can i fix this ? I am trying to figure out what is the problem but i couldn't find anything yet.

graph template debug ;

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="gw_m_acibadem_sigorta_izmir.izmir.bt - ***.***.***.***" \
--rigid \
--base=1000 \
--height=120 \
--width=675 \
--upper-limit=2048000 \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:13: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/lib/cacti/rra/gw_m_acibadem_sigorta_izmir_izmir_bt_traffic_in_5408.rrd":traffic_in:AVERAGE \
DEF:b="/var/lib/cacti/rra/gw_m_acibadem_sigorta_izmir_izmir_bt_traffic_in_5408.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdefe=b,8,* \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s\n"

RRDTool Says:

ERROR: opening '/var/lib/cacti/rra/gw_m_acibadem_sigorta_izmir_izmir_bt_traffic_in_5408.rrd': No such file or directory

[gandalf]

This may be a result of the poller not being able to finish in 5 Minutes. You should tackle this problem first. Even with > 5000 data sources, the poller should not run that long. Are you using mostly scripts?
Reinhard

[farukbostanci]

hi gandalf,

I am using mostly get snmp data ( indexed ).

actually this is occured when we use spine.

from last week we're using cmd for poller and system stats are below ;

11/26/2008 10:07:47 AM - SYSTEM STATS: Time:106.3170 Method:cmd.php Processes:4 Threads:N/A Hosts:282 HostsPerProcess:71 DataSources:5915 RRDsProcessed:3165
11/26/2008 10:07:47 AM - CMDPHP: Poller[0] Time: 105.7435 s, Theads: N/A, Hosts: 70

But sometimes there are gaps on download partition of rrd graphs.

You can see what i mean on attached gif ;

[gandalf]

Do you have the STATS of the time of dropout?
There are several possible reasons, e.g. a script lasting too long and blocking the thread for the snmp queries of this device.
Reinhard