Linux SNMP Memory Issue

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Moderators, Developers

Post Reply
Author
Message
jborg
Posts: 9
Joined: Mon Sep 07, 2015 3:26 am

Linux SNMP Memory Issue

#1 Post by jborg » Fri Aug 26, 2016 10:52 am

Hi,
I'm graphing the memory of a particular Linux server of ours and have noticed a discrepancy between what's graphed and the reality. Digging a little deeper, I found that the SNMP data being reported by the linux machine is different to what 'free -m' returns on the actual machine:

Output of free -m:

Code: Select all

free -m               
              total        used        free      shared  buff/cache   available
Mem:           4671         268        1743           9        2660        4105
Swap:          2047           0        2047
SNMP Output:

Code: Select all

UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 2097148 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 2097148 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 4783812 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 1832160 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 3929308 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memShared.0 = INTEGER: 9724 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 944 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 416204 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING: 
You'll notice that, whilst 'free -m' reports 2660M worth of buffered/cached memory, the SNMP output only reports 944kB + 416204kB. As a result of this, cacti is reporting high memory usage and alerts are being triggered for this particular server.

This is obviously not a Cacti issue but I thought of asking here in case anyone else has encountered this behavior?

Just as an FYI:
- The server in question is running CentOS Linux release 7.2.1511 (Core)
- The snmpd version on the server in question is 5.7.2; we have other servers running this version without any issue.
- I'm using the 'Host Memory - ucd/net - Memory Usage' though this is obviously fine...

Thanks,

Joe

jborg
Posts: 9
Joined: Mon Sep 07, 2015 3:26 am

Re: Linux SNMP Memory Issue

#2 Post by jborg » Mon Aug 29, 2016 5:09 am

I think I've more or less managed to answer this issue.

Looking at the output of cat /proc/meminfo on the server in question, I see:

Code: Select all

cat /proc/meminfo 
MemTotal:        4783812 kB
MemFree:          156800 kB
MemAvailable:    4205824 kB
Buffers:               0 kB
Cached:           184696 kB
SwapCached:          432 kB
Active:           193496 kB
Inactive:         131296 kB
Active(anon):      48808 kB
Inactive(anon):    97844 kB
Active(file):     144688 kB
Inactive(file):    33452 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        2089060 kB
Dirty:               420 kB
Writeback:             0 kB
AnonPages:        139688 kB
Mapped:            21936 kB
Shmem:              6556 kB
Slab:            4175288 kB
SReclaimable:    4124300 kB
SUnreclaim:        50988 kB
KernelStack:        6176 kB
PageTables:        16004 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4489052 kB
Committed_AS:     884692 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       76600 kB
VmallocChunk:   34359646208 kB
HardwareCorrupted:     0 kB
AnonHugePages:      8192 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       69568 kB
DirectMap2M:     5050368 kB
As can be seen, the memory is not used in buffers/cache but in SReclaimable. This is well explained here. The problem is that this detail is not reported by SNMP, so SReclaimable is included in used memory. I doubt there's any way around this other than writing some custom script and avoiding the use of SNMP to monitor this server. I thought I'd post my findings here in case anyone else faces this issue.

univsantamonica
Posts: 1
Joined: Tue Jan 14, 2020 9:34 am

Re: Linux SNMP Memory Issue

#3 Post by univsantamonica » Tue Jan 14, 2020 11:00 am

I also had this issue. The thing to note is that while SReclaimable is not available via SNMP, the definitions of what is available have changed in such a way that you can still create a workable stackable memory graph with a little bit of calculating.

My old memory graphs had (showing from top to bottom of graph):
Total mem (black line) = memTotalReal
Free (area:stack) = memTotalFree
Cached (area:stack) = memCached
Buffers (area:stack) = memBuffer
Used (area) = memTotalReal - memBuffer - memCached - memTotalFree

Everything added up and produced a clean graph.

With the changes to SNMP, memTotalFree had memAvailSwap added to it, and memCached had SReclaimable from /proc/meminfo added to it.

My new graph requires pulling in the memAvailReal data source, even though I never graph it.

I now use (again, showing from top to bottom of graph):
Total mem (black line) = memTotalReal
Free (area:stack) = memAvailReal - memBuffer - memCached
Cached (area:stack) = memCached
Buffers (area:stack) = memBuffer
Used (area) = memTotalReal - memAvailReal

While the new values may not match the old ones precisely, they are close enough to the values produced by /usr/bin/free to have the graph look accurate.

Below is the debug info for my graph, which shows the CDEFs I use. Note the DEF:f = mem_free = memAvaileReal isn't actually graphed. It's only used for calculations to get values as close as possible to free's used and free values.

The important thing is that the new calculations only involve values returned from SNMP memory OIDs.
DEF:a='/usr/share/cacti/rra/util2_mem_buffers_1101.rrd':'mem_buffers':AVERAGE \
DEF:b='/usr/share/cacti/rra/util2_mem_cached_1102.rrd':'mem_cached':AVERAGE \
DEF:c='/usr/share/cacti/rra/util2_mem_total_1105.rrd':'mem_total':AVERAGE \
DEF:d='/usr/share/cacti/rra/util2_swap_total_1106.rrd':'swap_total':AVERAGE \
DEF:e='/usr/share/cacti/rra/util2_swap_free_1104.rrd':'swap_free':AVERAGE \
DEF:f='/usr/share/cacti/rra/util2_mem_free_1103.rrd':'mem_free':AVERAGE \
CDEF:cdefa='c,f,-,1024,*' \
CDEF:cdefe='a,1024,*' \
CDEF:cdefi='b,1024,*' \
CDEF:cdefbc='f,b,-,a,-,1024,*' \
CDEF:cdefbg='c,1024,*' \
CDEF:cdefca='d,e,-,1024,*' \
CDEF:cdefce='d,1024,*' \
CDEF:cdefzzcf='e,1024,*' \
CDEF:cdefcg='f,1024,*' \
AREA:cdefa#007283FF:'Used Memory ' \
GPRINT:cdefa:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefa:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefa:MAX:'Maximum\:%8.2lf %s\n' \
AREA:cdefe#6EA100FF:'Buffers ':STACK \
GPRINT:cdefe:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefe:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefe:MAX:'Maximum\:%8.2lf %s\n' \
AREA:cdefi#C0C0C0FF:'Cached ':STACK \
GPRINT:cdefi:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefi:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefi:MAX:'Maximum\:%8.2lf %s\n' \
AREA:cdefbc#FFFFFFFF:'Free ':STACK \
GPRINT:cdefbc:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefbc:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefbc:MAX:'Maximum\:%8.2lf %s\n' \
LINE1:cdefbg#000000FF:'Total Memory\:' \
GPRINT:cdefbg:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefbg:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefbg:MAX:'Maximum\:%8.2lf %s\n' \
LINE1:cdefca#FFC73BFF:'Used Swap ' \
GPRINT:cdefca:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefca:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefca:MAX:'Maximum\:%8.2lf %s\n' \
GPRINT:cdefce:LAST:'Total Swap\:%8.2lf %s\n' \
GPRINT:cdefzzcf:LAST:'Free Swap\: %8.2lf %s\n' \
GPRINT:cdefcg:LAST:'Avail Mem%8.2lf %s\n'

Post Reply