Sendmail & Mailscanner (alternate)

Templates, scripts for templates, scripts and requests for templates.

Moderators: Moderators, Developers

Author
Message
warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

Sendmail & Mailscanner (alternate)

#1 Post by warnesj » Mon May 30, 2005 11:11 pm

Long time listener, first time caller ;). Anyway I thought I'd throw in a few little scripts that I whiped up that allow me to get Sendmail and Mailscanner stats through SNMP and come up with cool graphs using Cacti like the one shown below.
Image
Ignore the little gap in the graph, I had to move my server and so it was down for a bit.

Net-SNMP has a nice little feature that allows you to execute a script and return the result back when an SNMP OID is queried. I should mention that my scripts run a bit different than those posted by africanw in that there is no requirement for Mailscanner to be logging to a database. Instead I opted to scrape through the last 5000 lines of the /var/log/maillog file for entries added in the last 5 minutes, so as a result it is more disk and CPU intensive than africanw's method but it makes it pretty easy to query from any SNMP manager. And Cacti loves SNMP (right???).

NOTE: If you're email server is a high-volume email server and your load average is running a little high the scripts may not complete in time before SNMP times out (we are scraping through syslog files after all). What will happen is that no data will be returned by Net-SNMP and you're graphs may be missing bits. Just thought I'd mention it before I get flamed for stuff that is out of my control. :)

Sendmail Stats
You need to create a couple of scripts, one to count the recieved mail and another to count the sent mail. For simplicity sakes I stored mine in /opt. Each script is shown below,
/opt/count_recv.sh,

Code: Select all

#!/bin/sh
echo .1.3.6.1.4.100.5
echo gauge
tail -n 5000 /var/log/maillog|awk '
BEGIN {
        search_time = "^" strftime("%b %e %H:%M", systime() - 5 * 60);
        search_sent = "stat=Sent";
        search_domain_1 = "@domain.com";
        search_domain_2 = "@domain2.com";
        recv_count = 0;
        }
{
        if ($0 ~ search_time && $0 ~ search_sent) time_trigger = 1;
        if ($0 ~ search_sent) is_sent_trigger = 1;
        if ($0 ~ search_domain_1) is_domain_trigger = 1;
        if ($0 ~ search_domain_2) is_domain_trigger = 1;
        if (time_trigger && is_sent_trigger && is_domain_trigger) recv_count++;
        is_sent_trigger = 0;
        is_domain_trigger = 0;
}
END { print recv_count; }'
exit
NOTE:You'll need to change domain.com and domain2.com to whatever your domain(s) are. If you only have one domain then you can take out the domain2.com search.
Now for a brief explanation. Everyone likes AWK right? :) It's a pretty simple script, it examines the last 5000 lines of /var/log/maillog for entries added in the last 5 minutes and looks for ones that has a Sendmail status of "Sent" to your domain(s) and then increments a counter if there is one.

/opt/count_sent.sh,

Code: Select all

#!/bin/sh
echo .1.3.6.1.4.100.6
echo gauge
tail -n 5000 /var/log/maillog|awk '
BEGIN {
        search_time = "^" strftime("%b %e %H:%M", systime() - 5 * 60);
        search_sent = "stat=Sent";
        search_domain_1 = "@domain.com";
        search_domain_2 = "@domain2.com";
        recv_count = 0;
        }
{
        if ($0 ~ search_time && $0 ~ search_sent) time_trigger = 1;
        if ($0 ~ search_sent) is_sent_trigger = 1;
        if ($0 ~ search_domain_1) is_domain_trigger = 1;
        if ($0 ~ search_domain_2) is_domain_trigger = 1;
        if (time_trigger && is_spam_trigger) spam_count++;
        if (time_trigger && is_not_spam_trigger) not_spam_count++;
        if (time_trigger && is_sent_trigger && !is_domain_trigger) sent_count++;
        is_sent_trigger = 0;
        is_domain_trigger = 0;
}
END { print sent_count; }'
exit
NOTE:You'll need to change domain.com and domain2.com to whatever your domain(s) are. If you only have one domain then you can take out the domain2.com search.
This script is similar to the count_recv.sh script except for one key difference. Basically this script looks for the same Sendmail status of "Sent" for any domain that isn't yours.

Now with the scripts in place there are some changes you need to make to your snmpd.conf file so that when the .1.3.6.1.4.100.5 and .1.3.6.1.4.100.6 OIDs are queried the scripts get run.
/etc/snmp/snmpd.conf additions,

Code: Select all

# Count the number of received messages in the last 5 minutes
pass .1.3.6.1.4.100.5 /bin/sh /opt/count_recv.sh
# Count the number of sent messages in the last 5 minutes
pass .1.3.6.1.4.100.6 /bin/sh /opt/count_sent.sh
As you can probably guess now when OID .1.3.6.1.4.100.5 is queried Net-SNMP runs the /opt/count_recv.sh script and when OID .1.3.6.1.4.100.6 is queried Net-SNMP runs the /opt/count_sent.sh script.

Mailscanner Stats
africanw's method has an advantage over mine here in that SPAM isn't identified as HIGHSPAM or LOWSPAM, it's only SPAM. As well viruses include both blocked files and viruses. No distiction. Sorry, this is a first version. I might work on fixing that later. Now again you'll need to create a couple of scripts, one to count the number of viruses that Mailscanner has detected and another to count the number of messages detected as SPAM.
/opt/count_viruses.sh

Code: Select all

#!/bin/sh
echo .1.3.6.1.4.100.4
echo gauge
tail -n 5000 /var/log/maillog|awk '
BEGIN {
        search_time = "^" strftime("%b %e %H:%M", systime() - 5 * 60);
        search_virus = "Virus Scanning: Found";
        virus_count = 0;
        }
{
        if ($0 ~ search_time && $0 ~ search_virus) time_trigger = 1;
        if ($0 ~ search_virus) is_virus_trigger = 1;
        if (time_trigger && is_virus_trigger) virus_count = virus_count + $9;
        is_virus_trigger = 0;
}
END { print virus_count; }'
exit
This script scrapes through the last 5000 lines of /var/log/maillog for entries added in the last 5 minutes and looks for the "Virus Scanning: Found" line that indicates how many viruses Mailscanner found in it's scan batch. Then adds that number to a counter.

/opt/count_spam.sh

Code: Select all

#!/bin/sh
echo .1.3.6.1.4.100.2
echo gauge
tail -n 5000 /var/log/maillog|awk '
BEGIN {
        search_time = "^" strftime("%b %e %H:%M", systime() - 5 * 60);
        search_is_spam = "is spam";
        spam_count = 0;
        }
{
        if ($0 ~ search_time && $0 ~ search_is_spam) time_trigger = 1;
        if ($0 ~ search_is_spam) is_spam_trigger = 1;
        if (time_trigger && is_spam_trigger) spam_count++;
        is_spam_trigger = 0;
}
END { print spam_count; }'
exit
This script looks for the "is spam" tag in /var/log/maillog to indicate that the message that Mailscanner is examining (usually with SpamAssassin) is SPAM.

And again you'll need to modify your snmpd.conf file so that the scripts get run when the OIDs are queried,
/etc/snmp/snmpd.conf additions,

Code: Select all

# Count the number of SPAM messages in the last 5 minutes
pass .1.3.6.1.4.100.2 /bin/sh /opt/count_spam.sh
# Count the number of viruses detected in the last 5 minutes
pass .1.3.6.1.4.100.4 /bin/sh /opt/count_viruses.sh
An now when .1.3.6.1.4.100.2 or .1.3.6.1.4.100.4 OIDs are queried Net-SNMP will run the count_spam.sh or count_viruses.sh script respectively.

Cacti Templates
Below is the Cacti Graph Template that I made up too (the Data Templates are included in the XML file since they are dependents). Now all you need to do is setup Cacti to start querying the above SNMP OIDs and then graph them using the template.
Attachments
Sendmail_Mailscanner_Template_0.8.6d.zip
Sendmail and Mailscanner Data and Graph Templates.
(3.53 KiB) Downloaded 1818 times
Last edited by warnesj on Tue Jun 07, 2005 12:33 pm, edited 2 times in total.

D43m0n
Posts: 7
Joined: Wed Jun 01, 2005 2:09 am

huh... suddenly script returning correct values?

#2 Post by D43m0n » Wed Jun 01, 2005 2:22 am

Hey great stuff!

I like the flat file approach, it's not that slow at all, I expected to wait a few seconds before a value is returned but hey, I doubt if one full second is needed.

I have a question though. I'm using Postfix instead of Sendmail. That shouldn't make any difference since the only changes needed are:
stat -> status and Sent -> sent. Pretty simple and straightforward, I like that :wink:

I added the stuff to my cacti host and my mailserver (other machine) and gather the data using SNMP. I did this yesterday around 5:00 PM and came in this morning, expecting to see beautiful graphs. I do see nice graphs, but for some strange reason, my sent script doesn't return anything anymore. And my received script always returns a zero for some reason. I'm not able to think of any logical reason why the scripts don't return the values anymore. This happened overnight. It seems that just when the clock ticked 0:00, the magic stopped working.

It's not related to SNMP, because I get the same result when I myself execute the scripts as a regular user (non-root) I was able to get great results yesterday as a regular user. I don't know why I can't get results anymore from a simple script that does nothing more than a tail of the last 5000 lines, and then throws some awk lines at it... I'm only using the sent and received scripts that way. They both worked fine yesterday, but suddenly stopped working around midnight. I can't imagine any special cronjob doing anything...

Any clues of what I might be overseeing here?

Thnx!

D


!! EDIT !!

I found the problem. The cause of the problem lies in the string format used by awk. On my server, the format used is different than the format on the server on which these scripts were made.

The problem was discovered pretty soon luckily because the date has changed from may 31 to june 1. The scripts will format the time string to search for: May 31 09:34:56. Last night our servers turned to june. The format returned in maillog was: Jun 1 09:34:56. The script will format the string to Jun 01 09:34:56. The leading zero (or space) is never matched, thus the script will either return a zero or nothing.

To change this behaviour change the following line:

Code: Select all

        search_time = "^" strftime("%b %d %H:%M", systime() - 5 * 60);
into

Code: Select all

        search_time = "^" strftime("%b %e %H:%M", systime() - 5 * 60);
Then it works again. :D

The other thing you can do is wait for 10 days :wink:

Thanks to my collegue for helping me out so quickly

anTIDot
Posts: 7
Joined: Wed Mar 16, 2005 4:16 am
Contact:

#3 Post by anTIDot » Wed Jun 01, 2005 7:59 am

I can not establish a template. It can with a mistake?

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

Re: huh... suddenly script returning correct values?

#4 Post by warnesj » Wed Jun 01, 2005 9:21 am

D43m0n wrote:They both worked fine yesterday, but suddenly stopped working around midnight. I can't imagine any special cronjob doing anything...

Any clues of what I might be overseeing here?

!! EDIT !!

I found the problem. The cause of the problem lies in the string format used by awk. On my server, the format used is different than the format on the server on which these scripts were made.

The problem was discovered pretty soon luckily because the date has changed from may 31 to june 1. The scripts will format the time string to search for: May 31 09:34:56. Last night our servers turned to june. The format returned in maillog was: Jun 1 09:34:56. The script will format the string to Jun 01 09:34:56. The leading zero (or space) is never matched, thus the script will either return a zero or nothing.
Yeah I noticed all my graphs stopped at midnight too. Damn, I should have waited for a month turn-over before posting my scripts. ;) I made the same change you did and edited my original post so nobody else runs into that. 'man date' is now my new friend. :) Sorry about that. Sweet mod for Postfix too!
anTIDot wrote:I can not establish a template. It can with a mistake?
I'm not sure what you're asking here. Are you having problems installing the Template through the Import Template option in Cacti? Or are you having problems with one of the scripts and getting Net-SNMP to run.

anTIDot
Posts: 7
Joined: Wed Mar 16, 2005 4:16 am
Contact:

#5 Post by anTIDot » Thu Jun 02, 2005 3:05 am

Sorry for my bad English.
At me a problem import of a template file cacti_graph_template_sendmail_mailscanner.xml
Cacti does not wish it to import.
Cacti version 0.8.6d.
In what there can be a problem?

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#6 Post by warnesj » Thu Jun 02, 2005 9:48 am

No appologies required.

Do you get any specific error from Cacti when you try to import the template? Is there anything in the Cacti log file?

Thanks for the info.

anTIDot
Posts: 7
Joined: Wed Mar 16, 2005 4:16 am
Contact:

#7 Post by anTIDot » Fri Jun 03, 2005 6:49 am

I do all as usually:
Console > Import templates > Browse > cacti_graph_template_sendmail_mailscanner.xml > SAVE
And nothing occurs.

Poller Logging Level - DEBUG (poller errors, poller warnings )
In cacti.log - nothing.

yianniska
Posts: 4
Joined: Mon Jun 06, 2005 6:13 am

Import problem

#8 Post by yianniska » Mon Jun 06, 2005 6:22 am

I've got the same problem

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: Import problem

#9 Post by gandalf » Mon Jun 06, 2005 12:08 pm

yianniska wrote:I've got the same problem
This is typically a cacti version mismatch between exporter and importer. Importer must have same or higher (?) level than exporter.

HTH :wink:
Reinhard

africanw
Cacti User
Posts: 124
Joined: Thu Mar 31, 2005 4:24 pm
Location: Sydney, Asutralia
Contact:

#10 Post by africanw » Mon Jun 06, 2005 5:47 pm

Yep same prob with no errors. Running version d.

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#11 Post by warnesj » Mon Jun 06, 2005 10:08 pm

Hmmm, strange. I'm running v0.8.6d on Fedora Core 3. I've updated using the RPM that's posted on the main page, but I've redone the update using the tarball that's there. I've re-exported the template and included it below. See if that one works for you.

Sorry all for the problems with the template, hopefully this one works better.
Attachments
Sendmail_Mailscanner_Template_0.8.6d.zip
Second attempt at the Graph & Data Source template. Exported from Cacti v0.8.6d
(3.53 KiB) Downloaded 1323 times

yianniska
Posts: 4
Joined: Mon Jun 06, 2005 6:13 am

#12 Post by yianniska » Tue Jun 07, 2005 6:35 am

work's for me

thanks

yianniska
Posts: 4
Joined: Mon Jun 06, 2005 6:13 am

#13 Post by yianniska » Tue Jun 07, 2005 6:35 am

work's for me
thanks

yianniska
Posts: 4
Joined: Mon Jun 06, 2005 6:13 am

ok

#14 Post by yianniska » Tue Jun 07, 2005 6:36 am

import is ok but still i cannot get any graphs
when i try to create the graphs

i get this error

RRDTool Says:

ERROR: opening '/var/www/html/cacti0.8.6d/rra/eml2_mess_recv_62.rrd': No such file or directory.

i 've put the scripts to the mail server
i also put the extra lines in my ucdsnmp.conf
but no luck.

could you please help me with this because i find this graphs very
nice

thanks
Last edited by yianniska on Tue Jun 07, 2005 9:13 am, edited 1 time in total.

africanw
Cacti User
Posts: 124
Joined: Thu Mar 31, 2005 4:24 pm
Location: Sydney, Asutralia
Contact:

#15 Post by africanw » Tue Jun 07, 2005 7:46 am

Yep now imports.

Post Reply