Sendmail & Mailscanner version 2

Templates, scripts for templates, scripts and requests for templates.

Moderators: Moderators, Developers

Post Reply
Author
Message
warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#31 Post by warnesj » Sat Oct 07, 2006 8:53 pm

Sounds like a permissions problem. If you're running Linux, BSD, or some other UNIX varient make sure that the account your Apache daemon runs under has permission to create files in your watchmaillog directory. Same thing with Windows. Whatever account your IIS service is using needs write access to the watchmaillog directory.

What happens is when the counter file is read there is a new file created in that directory to tell the watchmaillog daemon to reset the counter. If it never sees the "reset file" then the counters will keep their indefinately.

rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

Example

#32 Post by rbl » Tue Oct 10, 2006 12:56 pm

Hi,

The rights are good. See the example to see what I meen.
Image

rbl
Last edited by rbl on Wed Oct 11, 2006 10:26 am, edited 1 time in total.

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#33 Post by warnesj » Wed Oct 11, 2006 1:33 am

Ah, I see what you mean now. I think I know what the problem is. The problems is two fold,
  1. The main watchmaillog daemon is responsible for reseting the counter values in the counter file.
  2. Your mail server doesn't recieve a lot of mail, so the main watchmaillog daemon isn't run as often as it would be run on busier servers.
So it is possible that when a counter value is read through SNMP the reset file is created, but the counter doesn't get reset until the main watchmaillog daemon is activated next. But the main watchmaillog daemon isn't activated because a new message doesn't come in. So when your counter is read again, the same value is returned.

I hope that made sense. But you've pointed out a big bug. :oops:

Try this code for a new Read Counters Script,
Read Counters Script
/opt/watchmaillog/readcounters.sh

Code: Select all

#!/usr/bin/perl
#
# Script called by Net-SNMP to read the watchmaillog counters
#
# By Jason Warnes
#
# 2006-09-18: Added new item mess_waiting, which is the number of messages MailScanner
#             detects when a new batch is started.
# 2006-10-11: Fixed bug where counter is flaged to be reset but the main watchmaillog
#             daemon hasn't run to reset the counter.  Don't reset it here, just return a
#             zero value.  (Thanks rbl!)

$counterfile="/opt/watchmaillog/watchmaillog_counters";
$resetfile="/opt/watchmaillog/watchmaillog_reset";
$oid{"spam"}=".1.3.6.1.4.100.2";
$oid{"virus"}=".1.3.6.1.4.100.4";
$oid{"mess_recv"}=".1.3.6.1.4.100.5";
$oid{"mess_sent"}=".1.3.6.1.4.100.6";
$oid{"mess_relay"}=".1.3.6.1.4.100.7";
$oid{"mess_waiting"}=".1.3.6.1.4.100.8";

&readcounterfile;
# If the counter is nothing set it to zero
if($counter{$ARGV[0]}==""){$counter{$ARGV[0]}=0;}
# If a counter reset file exists set the counter to zero
if(-e $resetfile."_".$ARGV[0]){$counter{$ARGV[0]}=0;}
# Return the value of the counter
print "$oid{$ARGV[0]}\ngauge\n$counter{$ARGV[0]}\n";
&createresetflagfile;

sub readcounterfile {   # Subroutine to read the contents of the counter file
        open(COUNTER,$counterfile);
        while($line=<COUNTER>){
                @line=split(/\:/,$line);
                chop($line[1]); # Drop the trailing LF off the value
                $counter{$line[0]}=$line[1];
        }
        close(COUNTER);
}

sub createresetflagfile {       # Subroutine to create the reset counter flag file
        open(RESET,">".$resetfile."_".$ARGV[0]);
        close(RESET);
        chmod(0666,$resetfile."_".$ARGV[0]);
}
That should fix it.

I've also updated the code on the first page of this post. Thanks rbl for pointing this out.

rbl
Posts: 20
Joined: Sat Oct 07, 2006 12:35 pm

great

#34 Post by rbl » Wed Oct 11, 2006 4:42 pm

Hi,
The fix works great now. Tanks :D

sdetroch
Posts: 26
Joined: Thu Mar 31, 2005 7:39 pm

#35 Post by sdetroch » Tue Oct 31, 2006 11:27 am

Hello,

I tried to install this very nice script on our (linux) cacti server to monitor our (fc6) mailscanner server.

I had a problem (like others) to import the E-Mail statistics template (unmet dependencies), the rest of the templates imported fine. I'm running cacti 0.8.6i

A graph is being created for the "MailScanner Messages Waiting" (but I'm not sure that it contains the correct data, since all values are zero.
AND I don't get a graph for the combined info (MailScanner - E-Mail Statistics)

In the GraphTemplate debugging, I get:

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="MailScanner - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="# of Messages" \
--slope-mode \
--font TITLE:12:/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf \
--font AXIS:6:/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf \
CDEF:cdefbd=a,-1,* \
AREA:#00CF00:"Messages Received"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#FF0000:"Viruses"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#EA8F00:"SPAM":STACK \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf" 
RRDTool Says:

ERROR: invalid rpn expression in: a,-1,*
and the graph item fields on the graph mgt page are empty as well.
I seem not to be able to save the selected values.



I assume that I'm connecting the correct way to my mailscanner server (create device / snmp version 1 / generic snmp enabled host)

FYI: SNMP from mailscanner is returning snmp info (system, location, ...), so the connection should be ok (and (empty???) rrd files are being created as well). On the mailscanner server the counters are being updated as well.

Anyone an idea where I should check first, I think some errors are caused by the failed import, but yeah, how to correct them?

edit: In fact, how does the scripts know where to retrieve the data from on the mailscanner server, I haven't given this location anywhere (or is it included in the scripts)???

thanks a lot,
Sven

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#36 Post by warnesj » Wed Nov 01, 2006 11:21 pm

sdetroch wrote:I had a problem (like others) to import the E-Mail statistics template (unmet dependencies), the rest of the templates imported fine. I'm running cacti 0.8.6i
I have yet to upgrade to the 0.8.6i version yet. That unmet dependency thing has been haunting me since the original release. I have the "Include Dependencies" check-box selected when I exported the templates. Argh! :( Just to check, did all the data templates import okay? How about the Messages Waiting graph template? Was it just the MailScanner Stats graph template that is giving you trouble?
sdetroch wrote:A graph is being created for the "MailScanner Messages Waiting" (but I'm not sure that it contains the correct data, since all values are zero.
AND I don't get a graph for the combined info (MailScanner - E-Mail Statistics)

In the GraphTemplate debugging, I get:

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="MailScanner - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="# of Messages" \
--slope-mode \
--font TITLE:12:/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf \
--font AXIS:6:/usr/share/fonts/truetype/ttf-dejavu/DejaVuSerif.ttf \
CDEF:cdefbd=a,-1,* \
AREA:#00CF00:"Messages Received"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#FF0000:"Viruses"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
AREA:#EA8F00:"SPAM":STACK \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT::LAST:"    Current\:%8.0lf"  \
GPRINT::AVERAGE:"Average\:%8.0lf"  \
GPRINT::MAX:"Maximum\:%8.0lf" 
RRDTool Says:

ERROR: invalid rpn expression in: a,-1,*
and the graph item fields on the graph mgt page are empty as well.
I seem not to be able to save the selected values.
That is a problem most definitely. The reason why the "invalid rpn expresion" is coming up as an error is because there are no DEF's, meaning that the graph doesn't think it has any data sources. In comparison here is the output when I debug that graph on my system,

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="mx1 - E-Mail Statistics" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--units-exponent=0 \
--vertical-label="# of Messages" \
--slope-mode \
DEF:a="/var/www/html/cacti/rra/mx1_mess_recv_1982.rrd":mess_recv:AVERAGE \
DEF:b="/var/www/html/cacti/rra/mx1_viruses_1981.rrd":viruses:AVERAGE \
DEF:c="/var/www/html/cacti/rra/mx1_spam_1980.rrd":spam:AVERAGE \
DEF:d="/var/www/html/cacti/rra/mx1_mess_sent_1984.rrd":mess_sent:AVERAGE \
CDEF:cdefbd=d,-1,* \
AREA:a#00CF00:"Messages Recieved"  \
GPRINT:a:LAST:"Current\:%8.0lf"  \
GPRINT:a:AVERAGE:"Average\:%8.0lf"  \
GPRINT:a:MAX:"Maximum\:%8.0lf\n"  \
AREA:b#FF0000:"Viruses"  \
GPRINT:b:LAST:"          Current\:%8.0lf"  \
GPRINT:b:AVERAGE:"Average\:%8.0lf"  \
GPRINT:b:MAX:"Maximum\:%8.0lf\n"  \
AREA:c#EA8F00:"SPAM":STACK \
GPRINT:c:LAST:"             Current\:%8.0lf"  \
GPRINT:c:AVERAGE:"Average\:%8.0lf"  \
GPRINT:c:MAX:"Maximum\:%8.0lf\n"  \
COMMENT:"\n"  \
AREA:cdefbd#7CB3F1:"Messages Sent"  \
GPRINT:d:LAST:"    Current\:%8.0lf"  \
GPRINT:d:AVERAGE:"Average\:%8.0lf"  \
GPRINT:d:MAX:"Maximum\:%8.0lf" 

RRDTool Says:

OK
Notice the DEF's for a,b,c and d.

When you edit the graph in the Graph Management page, are the data sources showing up in the drop-down list as available to be selected? Or are there any data sources in the drop-down list?
sdetroch wrote:I assume that I'm connecting the correct way to my mailscanner server (create device / snmp version 1 / generic snmp enabled host)

FYI: SNMP from mailscanner is returning snmp info (system, location, ...), so the connection should be ok (and (empty???) rrd files are being created as well). On the mailscanner server the counters are being updated as well.
I personally use SNMP version 2, I haven't tried version 1. And the host template I use is "ucd/net SNMP Host", but I would imagine the "Generic SNMP-enabled Host" should work just as well. I think the ucd/net SNMP Host gives you a few more templates like disk usage, CPU usage, load average and memory usage. To test that your Cacti server is able to read the SNMP OIDs that were created for MailScanner and sendmail try this command and see if you get a number returned,

Code: Select all

snmpget -v 1 -c %COMMUNITY_NAME% %IP_ADDR% .1.3.6.1.4.100.2
But replace %COMMUNITY_NAME% with what your community name is, and replace %IP_ADDR% with the IP address of your MailScanner host. That should do an SNMP v1 query on the OID for MailScanner SPAM detected. If that doesn't work try replacing the "-v 1" with "-v 2".
sdetroch wrote:Anyone an idea where I should check first, I think some errors are caused by the failed import, but yeah, how to correct them?
If you can give me a list of which templates worked and which didn't that will help. If all the data templates imported okay, and the messages waiting graph template imported okay, then that will narrow the search down to us trying to figure out what failed on the import of the Mailscanner stats graph template.
sdetroch wrote:edit: In fact, how does the scripts know where to retrieve the data from on the mailscanner server, I haven't given this location anywhere (or is it included in the scripts)???
The data templates tell Cacti what SNMP OID to look at for each item. Then by using those data templates on a device to create data sources, Cacti knows to poll those SNMP OIDs to save the values for each item in it's own RRA's. Then by using those data sources in the graph template the graph is suppose to display the values. Emphasis on "suppose to". :)

sdetroch
Posts: 26
Joined: Thu Mar 31, 2005 7:39 pm

#37 Post by sdetroch » Thu Nov 02, 2006 10:32 am

Thanks for your info.

I found the problem: The datasources were corrupt, problably caused by the failed import.
I recreated all datasources and the graphs started working fine. (the waiting message graph was working without the above fix).



However I still have one problem :P
The "SPAM" counters stays on 'zero' in the counters file (and, logically enough, in the rrd graph as well).
I'm using sendmail on FC6 and spamassasin in combination with Mailscanner.

A known problem or related to my config?
What does the script expects to find?

In my maillog:
[[email protected] watchmaillog]# grep spam /var/log/maillog
Nov 2 16:30:26 mailscanner MailScanner[19016]: Spam Checks: Found 2 spam messages
Nov 2 16:30:28 mailscanner MailScanner[18946]: Spam Checks: Found 1 spam messages
Nov 2 16:30:44 mailscanner MailScanner[18968]: Spam Checks: Found 1 spam messages

Is this logging ok?


EDIT:
I suppose that I need to change
# Look for MailScanner spam messages
if($line=~/is\ spam/ || $line=~/spamd\: identified spam/){
to reflect my logfile. Can someone tell me what to change before I do a trial and error :lol:

Is it normal I have other logfiles as the rest of you???

thanks,
Sven
Last edited by sdetroch on Thu Nov 02, 2006 11:14 am, edited 1 time in total.

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#38 Post by warnesj » Thu Nov 02, 2006 11:14 am

sdetroch wrote:However I still have one problem :P
The "SPAM" counters stays on 'zero' in the counters file (and, logically enough, in the rrd graph as well).
I'm using sendmail on FC6 and spamassasin in combination with Mailscanner.

A known problem or related to my config?
What does the script expects to find?
The script is looking for lines that say "is spam" or "spamd: identified spam" in your maillog file. Here are a couple of entries from my maillog.

Code: Select all

Oct 29 00:37:18 mx1 MailScanner[12968]: Message k9T6awnX022672 from 88.163.40.158 ([email protected]) to domain.com is spam, SBL+XBL, spamcop.net, SpamAssassin (cached, score=21.735, required 5, autolearn=disabled, HTML_MESSAGE 0.00, RAZOR2_CF_RANGE_51_100 0.50, RAZOR2_CF_RANGE_E4_51_100 1.50, RAZOR2_CF_RANGE_E8_51_100 1.50, RAZOR2_CHECK 0.50, RCVD_IN_BL_SPAMCOP_NET 1.33, RCVD_IN_NJABL_DUL 1.71, RCVD_IN_SORBS_DUL 1.99, RCVD_IN_XBL 3.11, URIBL_JP_SURBL 3.36, URIBL_SBL 1.09, URIBL_SC_SURBL 3.60, URIBL_WS_SURBL 1.53)
Oct 29 00:37:18 mx1 MailScanner[12139]: Message k9T6apeN022664 from 200.146.105.238 ([email protected]) to domain2.com is spam, SBL+XBL, SpamAssassin (not cached, score=7.563, required 5, autolearn=disabled, HTML_IMAGE_ONLY_16 0.63, HTML_MESSAGE 0.00, MIME_HTML_ONLY 0.00, RAZOR2_CHECK 0.50, RCVD_IN_BL_SPAMCOP_NET 1.33, RCVD_IN_SORBS_DUL 1.99, RCVD_IN_XBL 3.11)
NOTE:domain1.com and domain2.com are my domains. I just masked them to protect the innocent. ;)
I don't know why your MailScanner logs aren't as verbose. I'm running MailScanner v4.54.6-1 and SpamAssassin v3.1.3-1 and I know there is a setting in the MailScanner.conf file called Log Spam, mine is set to "yes" and that might be what's causing your logs to be so lite. But if you're running a later version there may be other configuration options that may to turn on and off verbose logging. At any rate, go through your MailScanner.conf file and check for any logging related settings.

sdetroch
Posts: 26
Joined: Thu Mar 31, 2005 7:39 pm

#39 Post by sdetroch » Thu Nov 02, 2006 11:17 am

oeps, we posted at the same time :wink:

in the mailscanner config I have disabled this option for performance reasons (75% of incoming mail = spam)


# Do you want all spam to be logged? Useful if you want to gather
# spam statistics from your logs, but can increase the system load quite
# a bit if you get a lot of spam.
Log Spam = no


Can you assist me to change your code to reflect my logfile?
I prefer to go that way and not extra stressing my mailserver.
If it's not possible, I don't have a lot of choice :wink:

Thanks.

Sven

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#40 Post by warnesj » Thu Nov 02, 2006 11:22 am

sdetroch wrote:oeps, we posted at the same time :wink:
LOL, yeah it happens.
sdetroch wrote:Can you assist me to change your code to reflect my logfile? I prefer to go that way and not extra stressing my mailserver.
Absolutely! I didn't try with the less verbose logging, but it's a good idea you have. Give me a few minutes to whip up a code change and we'll see if it works. I'll post it shortly.

sdetroch
Posts: 26
Joined: Thu Mar 31, 2005 7:39 pm

#41 Post by sdetroch » Thu Nov 02, 2006 11:31 am

ok, think it would be usefull for (some) other people as well.
Maybe add it to the next version as well, than everyone has the choice.

kind regards,
Sven

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#42 Post by warnesj » Thu Nov 02, 2006 11:44 am

Okay try this code,

Find the original code that looks for SPAM messages in the watchmaillog.sh script,

Code: Select all

        # Look for MailScanner spam messages
        if($line=~/is\ spam/ || $line=~/spamd\: identified spam/){
                $item="spam";
                &readcounterfile;
                $counter{$item}++;
                if($debug){print("Found a spam message, incrementing the spam counter to $counter{$item}.\n");}
                &writecounterfile;
        }
And replace that with this,

Code: Select all

        # Look for MailScanner spam scanning batch results
        if($line=~/Spam\ Checks\:\ Found/){
                $item="spam";
                $spam_count_pos = index($line,"Spam\ Checks\:\ Found");
                $spam_count_pos2 = index($line, "\ spam\ messages");
                $spam_count = substr($line,($spam_count_pos+19),($spam_count_pos2-($spam_count_pos+19)));
                &readcounterfile;
                $counter{$item}=$counter{$item}+$spam_count;
                if($debug){print("Found $spam_count SPAM in the MailScanner batch, incrementing the spam counter to $counter{$item}.\n");}
                &writecounterfile;
        }
I based this code off the way I look for virus scanning results. It seems to be working okay on my server. :)

I'm not sure the effect this will have on people that use the spamd daemon for SpamAssassin, but I think this change should work for them. If raiten is still following this thread if he could try these changes out on his installation that would be great!
sdetroch wrote:Maybe add it to the next version as well, than everyone has the choice.
I agree. I've actually added the change into this version because it is REALLY handy. I'll update the first page of this thread shortly. I just need to create a new ZIP file to include the new script.

sdetroch
Posts: 26
Joined: Thu Mar 31, 2005 7:39 pm

#43 Post by sdetroch » Thu Nov 02, 2006 12:12 pm

Warnesj,

The new code is working very fine!
Thanks a lot and I keep following this thread.

regards,
Sven

warnesj
Cacti User
Posts: 173
Joined: Sun May 29, 2005 7:34 pm

#44 Post by warnesj » Thu Nov 02, 2006 12:32 pm

sdetroch wrote:Warnesj,

The new code is working very fine!
Thanks a lot and I keep following this thread.

regards,
Sven
Great to hear! I've updated the first post too to include the code changes.

User avatar
sizulku
Cacti User
Posts: 110
Joined: Mon Nov 04, 2002 9:15 am
Location: ACEH
Contact:

#45 Post by sizulku » Thu Nov 02, 2006 9:38 pm

how to add "rejected messages" to this graph?

Post Reply