Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sun Mar 24, 2019 6:56 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: Odd Remote Poller Bug(?)
PostPosted: Tue Nov 20, 2018 12:53 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
I am observing some weird behavior I cannot explain (So far).

Note this is close to what I was experiencing some time back which was pinned down to duplicate SNMPv3 Engine ID's but not identical.

I will randomly encounter an issue (Infrequently) where logged into the mail console a node will give a "ERROR: Failed to connect to remote collector."

Once I receive this error ALL nodes connected to the affected poller I try to load will give the same error (Nodes connected to other pollers still work fine)

Configured nodes connected to the affected poller also still report data back, and show as "UP" but host.php gives the Collector error when I try to access them.

If I log into the web interface for the affected poller (Which shows online) and edit a node it immediately loads without issue, and then from that point the central console is able to resume editing nodes without error as well. Note I don't save anything on the remote poller, no service restarts, nothing.. Just loading a node on the affected poller resolves the issue.

I am running v1.1.36, is this behavior anyone else has experienced? Or potentially a bugfix (Or issue) I have missed?

The fix is really simple so it isn't a catastrophe, but I'm just at a loss as to what triggers the issue..


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Tue Nov 20, 2018 1:07 pm 
Offline
Developer

Joined: Thu Apr 07, 2005 3:29 pm
Posts: 2747
Location: B/CS Texas
The Remote Pollers were majorly overhauled in 1.2, so I am betting your problem will be resolved once that is released.

_________________
Report a bug
Download Releases
1.X Compatible Plugins


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Tue Nov 20, 2018 1:10 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
Sounds good!

Like I said it's really infrequent, and very easy to fix (Nothing has to be restarted and no data is ever lost) it's just more confusing than anything.

I'll table it until we have a prod release of 1.2.x and see if it goes away.

Thanks!


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Tue Nov 20, 2018 6:37 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
Just a quick addendum, (because I'm terrible at just leaving unexplained things be).

Digging around my servers I found a number of selinux denies on a few of my pollers, somehow (And I am at a loss here) on one side I had pollers where the selinux context on the Cacti web directory was user_home, and on the other side I had (the correct) httpd_sys_content.

I ran a restorecon on these trees, and checked audit2allow for any other goofy denies and fixed those, the denies were all related to Cacti so I'm anxious to see if the issues go away.. As I don't know what triggers it, and as such cannot reliably duplicate it I am going to just wait and see.. But definitely worth noting.

Not sure why I didn't check selinux sooner as it's pretty much the go to for x-files type mystery issues.


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Wed Nov 21, 2018 3:18 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2296
SELinux is good in many ways but so bad when it comes to blocking things without even a hint that it was involved. In fact, on most systems, I find it gets in the way far more than it helps especially for the novice linux admins who suddenly have to learn security contexts and auditing. It's a bit like learning the advanced ACL's that Linux can do but most systems never bother with.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Mon Nov 26, 2018 11:09 am 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
netniV wrote:
SELinux is good in many ways but so bad when it comes to blocking things without even a hint that it was involved. In fact, on most systems, I find it gets in the way far more than it helps especially for the novice linux admins who suddenly have to learn security contexts and auditing. It's a bit like learning the advanced ACL's that Linux can do but most systems never bother with.


You actually make an excellent point (Around people not bothering to learn SELinux), it actually is dauntingly complex when you first encounter it so I thought maybe I should add more detail on what was checked (And fixed) for anyone who reads this and is scratchign their head.

1. The logs for SELinux (At least on a Centos Build) is /var/log/audit/audit.log

To see if selinux is running at all you can type 'sestatus'

2. The audit log is pretty complicated honestly, but there is a tool for reading it that makes things MUCH easier. What I ran was 'audit2allow -a -w -l'
- the -a flag reads in the audit and message logs
- the -l flag reads in messages since last policy reload (So when you fix denies and re-run it you do not see things you have fixed)
- the -w flaw gives you 'why' the deny happened

This command will tell you in plain english what is being denied and how to fix it.

3. When listing files or directories you can add the '-Z' flag to display the SELinux context of a file or directory.

4. The 'restorecon' utility will restore files to their default (Usually correct) context, I ran 'restorecon -R -n -v' first.
- The -R flag is the same as many other commands, making the command run recursively down a directory tree.
- The -v flag, again like other commands specifies verbose (Also like many other commands you can add multiple levels by adding v's)
- The -n flag specifies to do nothing, and only list changes that WOULD be made, ALWAYS run this way first and review that it is going to actually do what you expect, you will save yourself a lot of headache! (In my case it did in fact show exactly what I wanted so I ran it again without the -n flag)

5. Probably the MOST important part of this, and something that I rant about QUITE often!!

SELINUX has multiple modes.

enforcing
permissive
disabled

enforcing is what you would expect, it is on and actively protecting your system using the specified policy.

disabled means 'off', selinux is doing nothing. This option did not exist until recently, and it only exists (IMO) because of stupidity. In disabled mode nothing is logged and the selinux daemon is idle.

permissive means 'off' (Weird right? Why have 2 modes for off??), or more accurately it is in PASSIVE mode, meaning the daemon is running, and logging exactly as it would in 'enforcing' mode. EXACTLY.. With the only difference being it only logs, and doesn't block anything or affect how your system works in any way.

The proper way to tune this for a new application, or where selinux may be an issue, is to set this to permissive, and then use the above tools to fix any noted issues (Noting Selinux will log EXACTLY as it would in enforcing mode, it just won't actually block anything meaning you can safely tune your system).

There is (IMO) no reason to ever select 'disabled', and (Again IMO) the option shouldn't even exist, if you don't want to tune SELinux that's fine, run permissive, at LEAST log what the daemon want's to do and occasionally look at what it is TRYING to do..


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Mon Nov 26, 2018 11:57 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2296
Great post about SELinux, but I will say that personally, the difference between 'disabled' and 'permissive' is massive if you never ever intend to use it. One is wasting resources, the other should be shutting the daemon down (don't know if it does that or just starts a process that is never needed).

Maybe you should post your above post separately for people to find it better in the HOWTO's ?

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Mon Nov 26, 2018 12:29 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
You're probably right.

I just know over the years I was asked SO many times "Why isn't there an option to disable SELinux" by people that didn't understand what permissive was.. Granted the naming could be more intuitive, as you might think it is just a less restrictive policy..

Apparently I still have strong feelings about this lol.. I'll pop my info into a howto, let me clean it up a bit.


Top
 Profile  
 
 Post subject: Re: Odd Remote Poller Bug(?)
PostPosted: Mon Nov 26, 2018 1:24 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2018 1:37 pm
Posts: 56
https://forums.cacti.net/viewtopic.php?f=6&t=59316

Give it a look if you have a free minute or two, it was the best I could do rushing but if you see areas to improve or clean up let me know!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: Google [Bot] and 11 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group