Troubleshooting SELinux – Part 2

SELinux Logo

Flashback

In Part 1 of this series we gave a practical overview of Security Enhanced Linux and a simplified explanation of the key terms and concepts.

In this post we will learn how to the determine the status and and current mode of operation on a system, and how to change it. Then we’ll touch on where and how SELinux logs information.

Checking the Status of SELinux

SELinux has two states enabled or disabled.

The enabled state has two modes: permissive or enforcing.

In permissive mode SELinux is doing everything except denying any activity. Log messages will appear just as though SELinux was enforcing policy on your system.

It is highly recommended to start in permissive mode and check your logs before going to enforcing, which we will do below. Permissive mode seems to be the default mode on most linux distros.

The most useful commands for checkiong the status of SELinux are:

sestatus
getenforce

On Centos 7, these utilities can be found in the following packages which are part of the base installation:

[sysadmin@server ~]$ sudo yum -q whatprovides /sbin/sestatus
policycoreutils-2.5-29.el7.x86_64 : SELinux policy core utilities
Repo : @base

[sysadmin@server ~]$ sudo yum -q whatprovides /sbin/getenforce
libselinux-utils-2.5-14.1.el7.x86_64 : SELinux libselinux utilies
Repo : @base
SELinux Disabled

In disabled mode, SELinux is not loaded into the kernel, and no functionality is available.

[sysadmin@server ~]# sestatus
SELinux status:                 disabled

If your machine looks like above, then check the following file:

cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

You will need to enable SELinux by changing “disabled” to “permissive” and then rebooting your machine.

sudo vim /etc/selinux/config
SELINUX=permissive

This should be a safe action. I performed this without issue on a cloud machine on which I have no console access. The machine took a little longer to boot the first time, but came back as expected. Of course, there’s no guarantees so take care!

SELinux ENabled

getenforce provides a simple one-line output, useful for confirming which enabled mode your system is in:

[sysadmin@server ~]$ getenforce
Permissive

sestatus provides a more detailed output:

[sysadmin@server ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

The most useful information here is the mode (permissive) and the policy name (targeted).

From here we’ll assume you are in permissive mode.

How to find SELinux Logs

On Centos, SELinux uses the linux audit daemon (service), known as auditd.

Auditd logs everything to /var/log/audit/audit.log which requires elevated privileges to read:

[sysadmin@server ~]$ less /var/log/audit/audit.log
less: /var/log/audit/audit.log: Permission denied

[sysadmin@server ~]$ sudo less /var/log/audit/audit.log
type=USER_AUTH msg=audit(1546124135.971:451246): pid=30238 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/sbin/sshd" hostname=112.85.42.88 addr=112.85.42.88 terminal=ssh res=failed'
type=USER_AUTH msg=audit(1546124138.373:451247): pid=30238 uid=0 auid=4294967295 ses=4294967295 msg='op=password acct="root" exe="/usr/sbin/sshd" hostname=? addr=112.85.42.88 terminal=ssh res=failed'
<...>

On the plus side, everything you need is in one place. On the negative side, this file can get quite large, contains a log of messages unrelated to SELinux, and the output is a little cryptic.

Filtering Logs

As a first step we can just filter down to the relevant logs, which will contain the text “type=AVC”. Filtering on just “AVC” will also work:

[sysadmin@server ~]$ sudo grep "AVC" /var/log/audit/audit.log | less

AVC stands for Access Vector Cache. This is basically an in-memory cache of all SELinux decisions and is primarily used to improve performance. It also handles the logging of those decisions.

On some of my systems there were no “AVC” entries in the audit file.  On another, most of the entries were generated from activity on the internet-facing web server.

I suggest running this command on some systems your administer and getting a feel for the number of messages being logged on each.

In Part 3…

In the next part we’ll get down in the weeds and troubleshoot some real world examples.

In doing so we’ll start with basic Linux tools, and progress to using some of the more specific SELinux tools. These tools can interpret raw logs for us and even make usable recommendations on how to resolve issues.