Category Archives: Linux

The Linux Category actually encompasses *BSD, RH, Fedora, Ubuntu, and the like.

Fail2ban Ubuntu SSH VSFTP

I like security. The more control I have over a network or system the better I feel. So when I had to push out a couple of secure FTP sites for clients, I had to make sure that they couldn’t be broken into.

First I started with Ubuntu 10.04.3LTS LAMP installation. The rest you can see below:

Update Linux

apt-get update
apt-get upgrade
apt-get install build-essential
apt-get dist-upgrade
reboot

Remove anything unneeded

apt-get autoremove

I installed SSH access to one external-facing system, on a completely separate network, but have edited the allowed hosts to be only my personal public IP. But it’s still a good idea to stop people from trying and filling up the logs. I also installed VSFTP on two external-facing systems – it is these systems that I worry most about.

Install fail2ban

apt-get install fail2ban

Edit the configuration

nano /etc/fail2ban/jail.local

Now I put the following in there:

[DEFAULT]
# “ignoreip” can be an IP address, a CIDR mask or a DNS host
ignoreip = 127.0.0.1 192.168.0.99
bantime = 60
maxretry = 3
backend = polling
# Destination email address used solely for the interpolations in
# jail.{conf,local} configuration files.
destemail = root@localhost
# Default action to take: ban only
action = iptables[name=%(__name__)s, port=%(port)s]
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
[vsftpd]
enabled = false
port = ftp,ftp-data,ftps,ftps-data
filter = vsftpd
logpath = /var/log/vsftpd.log
maxretry = 5

Restart the Fail2ban service

/etc/init.d/fail2ban restart

I checked to see if SSH would be “banned” by trying to connect from another system several times to the fail2ban system (see actions in BOLD):
tail -f /var/log/fail2ban.log

2011-08-24 07:40:29,300 fail2ban.jail : INFO Jail ‘ssh’ uses poller
2011-08-24 07:40:29,330 fail2ban.filter : INFO Added logfile = /var/log/auth.log
2011-08-24 07:40:29,332 fail2ban.filter : INFO Set maxRetry = 5
2011-08-24 07:40:29,336 fail2ban.filter : INFO Set findtime = 600
2011-08-24 07:40:29,337 fail2ban.actions: INFO Set banTime = 60
2011-08-24 07:40:29,481 fail2ban.jail : INFO Jail ‘ssh’ started
2011-08-24 07:40:39,561 fail2ban.actions: WARNING [ssh] Ban 192.168.0.253
2011-08-24 07:41:39,816 fail2ban.actions: WARNING [ssh] Unban 192.168.0.253

You can verify that the ban is active by listing out your iptables
iptables -L

Next I’ll verify that it works on vsftp.
Edit the fail2ban local jail
nano /etc/fail2ban/jail.local
Change the enabled to enabled = true
Restart the fail2ban service
/etc/init.d/fail2ban restart

Then I ran the regex checker
fail2ban-regex /var/log/vsftpd.log /etc/fail2ban/filter.d/vsftpd.conf
Which gave me 0 results.

Edit the failregex configuration
nano /etc/fail2ban/filter.d/vsftpd.conf
I changed from FAIL LOGIN to CONNECT
Save and quit, then restart fail2ban

2011-08-24 08:17:19,564 fail2ban.actions: WARNING [vsftpd] Ban 192.168.0.115
2011-08-24 08:18:19,660 fail2ban.actions: WARNING [vsftpd] Unban 192.168.0.115

***EDIT***
I had a few bots that were trying to get access to directories that didn’t exist.

In my /etc/fail2ban/jail.local

[owncloud]
enabled = true
port = http,https
filter = apache-owncloud
logpath = /var/log/apache2/error.log
maxretry = 3
bantime = 240

In my /etc/fail2ban/filter.d/apache-owncloud.conf

[Definition]
failregex = \[client \] (File does not exist|script not found or unable to stat): [^ ]*/([^ ]*\.asp|[^ ]*\.dll|[^ ]*\.exe|admin|Admin|Ads|ads|apps|archive|awstats|b0ard|bin|blog|board|cgi|clan|cms|community|cube|database|datenbank$

ignoreregex =

Unblock an existing entry
We all know that you can use iptables -L to find the entry and then -D to delete it. Fail2ban doesn’t really like that.

fail2ban-client set JAIL unbanip MYIP
If you forgot your jail name (haha) you can list them all out:
fail2ban-client status

Rancid Ubuntu 10.04

Rancid stands for Really Awesome New Cisco confIg Differ. It’s produced by Shrubbery Networks (http://www.shrubbery.net/rancid/). Basically it’s sole purpose is to make backups of your networking gear. And it can email you of any changes as well. This documentation is a work in progress – I had attempted to setup a rancid server one other time, but ran out of courage shortly after the beginning. And I’m compiling based on a few other internet sources.

Started with a 10.04.3 LTS server with LAMP/Mail(postfix) installed. x86 for those who care. For those who don’t care, it’s still on an x86 system.

I started installation, like all of my other projects, by switching to root:

su

I also created the rancid user with a password:

adduser rancid

Install the Rancid Base:

apt-get install rancid
y

Edit the Rancid Configuration by adding groups:

nano /etc/rancid/rancid.conf
LIST_OF_GROUPS="Group1 Group2 ... Group18"
Save and Quit

Edit the Mail Aliases:

nano /etc/aliases
rancid-Group1: root
rancid-admin-Group1: root
rancid-Group18: root
rancid-admin-Group18: root
Save and Quit

Restart Your Mail Services:

/etc/init.d/postfix restart

Run rancid CVS Groups:

sudo su -c /var/lib/rancid/bin/rancid-cvs -s /bin/bash -l rancid
You should now see a few new directories created in /var/lib/rancid:
/Group1
/Group2

/Group18

We want to edit the configuration files in each of these groups to reflect the devices. We’ll start with /Group1:

cd /var/lib/rancid/Group1
nano router.db
IPADDRESS_OR_HOSTNAME:brand:up_or_down
If you select down, rancid will not actively check this device.
Example:
10.10.0.1:cisco:up
10.15.0.1:cisco:up
10.20.0.20:cisco:up

At one time, whilst setting this up for the first time, I thought that you could only have one login/enable for the entire system. I found that quite “uncool”. Luckily for everyone reading this, I found out that I was incorrect with my assumption. It’s actually quite easy now that I look back on the configuration.

Create A Password File:

The installation of rancid on ubuntu will attempt to create a user called “rancid” with a home directory of /var/lib/rancid, but a login point of /dev/null. But, since we already created the “rancid” user, we must create a cloginrc file to house the passwords for your cisco gear in this user’s home directory. If you run this as root, it’ll be in /root/.clogin.rc
nano /home/rancid/.cloginrc
add method * telnet
add password IPADDRESS_OR_HOSTNAME LOGINPASSWORD ENABLEPASSWORD
add password OTHERIP_OR_HOSTNAME LOGINPASSWORD ENABLEPASSWORD
Save and exit

For SSH:

add autoenable * 1
add method IPADDRESS ssh
add user IPADDRESS USERNAME
add userpassword IPADDRESS PASSWORD
add password IPADDRESS PASSWORD ENABLEPASSWORD

Change permissions of the password file (isn't necessary now that I created the user to begin with):

chmod 640 /var/lib/rancid/.cloginrc
chown rancid /home/rancid/cloginrc

Run the rancid server:

sudo su -c /var/lib/rancid/bin/rancid-run -s /bin/bash -l rancid

View the Log Files:

nano /var/log/rancid/GROUPNAME.DATE.TIME

Where Are The Configuration Files?

/var/lib/rancid/CVS/GROUPNAME/configs/IP_OR_HOSTNAME,v

Too Much Information:

If you feel like you only want to have the configuration files and not the proc info/mem info etc, just edit the following file:
nano /var/lib/rancid/bin/rancid
Search for "# Mail"
Comment out lines you no longer wish to document
Save and Quit

Add the Web GUI:

apt-get install cvsweb
nano /etc/cvsweb/cvsweb.conf
Find "@CVSrepositories"
Add:
'rancid' => ['Rancid', '/var/lib/rancid/CVS'],
Save and Quit
Restart apache
apache2ctl restart
sudo ~rancid/bin/rancid-run (or, if in as root, rancid-run)
You can now open a web browser to http://IPOFSERVER/cgi-bin/cvsweb

EMail testing
I am having a few issues with email aliases - you're supposed to be able to utilize "rancid-NAMEOFDEVICE: real email" or the like for it to forward. I was getting NDRs as it was trying to send locally. So I temporarily changed the NDR to all send to the email address I wanted to send anyway.

Cron job
I cron this for every 10 minutes - but it takes longer and longer to check all the devices on the network when I keep adding more and more to the configuration.

Sample Configuration (/home/rancid/.cloginrc):

add method 10.15.1.245 telnet
add method 10.15.1.246 telnet
add method 10.11.0.2 ssh
add user 10.11.0.2 rancid
add userpassword 10.5.0.2 P@SSw0rd
add password 10.15.1.246 P@55 3n@bL3
add password 10.15.1.245 P@55 3n@bL3
add password 10.11.0.2 P@SSw0rd 3n@bL3

***EDIT***
Sometimes I'm a little slow when it comes to problems with workarounds - I mean, why fix something when it's only halfway broken?
I forgot that when you edit the /etc/aliases file that you have to run the command newaliases to update the /etc/aliases.db file. Otherwise you'll see "warning: database /etc/aliases.db is older than source file /etc/aliases" in your /var/log/mail.info file. And away we go!

***EDIT 2***
If you put in hostnames instead of IP addresses (preferred method as it makes it easier to find in the list later), it's recommended to add static address aliases instead of relying on DNS.
nano /etc/hosts
IPADDRESSOFDEVICE NAMEOFDEVICE NAMEOFDEVICE.DOMAIN.TLD
Ping the name of the device and verify you have resolution.

Also, I found that rancid HATES capital letters. Recommended that everything be lowercase. ESPECIALLY in your .clogin file!!! I found this out with tail -45 /var/log/rancid/LATESTFILEHERE

And, if you follow my above installation procedure, you will find the main page loads without any icons. I sorted this out by the following:
mkdir /var/www/icons
nano /etc/apache/sites-enabled/000-default
Find Alias /doc/ "/usr/share/doc/"
Add the following below it
Alias /cvsweb/icons/ "/var/www/icons/"
Save and quit
apache2ctl restart
Load the dir.gif, back.gif, and text.gif files of your choosing into the /var/www/icons directory

***EDIT 3***
I set this to cron every hour and email changes - it kept emailing the same router.db retrieving version 1.21 over and over - every hour.
I checked the logs (/var/lib/rancid/logs/NEWESTDATE) and found:

cvs commit: Up-to-date check failed for `configs/IPADDRESS'

Just go to
cd /var/lib/rancid/NAMEOFGROUP/configs
and run
cvs update
Rerun rancid and verify
rancid-run

Nagios Add NRPE

NRPE Install and Configuration
This assumes nagios has been installed and configured already. I needed to add nagios monitoring for some remote linux servers. I had only dabbled with nagios before and had setup PING only for testing if a server was up or down. But we now required service testing and HD monitoring.

REMOTE CLIENT INSTALL
Installed on Ubuntu 10.04.3 LTS

Update the system

apt-get update
apt-get upgrade

Install the NRPE server

apt-get install nagios-nrpe-server
This will ask if you want to install a bunch of extra packages – hit Y

Verify the service has started

ps aux | grep nrpe
You should see a command running from /usr/sbin/nrpe

Test a command you want to run. In my case, I wanted to check the /dev/mapper/pve-data drive

/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/pve-data
DISK OK

Edit the NRPE configuration file

nano /etc/nagios/nrpe.cfg
Look for allowed_hosts=127.0.0.1
Change to allowed_hosts=127.0.0.1,IPADDRESSOFNAGIOSSERVER
I then added the following command
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/pve-data
Save the configuration, then restart the service
/etc/init.d/nagios-nrpe-server restart

NAGIOS SERVER CONFIGURATION

Download the latest version of NRPE from http://exchange.nagios.org/directory/Addons/Monitoring-Agents/NRPE–2D-Nagios-Remote-Plugin-Executor/details
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nrpe-2.12.tar.gz
tar zvxf nrpe-2.12.tar.gz
cd nrpe-2.12
./configure --enable-command-args
make all
make install-plugin

Verify connectivity with the remote system from Nagios

/usr/local/nagios/libexec/check_nrpe -H IP.OF.SER.VER
If this spits back NRPE v2.12 then you’re in business. If not, then it won’t connect.

We now have to make a command for Nagios to use NRPE

nano /usr/local/nagios/etc/objects/commands.cfg
Add the following
define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}
Save and close

Now we can create the remote machine configuration file

nano /usr/local/nagios/etc/objects/SOMENAME.cfg

Then restart the Nagios service and test

/etc/init.d/nagios restart

For CentOS/RHEL (RPM)
Navigate to http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
Find the latest release for your distribution (ie RHEL6 and CentOS6 x64 is http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm)
Copy the link location
Open a command shell terminal window on your CentOS/RHEL machine
rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
yum install nagios-nrpe nagios-plugins-nrpe

df -h
find the name of the drive you want to monitor (or drives, ie /dev/mapper/vg_centos-lv_root)
nano /etc/nagios/nrpe.cfg
Find allowed_hosts=127.0.0.1
Change to allowed_hosts=127.0.0.1,10.4.0.253
Find command[check_hda1]
Add a line underneath to be command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/vg_centos-lv-root (same as df -h output)
Verify your command will work:
/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/mapper/vg_centos-lv-root
nano /etc/xinetd.d/nrpe
Find allowed_hosts=127.0.0.1
Change to allowed_hosts=127.0.0.1,10.4.0.253
nano /etc/services
Add nrpe 5666/tcp # nrpe (follow along with the template)
service nrpe restart
And to have this automatically start:
chkconfig --add nrpe
chkconfig nrpe on

I ran the test from nagios (/usr/local/nagios/libexec/check_nrpe -H IP.OF.Server) and was receiving an error:
CHECK_NRPE: Error – Could not complete SSL handshake.

Change the allowed_hosts to only be the IP of the nagios server, not localhost, then service nrpe restart

Add to the nagios monitoring server
Profit!

Delete Files Older Than X Days

I was tasked with auto removing files from a linux FTP site that were older than 14 days. Unfortunately the files were all located in the ~ (home) directories.

I originally tried:
find /home/* -mtime +14 -type f
This command attempts to find ALL (*) files in the home (/home/) directory with a modified time of 14 days (-mtime +14) and only files not directories (-type f)

The command found all the files ok, but it was also listing the hidden files:

/home/user/.profile
/home/user/.bash_logout
/home/user/.bash_history
/home/user/.bashrc

So I decided to use regex:
find /home/* \( ! -regex '.*/\..*' \) -type f -mtime +14
This worked!

Now to add the removal of the files:
find /home/* \( ! -regex '.*/\..*' \) -type f -mtime +14 -exec rm {} \;

I saved this as delete_files, then made it runnable, and cron’d it up:
chmod +x delete_files
crontab -e
@daily ./delete_files #Delete Files Older Than 14 Days in Home Dirs

This worked on my Ubuntu 10.04.3 LTS system, but the commands should work on all the flavors.

Configure Nagios on Ubuntu Server

I’m assuming that you’ve already read the post on installing nagios AND have successfully installed nagios on your LAMP server. I have recently wiped out the 9.04-upgraded-to-10.04.1 server in favor of a fresh install of 10.04.2LTS. I followed my own guide and nagios is installed and running on production equipment.

I needed to add a new user to be able to access all the same admin functionality of the nagios website.
Create the login for apache:

htpasswd /usr/local/nagios/etc/htpasswd.users USERNAME
PASSWORD
PASSWORD

Add the user to the contacts list:

nano /usr/local/nagios/etc/objects/contacts.cfg
define contact{
contact_name USERNAME
use generic-contact
alias FULL NAME
email EMAIL@DOMAIN.TLD
}
Add the USERNAME after “nagiosadmin” under the define contactgroup{ area
Note, you will need to add a “, ” after nagiosadmin

Restart Nagios to re-read the configuration:

/etc/init.d/nagios restart

I would like to make a parent of an object (switch parent of server, or server hosting virtual machines etc)

define host{
use windows-server
host_name HOSTNAME
alias LONGNAME
address IPADDRESS
parents SERVER_OR_SWITCH_NAME – this must have a valid .cfg file already created
hostgroups SERVERGROUP1, SERVERGROUP3, ETC

Add Icons to Nagios Network Map
Upload your png file to /usr/local/nagios/share/images/logos/
Install the PNG to GD2 application:
apt-get install libgd-tools
Convert your PNG to a GD2:
pngtogd2 PNGFILE.png PNGFILE.gd2 cs 1

Create the host image config file:
nano /usr/local/nagios/etc/objects/hostimages.cfg

define hostextinfo{
host_name HOSTNAME, HOSTNAME2, ETC
# notes_url http://sitefornoteshere
icon_image FILENAME.png
icon_image_alt 0 ALTNAMEOFDEVICE
vrml_image FILENAME.png
statusmap_image FILENAME.gd2
}

Restart Nagios:
/etc/init.d/nagios restart